Robust Speech Recognition via Large-Scale Weak Supervision

They learn excellent audio encoders from unlabeled speech but have no comparably good decoder, so each use needs supervised fine-tuning, which is complex and risks exploiting dataset-specific quirks that don’t generalize. Whisper aims for a system that works “out of the box” without fine-tuning.

Scale weakly supervised training, on imperfect transcripts found on the internet, to 680,000 hours of labeled audio, an order of magnitude more than earlier weakly supervised datasets, and make it multilingual and multitask; the resulting models transfer zero-shot to standard benchmarks.

Mostly English speech recognition, plus 117,000 hours in 96 other languages and 125,000 hours of X→English speech translation.

They removed machine-generated transcripts (detected by heuristics such as missing punctuation or all-caps text), dropped pairs whose spoken and transcript languages disagreed (keeping English transcripts of non-English audio as translation data), fuzzy de-duplicated transcripts, and after training an initial model inspected high-error data sources to remove misaligned or partial ones.

A sequence-to-sequence model can learn the mapping from speech to naturally written text, so no separate inverse text normalization step (e.g. turning “twenty dollars” into “$20”) is needed in the pipeline.

A standard encoder–decoder Transformer, deliberately off the shelf so that results reflect the data. Audio is resampled to 16 kHz and turned into an 80-channel log-Mel spectrogram (25 ms windows, 10 ms stride) over 30-second segments; two convolution layers (the second with stride 2) precede the encoder; the decoder predicts byte-level BPE tokens.

Tasks are specified as a token sequence for the decoder: a start token, a language token (or a no-speech token), a task token (transcribe or translate), an optional no-timestamps token, then the text (with interleaved time tokens if timestamps are on), and an end token. Earlier transcript text can be given as context.

As extra vocabulary tokens for times relative to the 30-second segment, quantized to 20 ms; a start time token precedes each caption’s text and an end time token follows it.

They often “transcribed” plausible but wrong speaker names, because many training transcripts name the speaker; brief fine-tuning on transcripts without speaker annotations removed it.

WER penalizes every difference from the reference, including harmless formatting differences, which hits zero-shot models hard since they never saw a dataset’s transcription style. Their normalizer removes such non-semantic differences, with the caveat that it might be overfit to Whisper’s style.

Human performance measures out-of-distribution generalization (people get no training on the test distribution), while models trained on LibriSpeech are measured in-distribution. The same test set thus measures two different abilities.

How much better a model does on out-of-distribution datasets than expected from its score on a reference dataset (here LibriSpeech). A zero-shot Whisper model and a supervised model with nearly equal LibriSpeech WER differed hugely elsewhere: Whisper made 55.2% fewer errors on average across other datasets, roughly matching a human’s robustness.

Log WER is strongly correlated with log training hours for that language (\(R^2 = 0.83\)); WER roughly halves for every 16× more data. The worst outliers were languages with unique scripts and distant from the Indo-European majority, such as Hebrew, Telugu, Chinese and Korean.

It set a zero-shot state of the art of 29.1 BLEU on CoVoST2, helped by 68,000 hours of noisy translation data for those languages (vs 861 hours of CoVoST2 training data), with the biggest gains on low-resource languages but no gain on high-resource ones.

Its training data covered only 82 of Fleurs’ 102 languages, capping accuracy at 80.4%; overall it reached 64.5% vs 77.7% for the supervised state of the art.

Every increase helped, but with diminishing returns: English WER improved quickly up to 13,000 hours and then slowed, gaining only about 1 point from the final 12.5× increase; translation was near zero below 7,000 hours, then improved roughly log-linearly up to 54,000 hours.

For small models, yes: joint models underperformed English-only models trained with the same compute (negative transfer). But joint models scaled better, and the largest ones outperformed English-only training, showing positive transfer.

It transcribes consecutive 30-second windows, shifting by predicted timestamps, with heuristics: beam search, raising temperature when output looks repetitive (high gzip compression ratio) or low-probability, conditioning on the previous window’s text, combining no-speech probability with log-probability for voice activity detection, and constraining the first timestamp.