wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Speech recognition systems need thousands of hours of transcribed speech, unavailable for most of the world’s ~7,000 languages. wav2vec 2.0 learns representations from unlabelled audio and then fine-tunes on small amounts of transcribed speech.

A CNN encodes raw audio into latent speech representations; spans of latents are masked and fed to a Transformer; the model must identify the true quantized latent for each masked step among distractors (a contrastive task), with the discrete speech units learned jointly via Gumbel softmax.

With all 960 h of labelled LibriSpeech, 1.8/3.3 WER on test-clean/other. With only 1 h labelled, it beat the previous best on the 100-hour subset using 100× less labelled data; with just 10 minutes labelled (plus 53k h unlabelled), 4.8/8.2 WER.

Those learned discrete units in a first step and then contextual representations; wav2vec 2.0 learns both end to end, builds context over continuous (not quantized) latents, and uses quantization only for the targets, cutting WER by about a third over Discrete BERT.

A multi-layer convolutional feature encoder \(f: \mathcal{X} \to \mathcal{Z}\) from raw audio to latents \(z_1, \ldots, z_T\); a Transformer context network \(g: \mathcal{Z} \to \mathcal{C}\) giving contextual representations \(c_1, \ldots, c_T\); and a quantization module \(\mathcal{Z} \to \mathcal{Q}\) producing discrete targets \(q_t\).

Instead of fixed absolute positional embeddings, a convolutional layer over the latents acts as a relative positional embedding; its output (after GELU) is added to the inputs, followed by layer normalization.

Product quantization: \(G\) codebooks with \(V\) entries each; choose one entry per codebook, concatenate, and apply a linear map. Choices use a Gumbel softmax, \(p_{g,v} = \frac{\exp((l_{g,v} + n_v)/\tau)}{\sum_k \exp((l_{g,k} + n_k)/\tau)}\), with hard argmax forward and straight-through gradients backward. With \(G = 2\), \(V = 320\) there are up to 102.4k codewords.

In latent space: sample a proportion \(p\) of time steps as span starts (without replacement) and mask the following \(M\) steps, replacing them with one shared learned vector; spans may overlap. With \(p = 0.065\), \(M = 10\), about 49% of steps are masked, mean span about 300 ms. Inputs to the quantizer are not masked.

For a masked step \(t\), the context vector \(c_t\) must pick the true quantized latent \(q_t\) among \(K + 1\) candidates (\(K\) distractors sampled from other masked steps of the same utterance): \(\mathcal{L}_m = -\log \frac{\exp(\text{sim}(c_t, q_t)/\kappa)}{\sum_{\tilde q \in Q_t} \exp(\text{sim}(c_t, \tilde q)/\kappa)}\), with cosine similarity and temperature \(\kappa = 0.1\).

\(\mathcal{L}_d = \frac{1}{GV} \sum_{g=1}^{G} -H(\bar p_g)\), minimizing negative entropy of the batch-averaged codebook distributions, so all \(V\) entries of each codebook get used; the contrastive task depends on the codebook for both positives and negatives. Total loss \(\mathcal{L} = \mathcal{L}_m + \alpha \mathcal{L}_d\) with \(\alpha = 0.1\).

Add a randomly initialized linear projection on top of the Transformer into the task vocabulary (29 characters plus a word boundary for LibriSpeech), train with a CTC loss, and mask time steps and channels as in SpecAugment, which delays overfitting, especially with very little labelled data. The feature encoder is frozen.

Unlabelled: LibriSpeech audio (960 h) or LibriVox (LV-60k, 53.2k h). Labelled fine-tuning: 960 h, the 100 h clean subset, and Libri-light’s 10 h, 1 h and 10 min splits; TIMIT (5 h) for phoneme recognition with 39 phone classes.

Seven convolutional blocks with strides (5, 2, 2, 2, 2, 2, 2) give a total stride of \(5 \cdot 2^6 = 320\) samples, about 20 ms at 16 kHz (49 Hz), with a 25 ms receptive field.

Same encoder; BASE has 12 Transformer blocks (dimension 768, FFN 3,072, 8 heads, 95M parameters), LARGE has 24 (1,024, 4,096, 16 heads, 317M). LARGE on LV-60k trained 5.2 days on 128 V100 GPUs.

LARGE pre-trained on LV-60k: with 10 minutes (48 recordings of ~12.5 s) 4.8/8.2 test WER with a Transformer LM; with 1 h 2.9/5.8; with 10 h 2.6/4.9; with 100 h 2.0/4.0, beating iterative self-training, which needs repeated labelling, filtering and retraining.

Both helped: LARGE beat BASE on all setups, most on the noisy test-other, and pre-training on 53k h (LV-60k) instead of 960 h gave large further gains.

1.8/3.3 WER on test-clean/other, state of the art on noisy speech, despite a weaker setup than competitors: CTC rather than seq2seq, character rather than word-piece units, and no data balancing. The same architecture trained from scratch got 2.1/4.6.

New state of the art without a language model: 7.4/8.3 phoneme error rate on dev/test, a 23%/29% relative reduction over the next best.

Ablations showed it was best (5.4 dev WER vs 7.97 for quantized inputs and targets, 12.18 for quantized inputs with continuous targets). Continuous inputs retain information for context; continuous targets can encode speaker and background details that make the task too easy (training accuracy rose from 62% to 78%) and the representations less general.

Speech recognition with very small amounts of transcribed data could make the technology available for many more of the world’s languages and dialects, most of which lack the hundreds or thousands of labelled hours current systems need.