wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

2 Model

1. What are the three components of wav2vec 2.0?
   A multi-layer convolutional feature encoder \(f: \mathcal{X} \to \mathcal{Z}\) from raw audio to latents \(z_1, \ldots, z_T\); a Transformer context network \(g: \mathcal{Z} \to \mathcal{C}\) giving contextual representations \(c_1, \ldots, c_T\); and a quantization module \(\mathcal{Z} \to \mathcal{Q}\) producing discrete targets \(q_t\).

   • Baevski et al., 2020, wav2vec 2.0 (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §2
2. How does wav2vec 2.0 encode position in its Transformer?
   Instead of fixed absolute positional embeddings, a convolutional layer over the latents acts as a relative positional embedding; its output (after GELU) is added to the inputs, followed by layer normalization.
   • Baevski et al., 2020, wav2vec 2.0 (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §2
3. How does wav2vec 2.0 quantize latents?
   Product quantization: \(G\) codebooks with \(V\) entries each; choose one entry per codebook, concatenate, and apply a linear map. Choices use a Gumbel softmax, \(p_{g,v} = \frac{\exp((l_{g,v} + n_v)/\tau)}{\sum_k \exp((l_{g,k} + n_k)/\tau)}\), with hard argmax forward and straight-through gradients backward. With \(G = 2\), \(V = 320\) there are up to 102.4k codewords.
   • Baevski et al., 2020, wav2vec 2.0 (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §2