wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

1 Introduction

1. What problem does wav2vec 2.0 address?
   Speech recognition systems need thousands of hours of transcribed speech, unavailable for most of the world’s ~7,000 languages. wav2vec 2.0 learns representations from unlabelled audio and then fine-tunes on small amounts of transcribed speech.
   • Baevski et al., 2020, wav2vec 2.0 (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §1
2. Summarize how wav2vec 2.0 is pre-trained.
   A CNN encodes raw audio into latent speech representations; spans of latents are masked and fed to a Transformer; the model must identify the true quantized latent for each masked step among distractors (a contrastive task), with the discrete speech units learned jointly via Gumbel softmax.
   • Baevski et al., 2020, wav2vec 2.0 (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §1
3. What were wav2vec 2.0’s headline results?
   With all 960 h of labelled LibriSpeech, 1.8/3.3 WER on test-clean/other. With only 1 h labelled, it beat the previous best on the 100-hour subset using 100× less labelled data; with just 10 minutes labelled (plus 53k h unlabelled), 4.8/8.2 WER.
   • Baevski et al., 2020, wav2vec 2.0 (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §1
4. How does wav2vec 2.0 improve on vq-wav2vec and Discrete BERT?
   Those learned discrete units in a first step and then contextual representations; wav2vec 2.0 learns both end to end, builds context over continuous (not quantized) latents, and uses quantization only for the targets, cutting WER by about a third over Discrete BERT.
   • Baevski et al., 2020, wav2vec 2.0 (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §1