wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
Baevski et al., 2020, wav2vec 2.0 (Advances in Neural Information Processing Systems 33 (NeurIPS 2020))
Subscriber deck — free during early access
1 cards in “3.1 Masking” · whole deck
3.1 Masking
- How does wav2vec 2.0 mask its input?In latent space: sample a proportion \(p\) of time steps as span starts (without replacement) and mask the following \(M\) steps, replacing them with one shared learned vector; spans may overlap. With \(p = 0.065\), \(M = 10\), about 49% of steps are masked, mean span about 300 ms. Inputs to the quantizer are not masked.
- Baevski et al., 2020, wav2vec 2.0 (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §3.1
Source
- Alexei Baevski, Henry Zhou, Abdelrahman Mohamed and Michael Auli. 2020. wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020). https://papers.nips.cc/paper/2020/hash/92d1e1eb1cd6f9fba3227870bb6d7f07-Abstract.html