learn.ifs.name

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Baevski et al., 2020, wav2vec 2.0 (Advances in Neural Information Processing Systems 33 (NeurIPS 2020))

Subscriber deck — free during early access

2 cards in “4.2 Pre-training” · whole deck

4.2 Pre-training

  1. What is the frame rate of wav2vec 2.0’s feature encoder, and why?
    Seven convolutional blocks with strides (5, 2, 2, 2, 2, 2, 2) give a total stride of \(5 \cdot 2^6 = 320\) samples, about 20 ms at 16 kHz (49 Hz), with a 25 ms receptive field.
    • Baevski et al., 2020, wav2vec 2.0 (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §4.2
  2. What are wav2vec 2.0’s BASE and LARGE configurations?
    Same encoder; BASE has 12 Transformer blocks (dimension 768, FFN 3,072, 8 heads, 95M parameters), LARGE has 24 (1,024, 4,096, 16 heads, 317M). LARGE on LV-60k trained 5.2 days on 128 V100 GPUs.
    • Baevski et al., 2020, wav2vec 2.0 (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §4.2

Source