wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

3.2 Objective

1. Give wav2vec 2.0’s contrastive loss.
   For a masked step \(t\), the context vector \(c_t\) must pick the true quantized latent \(q_t\) among \(K + 1\) candidates (\(K\) distractors sampled from other masked steps of the same utterance): \(\mathcal{L}_m = -\log \frac{\exp(\text{sim}(c_t, q_t)/\kappa)}{\sum_{\tilde q \in Q_t} \exp(\text{sim}(c_t, \tilde q)/\kappa)}\), with cosine similarity and temperature \(\kappa = 0.1\).
   • Baevski et al., 2020, wav2vec 2.0 (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §3.2
2. What is the diversity loss in wav2vec 2.0, and why is it needed?
   \(\mathcal{L}_d = \frac{1}{GV} \sum_{g=1}^{G} -H(\bar p_g)\), minimizing negative entropy of the batch-averaged codebook distributions, so all \(V\) entries of each codebook get used; the contrastive task depends on the codebook for both positives and negatives. Total loss \(\mathcal{L} = \mathcal{L}_m + \alpha \mathcal{L}_d\) with \(\alpha = 0.1\).
   • Baevski et al., 2020, wav2vec 2.0 (Advances in Neural Information Processing Systems 33 (NeurIPS 2020)), §3.2