Distributed Representations of Words and Phrases and their Compositionality

2.2 Negative Sampling

1. What is the negative sampling (NEG) objective?
   Replace each \(\log p(w_O \mid w_I)\) with \(\log \sigma(v'^\top_{w_O} v_{w_I}) + \sum_{i=1}^{k} \mathbb{E}_{w_i \sim P_n(w)} \log \sigma(-v'^\top_{w_i} v_{w_I})\): logistic regression distinguishing the real context word from \(k\) words sampled from a noise distribution.
   • Mikolov et al., 2013, Distributed Representations of Words and Phrases and their Compositionality (Advances in Neural Information Processing Systems 26 (NIPS 2013)), §2.2
2. How does negative sampling differ from Noise Contrastive Estimation?
   NCE needs both noise samples and their numerical probabilities, and approximately maximizes the softmax log probability. NEG uses only samples and doesn’t approximate the softmax, which is fine since the goal is good vectors, not a language model.
   • Mikolov et al., 2013, Distributed Representations of Words and Phrases and their Compositionality (Advances in Neural Information Processing Systems 26 (NIPS 2013)), §2.2
3. Which \(k\) and noise distribution worked best for negative sampling?
   \(k\) = 5–20 for small data sets, 2–5 for large ones. The unigram distribution raised to the 3/4 power, \(U(w)^{3/4}/Z\), significantly beat both the plain unigram and the uniform distribution.
   • Mikolov et al., 2013, Distributed Representations of Words and Phrases and their Compositionality (Advances in Neural Information Processing Systems 26 (NIPS 2013)), §2.2