Distributed Representations of Words and Phrases and their Compositionality

2.3 Subsampling of Frequent Words

1. How does Skip-gram subsample frequent words, and why?
   Each occurrence of word \(w_i\) is discarded with probability \(P(w_i) = 1 - \sqrt{t / f(w_i)}\), where \(f\) is its frequency and \(t \approx 10^{-5}\). Frequent words like the carry little information (seeing France with Paris helps more than with the) and their vectors stop changing after millions of examples.
   • Mikolov et al., 2013, Distributed Representations of Words and Phrases and their Compositionality (Advances in Neural Information Processing Systems 26 (NIPS 2013)), §2.3
2. What does the subsampling formula preserve, and what did it achieve?
   It aggressively subsamples words with frequency above \(t\) while keeping the frequency ranking. It sped training up several times and significantly improved vectors of rarer words.
   • Mikolov et al., 2013, Distributed Representations of Words and Phrases and their Compositionality (Advances in Neural Information Processing Systems 26 (NIPS 2013)), §2.3