Distributed Representations of Words and Phrases and their Compositionality

3 Empirical Results

1. How is the word analogy task evaluated?
   For “Germany : Berlin :: France : ?”, find the word whose vector is closest by cosine to vec(“Berlin”) − vec(“Germany”) + vec(“France”), excluding the input words; it is correct only if it is Paris. Questions are syntactic (quick : quickly :: slow : slowly) or semantic (country–capital).
   • Mikolov et al., 2013, Distributed Representations of Words and Phrases and their Compositionality (Advances in Neural Information Processing Systems 26 (NIPS 2013)), §3
2. How did negative sampling, NCE and hierarchical softmax compare on word analogies?
   With 300-dimensional vectors on 1B words of news: NEG-15 61%, NEG-5 59%, NCE-5 53%, hierarchical softmax 47% total accuracy. With \(10^{-5}\) subsampling, training got 2–3× faster and accuracy rose (NEG-15 61%, NEG-5 60%, HS 55%).
   • Mikolov et al., 2013, Distributed Representations of Words and Phrases and their Compositionality (Advances in Neural Information Processing Systems 26 (NIPS 2013)), §3
3. Is the linear analogy structure just an artefact of Skip-gram being linear?
   Probably not: earlier work found that highly non-linear recurrent network language models also improve on analogies as training data grows, suggesting non-linear models also prefer a linear structure of word representations.
   • Mikolov et al., 2013, Distributed Representations of Words and Phrases and their Compositionality (Advances in Neural Information Processing Systems 26 (NIPS 2013)), §3