Distributed Representations of Words and Phrases and their Compositionality

5 Additive Compositionality

1. What is additive compositionality of Skip-gram vectors? Give examples.
   Element-wise addition of word vectors often gives meaningful results: vec(“Russian”) + vec(“river”) is close to vec(“Volga River”), vec(“Vietnam”) + vec(“capital”) to vec(“Hanoi”), vec(“German”) + vec(“airlines”) to vec(“Lufthansa”).
   • Mikolov et al., 2013, Distributed Representations of Words and Phrases and their Compositionality (Advances in Neural Information Processing Systems 26 (NIPS 2013)), §5
2. Why does adding Skip-gram vectors work, according to Mikolov et al.?
   Word vectors are trained to predict context, so they represent a word’s context distribution, related logarithmically to the output probabilities. Summing two vectors corresponds to multiplying the two context distributions, an AND: words likely in both contexts (e.g. “Volga River” near both Russian and river) score high.
   • Mikolov et al., 2013, Distributed Representations of Words and Phrases and their Compositionality (Advances in Neural Information Processing Systems 26 (NIPS 2013)), §5