Embeddings

5.2 Vector semantics: the intuition

1. Which two 1950s ideas converged into vector semantics?
   Osgood’s representation of connotation as a point in space, and the distributional idea (Joos, Harris, Firth) of defining a word by its distribution: its neighboring words or grammatical environments.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, §5.2, p. 131
2. How can you figure out that the unknown word ongchoi is a leafy green?
   It occurs with words like garlic, rice, delicious, salty, leaves, as do spinach, chard and collard greens. Computationally: count the words in its context.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, §5.2, p. 131
3. Why do embeddings help a sentiment classifier generalize?
   A classifier on word strings needs the same words in training and test. With embeddings it can assign sentiment as long as it has seen words with similar meanings, which lie nearby in space.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, §5.2, p. 132
4. What is the difference between sparse and dense word vectors?
   Sparse vectors (count-based) are long (dimension \(|V|\)) and mostly zeros. Dense vectors (e.g. word2vec) are short (50–1000 dimensions) and contain real values, possibly negative.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, §5.2, p. 132