Embeddings

5.7 Semantic properties of embeddings

1. How does the context window size affect what embeddings capture?
   Short windows (±1–3) give more syntactic representations: the nearest words are semantically similar words of the same part of speech. Long windows give topically related words that aren’t necessarily similar.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, §5.7, p. 143
2. With skip-gram, what are the nearest neighbors of Hogwarts with window ±2 versus ±5?
   ±2: other fictional schools (Sunnydale, Evernight). ±5: words from the Harry Potter topic (Dumbledore, Malfoy, half-blood).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, §5.7, p. 143
3. What is the difference between first-order and second-order co-occurrence?
   First-order (syntagmatic) association: words typically near each other (wrote and book). Second-order (paradigmatic) association: words with similar neighbors (wrote and said, remarked).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, §5.7, p. 143
4. What is the parallelogram model of analogy?
   For “a is to b as a* is to ?” (apple : tree :: grape : ?), add the offset \(b - a\) to \(a^*\) and return the nearest word (vine): \(\hat{b}^* = \arg\min_x \text{distance}(x, b - a + a^*)\).

   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, §5.7, p. 143
5. Give classic examples of analogies solved with word2vec or GloVe offsets.
   king − man + woman ≈ queen; Paris − France + Italy ≈ Rome. Offsets also seem to capture relations like comparative/superlative morphology.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, §5.7, p. 144
6. What caveats apply to the parallelogram method with embeddings?
   The nearest vector is usually one of the input words or a morphological variant (cherry : red :: potato : potato), so those must be excluded; it works for frequent words, small distances and certain relations (capitals, inflections) but poorly for others, and is probably too simple a model of human analogy.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, §5.7, pp. 144–145
7. How can embeddings be used to study historical semantic change?
   Train separate embedding spaces on texts from each period (e.g. per decade) and compare a word’s neighbors over time: gay moved from “cheerful” to “homosexual”, broadcast from sowing seeds to transmitting signals, awful from “full of awe” to “terrible” (pejoration).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, §5.7, p. 145