Embeddings

Historical notes

1. Which three fields in the 1950s contributed to vector semantics?
   Linguistics (distributionalists Harris, Joos, Firth: meaning from distribution), psychology (Osgood: meaning as a point in space, similarity as distance), and mechanical indexing, i.e. information retrieval (the vector space model, idf, mutual information).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, Historical notes, p. 148
2. What was Wittgenstein’s view that prefigured distributional semantics?
   “The meaning of a word is its use in the language”: words should be defined by how people use them rather than by formal definitions or truth values.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, Historical notes, p. 149
3. What are semantic features?
   Symbols for primitive meanings used to define words (e.g. hen = +female, +chicken, +adult; chick = +chicken, −adult), from Hjelmslev and early generative grammar.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, Historical notes, p. 149
4. What is LSA?
   Latent semantic analysis: apply singular value decomposition (SVD) to a weighted term-document matrix and keep the top ~300 dimensions as dense embeddings. It was used as a cognitive model and for many tasks.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, Historical notes, p. 149
5. How did the meaning of the word “embedding” shift?
   In LSA it meant the mapping from sparse count space to the dense latent space; it shifted metonymically to mean the resulting dense vector itself.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, Historical notes, p. 150
6. Who proposed skip-gram and negative sampling?
   Mikolov et al. (2013a) proposed skip-gram and CBOW (simplifying neural LM hidden layers); negative sampling was proposed in Mikolov et al. (2013b).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, Historical notes, p. 150