Embeddings

Introduction

1. State the distributional hypothesis, and who formulated it.
   Words that occur in similar contexts tend to have similar meanings; the difference in meaning between two words corresponds roughly to the difference in their environments. Formulated in the 1950s by linguists Joos, Harris and Firth.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, Introduction, p. 127
2. What are embeddings?
   Vector representations of word meaning learned directly from word distributions in texts.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, Introduction, p. 127
3. What is vector semantics?
   The field that studies embeddings and their meanings: representing word meaning as points in a semantic space derived from word distributions.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, Introduction, p. 127
4. Why are embeddings an example of representation learning?
   The useful representations of the input are learned automatically (self-supervised) from text, instead of being designed by hand through feature engineering.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, Introduction, p. 127
5. How do static embeddings differ from contextualized embeddings?
   Static embeddings (this chapter) give each word one fixed vector. Contextualized (dynamic) embeddings like BERT give a different vector for each occurrence, depending on its context.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, Introduction, p. 127