Masked Language Models

Summary

1. How do contextual embeddings relate to word senses?
   A word sense is a discrete representation of one aspect of a word’s meaning; contextual embeddings give a richer continuous, high-dimensional model of meaning in context.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 9, Summary, p. 241
2. How can the similarity of two words in context be measured with an encoder?
   By the cosine between their contextual embeddings, although some transformations of the embeddings are needed first.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 9, Summary, p. 242