Embeddings

5.9 Evaluating vector models

1. What is the most important evaluation of vector models, and what is the main intrinsic one?
   Extrinsic: whether the vectors improve performance on an NLP task. Intrinsic: the correlation between the model’s word similarity scores (e.g. cosine) and human similarity ratings.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, §5.9, p. 147
2. What datasets exist for intrinsic evaluation of word embeddings?
   Out of context: WordSim-353, SimLex-999 (similarity rather than relatedness), TOEFL synonym questions. In context: SCWS (Stanford Contextual Word Similarity) and WiC (Word-in-Context). Sentence level: semantic textual similarity (STS). Plus analogy datasets (morphological, lexicographic, encyclopedic relations, SemEval-2012 Task 2).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, §5.9, p. 147
3. Why must one be careful when using embeddings to study word associations in a particular corpus?
   Embedding algorithms are inherently variable (random initialization, random negative sampling), and individual documents can strongly affect results. Best practice: train multiple embeddings with bootstrap sampling over documents and average.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 5, §5.9, p. 147