N-gram Language Models

3.2 Training and test sets

1. What is the difference between extrinsic and intrinsic evaluation?
   Extrinsic (end-to-end) evaluation embeds the model in an application and measures how much the application improves: the only real proof. Intrinsic evaluation measures model quality independently of any application (e.g. perplexity): cheaper and faster.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.2, p. 74
2. What three datasets do we need to evaluate a machine learning model?
   A training set (to learn parameters), a development set (devset, for testing during development), and a test set (held out for final evaluation).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.2, p. 74
3. Why do we need a separate, held-out test set?
   To get an unbiased estimate of how well the model generalizes to new data; a model that fits the training data perfectly may perform terribly elsewhere.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.2, p. 75
4. How should a test set be chosen?
   It should reflect the language the model will be used for (chemistry lectures, hotel booking requests…). For a general-purpose model, draw it from a wide variety of texts, not one document or author.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.2, p. 75
5. How do we decide which of two language models better “fits” a test set?
   The one that assigns the higher probability to the test set, i.e. predicts it more accurately.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.2, p. 75
6. What is training on the test set (data contamination), and what does it do?
   Letting test sentences into the training data. They get artificially high probabilities, making perplexity look much better than it really is.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.2, p. 75
7. Why do we need a devset in addition to the test set?
   Testing repeatedly on the test set while making changes implicitly tunes the model to it. So we do all testing on the devset and touch the test set only once, at the end.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.2, p. 75
8. How large should the test set be, and what must the devset resemble?
   Large enough to have the statistical power to detect a significant difference between models (but leaving as much as possible for training). The devset must be the same kind of text as the test set.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.2, pp. 75–76