N-gram Language Models

3.3 Perplexity

1. Why don’t we use raw test-set probability as the LM evaluation metric?
   It depends on length: the longer the text, the smaller its probability. We want a per-word metric normalized by length.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.3, p. 76
2. Define perplexity.
   The inverse probability of the test set, normalized by the number of words: \(\text{PP}(W) = P(w_1 \ldots w_N)^{-1/N} = \sqrt[N]{\frac{1}{P(w_1 \ldots w_N)}}\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.3, p. 76
3. Is lower or higher perplexity better?
   Lower. Because of the inverse, higher test-set probability means lower perplexity; minimizing perplexity is maximizing test-set probability.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.3, p. 76
4. Write perplexity for a unigram and for a bigram model.
   Unigram: \(\sqrt[N]{\prod_{i=1}^{N} \frac{1}{P(w_i)}}\). Bigram: \(\sqrt[N]{\prod_{i=1}^{N} \frac{1}{P(w_i \mid w_{i-1})}}\). Both are geometric means of inverse probabilities.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.3, p. 76
5. When computing perplexity over a test set with sentence markers, which markers count toward \(N\)?
   Include </s> but not <s>: the end token is followed by a start token with probability ≈ 1, so that fake transition shouldn’t affect perplexity.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.3, p. 77
6. How does perplexity change from unigram to bigram to trigram models, and why?
   It drops sharply: longer contexts give more information about the next word, so the model is less surprised and assigns higher probability to the test text.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.3, p. 77
7. Under what conditions are two perplexities comparable?
   The models must be built without knowledge of the test set, and must use identical vocabularies.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.3, p. 77
8. Does a better perplexity guarantee better task performance?
   No. It usually correlates with extrinsic improvements, so it is a convenient metric, but gains should be confirmed with end-to-end evaluation on a real task.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.3, p. 77
9. How can perplexity be seen as a weighted average branching factor?
   The branching factor is the number of possible next words. A uniform LM over 3 colors has perplexity 3 on any test set. If one color is very likely (P(red) = .8), the branching factor is still 3 but the perplexity on mostly-red text drops (≈ 1.89): it is a probability-weighted branching factor.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.3, pp. 77–78
10. Compute the perplexity of a uniform LM over {red, blue, green} on the test set “red red red red blue”.
    \(\big((1/3)^5\big)^{-1/5} = 3\).
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.3, p. 78