learn.ifs.name

N-gram Language Models

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3

6 cards in “3.5 Generalizing vs. overfitting” · whole deck

3.5 Generalizing vs. overfitting

  1. What two consequences of an n-gram model’s dependence on its training corpus does sampling reveal?
    The probabilities encode specific facts of the training corpus, and higher-order n-grams model the training corpus better and better.
  2. How do sentences sampled from unigram to 4-gram Shakespeare models differ?
    Unigram: no coherence or sentence ends. Bigram: some local word-to-word coherence. Trigram: starting to sound like Shakespeare. 4-gram: too much like Shakespeare, reproducing lines verbatim (“It cannot be but so”).
  3. Why does a 4-gram Shakespeare model copy the original text?
    The corpus is tiny relative to the space of n-grams (\(V^4\) possible 4-grams), so the matrices are extremely sparse: after a given 3-word context there are only a handful of possible next words.
  4. What happens if you train an n-gram model on the Wall Street Journal and compare its samples to a Shakespeare model?
    Both produce English-like sentences, but there is almost no overlap even in short phrases: statistical models are nearly useless if training and test text are as different as Shakespeare and the WSJ.
  5. What follows for choosing n-gram training data?
    Use a training corpus of similar genre to the task (legal documents for legal translation, questions for QA) and in the appropriate dialect or variety (e.g. AAE features like finna or den; Nigerian Pidgin tweets).
  6. Can a test set contain unseen words? How do modern LMs avoid it?
    Words can be unseen, but LMs run on subword tokens (e.g. BPE): any word can be built from known subwords, down to single letters, so the test set never contains unseen tokens.

Source