N-gram Language Models

Introduction

1. What does a language model do, formally?
   It assigns a probability to each possible next word (a probability distribution over next words), and can also assign a probability to an entire sentence or word sequence.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, Introduction, p. 68
2. Why is predicting upcoming words so important?
   Large language models are built just by training them to predict words. Word probabilities also help directly: correcting grammar or spelling (There are beats Their are), speech recognition (back soonish beats bassoon dish), and word suggestions in augmentative and alternative communication (AAC).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, Introduction, p. 68
3. What is an n-gram, and what is an n-gram model?
   An n-gram is a sequence of n words (bigram: “water of”; trigram: “water of Walden”). An n-gram model is a probabilistic model that estimates the probability of a word given the previous n−1 words, and thereby of whole sequences.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, Introduction, p. 69
4. Which language-modeling concepts does the chapter introduce through n-grams?
   Training and test sets, perplexity, sampling, and smoothing/interpolation: all of which carry over to neural LLMs.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, Introduction, p. 69