N-gram Language Models

3.4 Sampling sentences from a language model

1. What does it mean to sample from a language model, and why do it?
   Generate sentences by choosing them according to their probability under the model. It visualizes what knowledge the model embodies (a technique going back to Shannon and to Miller & Selfridge).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.4, p. 78
2. How do you sample words from a unigram model?
   Lay out all words on the interval [0, 1], each covering a length equal to its probability. Pick a random number, output the word whose interval contains it, and repeat until </s> is generated.

   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.4, pp. 78–79
3. How do you sample a sentence from a bigram model?
   Draw a random bigram starting with <s> according to its probability; say its second word is \(w\). Then draw a bigram starting with \(w\), and so on until </s>.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 3, §3.4, p. 79