Transformers and Pretraining

7.6 Decoding

1. What is decoding?
   Choosing which token to generate based on the model’s probabilities. Repeatedly choosing the next token conditioned on previous choices, left to right, is causal or autoregressive generation.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.6, p. 196
2. What is greedy decoding?
   Always generating the most probable token: \(\hat{w}_t = \arg\max_{w \in V} P(w \mid w_{<t})\). A greedy algorithm makes locally optimal choices.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.6, p. 196
3. Why isn’t greedy decoding used with large language models?
   Its choices are extremely predictable, so the text is generic and repetitive; it is even deterministic (same context → same output). An extension, beam search, works for constrained tasks like MT.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.6, p. 197
4. What is random (multinomial) sampling?
   Repeatedly choosing the next token at random according to its probability under the model, conditioned on previous choices, until the end-of-sequence token: \(w_i \sim p(w_i \mid w_{<i})\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.6, p. 197
5. Why doesn’t plain random sampling work well?
   The long tail of odd, low-probability tokens adds up to a non-trivial share of the probability mass, so weird tokens get chosen often enough to produce strange sentences.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.6, p. 198
6. What trade-off do temperature, top-k and top-p sampling control?
   Quality vs diversity: emphasizing the most probable words gives more accurate, coherent, factual but boring and repetitive text; giving more weight to middle-probability words gives more creative, diverse but less factual, less coherent text.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.6, p. 198
7. How does temperature sampling work?
   Divide the logits by a temperature \(\tau\) before the softmax: \(y = \text{softmax}(u/\tau)\). With \(\tau < 1\), high-probability tokens become more likely and rare ones less likely; as \(\tau \to 0\) it approaches greedy decoding; \(\tau > 1\) flattens the distribution toward uniform.

   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.6, pp. 198–199
8. Why does dividing logits by \(\tau < 1\) make the distribution greedier?
   It makes the scores larger in magnitude, and the softmax pushes larger differences further toward 1 for the top value and 0 for the rest.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.6, p. 199
9. Where does the name “temperature” come from?
   Thermodynamics: a system at high temperature is flexible and explores many states; at low temperature it stays in a subset of low-energy (better) states.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.6, p. 199
10. Describe top-k sampling.
    Keep only the \(k\) most probable tokens, renormalize their probabilities, and sample from them. \(k = 1\) is greedy decoding; larger \(k\) gives more diverse but still good text.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.6, p. 200
11. What is the weakness of top-k sampling that top-p (nucleus) sampling addresses?
    A fixed \(k\) ignores the shape of the distribution: sometimes the top 10 words hold almost all the mass, sometimes only a small part. Top-p keeps the smallest set of top words whose probabilities sum to at least \(p\), so the pool grows and shrinks with the model’s confidence.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.6, p. 200
12. How are top-k and top-p often combined?
    Top-k first as a hard filter to eliminate garbage, then top-p to adapt to the model’s confidence.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.6, p. 201