Transformers and Pretraining

7.8 Current models

1. Name architectural refinements used in current transformer LLMs to improve accuracy.
   Rotary position embeddings (RoPE), RMSNorm, and gated feedforward layers with SwiGLU.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.8, p. 204
2. What is the KV cache?
   An inference-time optimization: since tokens are generated one at a time, the key and value vectors of all previous tokens are stored in memory and reused, so only the new token’s query, key and value must be computed.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.8, p. 204
3. Why can’t inference use the same fully parallel attention computation as training?
   At inference, tokens are generated iteratively one by one; the future tokens don’t exist yet, so each new token’s attention must be computed when it is produced.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.8, p. 204
4. Name some other efficiency-focused innovations in current LLMs.
   Mixture-of-Experts models, Grouped-Query Attention (GQA), and quantization.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.8, p. 204
5. What parameters describe the scale of a transformer LM configuration?
   Number of layers \(L\), model dimension \(d\), number of attention heads \(A\), feedforward dimension \(d_{ff}\), vocabulary size \(|V|\), context length, and total parameter count.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.8, pp. 204–205