Transformers and Pretraining

7.5 The language modeling head

1. What does “head” mean in “language modeling head”?
   Additional neural circuitry added on top of the basic transformer to apply it to a task; the LM head is the circuitry needed for language modeling.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.5, pp. 193–194
2. How large is the context of a transformer LM compared with an n-gram model?
   An n-gram model conditions on \(n-1\) words; a transformer on its whole context window, which can be hundreds of thousands of tokens or more.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.5, p. 194
3. What does the LM head compute?
   From the last layer’s output for token \(N\), \(h^L_N\) \([1 \times d]\): logits \(u = h^L_N E^\top\) of shape \([1 \times |V|]\), then probabilities \(y = \text{softmax}(u)\). In training it predicts at every position, in inference just at the last.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.5, p. 194
4. What is weight tying, and why is \(E^\top\) called the unembedding?
   Using the same weights for two matrices: the embedding matrix \(E\) maps one-hot tokens to embeddings at the input, and its transpose \(E^\top\) maps the final embedding back to scores over the vocabulary. Training makes \(E\) good at both mappings.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.5, pp. 194–195
5. Why are causal transformer LMs called decoder-only models?
   They correspond roughly to the decoder half of the original encoder-decoder transformer (built for machine translation); the causal LM paradigm later used only that part.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.5, pp. 195–196