Transformers and Pretraining

Introduction

1. What kind of language modeling does the chapter introduce transformers for?
   Left-to-right language modeling (also called causal, autoregressive or decoder): given input tokens, predict output tokens one by one, conditioning on the prior context.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, Introduction, p. 177
2. Follow one token through a transformer language model.
   It is embedded (row of the embedding matrix \(E\)) plus a positional encoding, passed up through \(L\) transformer blocks (each adding attention and feedforward outputs to its residual stream), then the language modeling head (unembedding + softmax) outputs a distribution over next tokens.

   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, Introduction, pp. 177–178
3. What is the residual stream?
   The view of a token’s processing as one stream of \(d\)-dimensional vectors: the input embedding is passed straight up, and each component (attention, feedforward) adds its output to it. At any layer the stream is the sum of the original embedding and all previous outputs.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, Introduction, p. 178
4. What component distinguishes the transformer from a feedforward network, and what are two ways to think of it?
   Multi-head (self-)attention. It builds contextual representations of a token by integrating information from surrounding tokens, and it moves information from other tokens’ residual streams into the current one.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, Introduction, p. 178
5. What is the language modeling head?
   The final unembedding matrix \(U\) plus softmax that map the last block’s output embedding to a probability distribution over next tokens.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, Introduction, p. 178
6. How many blocks does a transformer stack, and what does the stack map?
   From about 12 to 96 or more blocks. Over a context window it maps input vectors \((x_1, \ldots, x_n)\) to output vectors \((h_1, \ldots, h_n)\) of the same length.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, Introduction, p. 178