Transformers and Pretraining

7.2 Transformer blocks

1. What layers does a transformer block contain?
   A multi-head self-attention layer, a feedforward layer, residual connections, and layer normalization (“layer norm”).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.2, p. 184
2. In the prenorm transformer block, what is the order of operations?
   Input → layer norm → attention, whose output is added to the input; the sum → layer norm → feedforward, whose output is added again, giving \(h_i\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.2, p. 185
3. What is the transformer feedforward layer, and what does “position-wise” mean?
   A fully connected 2-layer network (one hidden layer) applied to each token position independently. The same weights are shared across positions within a layer, but differ between layers.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.2, p. 186
4. How large is the feedforward hidden layer, and what does that imply?
   Usually \(d_{ff} \approx 4d\) (e.g. 2048 for \(d = 512\)). The feedforward networks thus hold most of a transformer’s parameters and seem to store most of its factual knowledge.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.2, p. 186
5. What is SwiGLU?
   A gated feedforward activation used by most LLMs: a component-wise product of two linear transforms of the input, one acting as the value and one, passed through the Swish function \(\text{Swish}_\beta(x) = x\,\sigma(\beta x)\), as a gate: \(\text{FFN} = (\text{Swish}(xW_1 + b_1) \otimes (xW_2 + b_2))W_3 + b_3\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.2, p. 186
6. What is layer norm, and to what is it applied?
   A normalization that keeps hidden values in a range good for gradient-based training. Despite its name it is applied to a single token’s embedding vector, not a whole layer.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.2, p. 186
7. Write the layer norm computation.
   Compute the mean \(\mu = \frac{1}{d}\sum_j x[j]\) and standard deviation \(\sigma = \sqrt{\frac{1}{d}\sum_j (x[j] - \mu)^2}\), normalize, and apply a learnable gain and offset: \(\text{LayerNorm}(x) = \gamma\frac{x - \mu}{\sigma} + \beta\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.2, pp. 186–187
8. What is RMSNorm?
   A simpler norm used by many modern networks: it rescales the vector by its root-mean-square, skipping the mean-centering step.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.2, p. 187
9. Write the equations of a prenorm transformer block for token \(i\).
   \(t^1_i = \text{LayerNorm}(x_i)\); \(t^2_i = \text{MultiHeadAttention}(t^1_i, [t^1_1, \ldots, t^1_{i-1}])\); \(t^3_i = t^2_i + x_i\); \(t^4_i = \text{LayerNorm}(t^3_i)\); \(t^5_i = \text{FFN}(t^4_i)\); \(h_i = t^5_i + t^3_i\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.2, p. 187
10. Which component of a transformer block is “token-mixing”, and why?
    Multi-head attention: it is the only component that takes input from other tokens’ residual streams, moving information from neighboring streams into the current one (in different subspaces).
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.2, p. 187
11. What does the residual stream represent at early versus late layers?
    At early blocks it mainly represents the current token; at the highest blocks it usually represents the following token, since the model is trained to predict the next token.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.2, p. 188
12. What is the difference between prenorm and postnorm transformers?
    Prenorm (most common today) applies layer norm before attention and the FFN, and needs one extra layer norm at the very top. The original postnorm transformer applied it after them; prenorm works better.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7, §7.2, p. 188