Interpretability

10.3 The logit lens

1. What is the logit lens?
   A visualization trick: take a vector from any intermediate layer, pretend it is the final embedding, multiply it by the unembedding matrix and apply softmax, to see which words it might represent.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 10, §10.3, p. 249
2. Why doesn’t the logit lens always work well?
   The network wasn’t trained to make internal layers interpretable this way; it implicitly assumes earlier layers live in the same embedding space as the final layer.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 10, §10.3, p. 249
3. What is the Jacobian lens (J-lens), and how does it differ from the logit lens?
   A variant that uses the derivatives of the final layer’s output with respect to an intermediate layer: how a small change there would alter the final output, averaged over many contexts. It acts like a translator through an approximation of the rest of the network, and works better at earlier layers.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 10, §10.3, p. 249
4. What have lens analyses suggested about how models reason internally?
   A kind of layer-by-layer workspace: for “the color of the planet fourth from the sun”, layers first encode color, then resolve Mars, then red. For multilingual prompts, meanings seem represented mostly in English and translated into the output language only in later layers.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 10, §10.3, pp. 249–250