Interpretability

10.2 Probing

1. What is probing?
   Testing whether a representation encodes some information (e.g. that a word is a noun, or the tense of sublet) by training a simple classifier (logistic regression or a small MLP) on embeddings from a chosen position and layer to predict that property.

   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 10, §10.2, pp. 247–248
2. What have probing studies suggested about where information lives across transformer layers?
   Grammatical information about tokens seems encoded at lower layers, semantic information at middle layers, and discourse information like coreference higher still (Tenney et al., 2019).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 10, §10.2, p. 248
3. What are the steps of training a probe?
   (1) Choose a property; (2) create train and test sets of embeddings from the position/layer of interest, labeled with the property; (3) pass the sentences through the LLM and extract activations; (4) freeze the LLM and train only the classifier.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 10, §10.2, p. 248
4. What controls and constraints does good probing need?
   Comparison to control conditions (e.g. probing with random labels), and a probe that isn’t too powerful, so it doesn’t succeed by learning the information itself rather than reading it from the representation.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 10, §10.2, p. 248
5. What can’t a successful probe tell you?
   That the model uses the information: the property may be correlated with the embedding without having any causal effect on the model’s behavior.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 10, §10.2, p. 248