learn.ifs.name

Interpretability

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 10

6 cards in “10.6 In-context learning and induction heads” · whole deck

10.6 In-context learning and induction heads

  1. What is in-context learning?
    Language models learning to do new tasks, predicting tokens better, or generally reducing their loss during the forward pass at inference time, from the prompt alone, without any gradient updates to their parameters (term from Brown et al., 2020).
  2. Why can prompting even without demonstrations be seen as learning?
    The further a model gets into a prompt, the better it tends to predict upcoming tokens: information in the context gives it more predictive power.
  3. What is a circuit in interpretability?
    An abstract functional component of a network, such as the induction head.
  4. What does an induction head do?
    It predicts repeated sequences: seeing AB … A, it predicts B (the rule AB…A → B). A prefix-matching attention component finds an earlier occurrence of the current token A, and a copying mechanism raises the probability of the token B that followed it.
    vintagecars…vintagecars?1. find the earlier “vintage”; attend to what followed2. copy “cars” as the prediction
  5. How might induction heads explain in-context learning?
    Olsson et al. (2022) propose that a fuzzy generalization, A*B* … A → B where A* ≈ A and B* ≈ B semantically, implements in-context learning from patterns in the prompt.
  6. What is ablation, and what evidence did ablating induction heads give?
    Removing (knocking out) a hypothesized cause to test whether its effect disappears. Zeroing the outputs of induction heads made models much worse at in-context learning from demonstrations (Crosbie & Shutova).

Source