Interpretability

Research into methods for understanding how language models do what they do: uncovering their internal representations (what meanings they encode) and their reasoning paths, e.g. describing specific “circuits”.

Static embeddings give one vector per word type; contextual embeddings give a different vector each time a word appears in a different context.

The output of each component (attention heads, feedforward network) added to the residual stream, at every layer and every token, is a contextual embedding. Commonly the final layer’s \(h^L_i\) is used as the meaning of token \(x_i\) in context.

By mean- or max-pooling the token’s output vectors from the last four layers, rather than using the final layer alone.

Word instances: a particular occurrence of a word type in a particular context.

It has multiple senses: mouse (rodent / cursor device), bank (financial institution / sloping mound).

A discrete representation of one aspect of a word’s meaning (bank¹, bank²), listed in thesauruses like WordNet, which exists for many languages.

Sometimes polysemy means only senses related in a structured way (“structured polysemy”), with homonymy for ambiguity between unrelated senses. The book uses polysemy for any sense ambiguity.

Instances cluster by meaning: separate clusters for die as singular of dice, the verb to die, and the German article die.

WordNet gives discrete lists of senses; embeddings offer a continuous high-dimensional model of meaning that can be clustered but doesn’t divide into fully discrete senses.

Selecting the correct sense of a word in context, given a fixed inventory of possible senses (e.g. from WordNet).

A word tends to be used in roughly the same sense throughout a discourse.

1-nearest-neighbor with contextual embeddings: for each sense in a sense-labeled corpus (e.g. SemCor), average the contextual embeddings of its tokens into a sense embedding \(v_s\). At test time, embed the target token \(t\) and choose \(\arg\max_s \cos(t, v_s)\).

By the cosine between the contextual embeddings of the two word instances (after some transformation of the embeddings).

All vectors tend to point in the same direction, so even random word pairs have cosines close to 1 (anisotropy = the expected cosine between random pairs). This makes raw cosines uninformative.

A few “rogue dimensions” with very large magnitudes and variance dominate the cosine. Standardizing (z-scoring) each dimension over a corpus, \(z = \frac{x - \mu}{\sigma}\), makes embeddings more isotropic.

Cosine tends to underestimate human similarity judgments for very frequent words.

Testing whether a representation encodes some information (e.g. that a word is a noun, or the tense of sublet) by training a simple classifier (logistic regression or a small MLP) on embeddings from a chosen position and layer to predict that property.

Grammatical information about tokens seems encoded at lower layers, semantic information at middle layers, and discourse information like coreference higher still (Tenney et al., 2019).

(1) Choose a property; (2) create train and test sets of embeddings from the position/layer of interest, labeled with the property; (3) pass the sentences through the LLM and extract activations; (4) freeze the LLM and train only the classifier.

Comparison to control conditions (e.g. probing with random labels), and a probe that isn’t too powerful, so it doesn’t succeed by learning the information itself rather than reading it from the representation.

That the model uses the information: the property may be correlated with the embedding without having any causal effect on the model’s behavior.

A visualization trick: take a vector from any intermediate layer, pretend it is the final embedding, multiply it by the unembedding matrix and apply softmax, to see which words it might represent.

The network wasn’t trained to make internal layers interpretable this way; it implicitly assumes earlier layers live in the same embedding space as the final layer.

A variant that uses the derivatives of the final layer’s output with respect to an intermediate layer: how a small change there would alter the final output, averaged over many contexts. It acts like a translator through an approximation of the rest of the network, and works better at earlier layers.

A kind of layer-by-layer workspace: for “the color of the planet fourth from the sun”, layers first encode color, then resolve Mars, then red. For multilingual prompts, meanings seem represented mostly in English and translated into the output language only in later layers.

Language models learning to do new tasks, predicting tokens better, or generally reducing their loss during the forward pass at inference time, from the prompt alone, without any gradient updates to their parameters (term from Brown et al., 2020).

The further a model gets into a prompt, the better it tends to predict upcoming tokens: information in the context gives it more predictive power.

An abstract functional component of a network, such as the induction head.

It predicts repeated sequences: seeing AB … A, it predicts B (the rule AB…A → B). A prefix-matching attention component finds an earlier occurrence of the current token A, and a copying mechanism raises the probability of the token B that followed it.

Olsson et al. (2022) propose that a fuzzy generalization, A*B* … A → B where A* ≈ A and B* ≈ B semantically, implements in-context learning from patterns in the prompt.

Removing (knocking out) a hypothesized cause to test whether its effect disappears. Zeroing the outputs of induction heads made models much worse at in-context learning from demonstrations (Crosbie & Shutova).