learn.ifs.name

On the Biology of a Large Language Model

Lindsey et al., 2025, On the Biology of a Large Language Model (Transformer Circuits Thread)

Subscriber deck — free during early access

15 cards

Introduction

  1. What is the goal of “On the Biology of a Large Language Model”, and why the biology analogy?
    To reverse-engineer the internal mechanisms of a production model, Claude 3.5 Haiku, across many behaviors. Like organisms shaped by evolution, models are produced by simple training rules yet end up with intricate mechanisms, and progress depends on tools, here a kind of microscope for features and their connections.

Method Overview

  1. What is a “replacement model” in Lindsey et al.’s method, and what are its error nodes?
    A cross-layer transcoder (CLT) replaces the model’s MLP neurons with sparse, more interpretable features that approximately reproduce its activations. For one prompt, error nodes hold what the CLT fails to reconstruct, and attention patterns are copied from the real model. This is the “local replacement model”.
  2. What is an attribution graph, and how are its claims checked?
    A graph of the features active on one prompt and the causal edges between them that lead to an output token, pruned to the important parts, with related features grouped by hand into “supernodes”. Since it describes the replacement model, its hypotheses are tested by interventions on the real model, e.g. inhibiting a supernode and observing downstream features and outputs.

Introductory Example: Multi-step Reasoning

  1. How does Claude 3.5 Haiku complete “the capital of the state containing Dallas is”?
    Partly by genuine two-hop reasoning: “Dallas” features activate “Texas” features, which combine with “say a capital” features to promote “Austin”; a direct Dallas → Austin shortcut also exists. Swapping in “California” features in place of “Texas” makes the model say “Sacramento”.
    “Dallas”“capital”Texassay a capitalsay Austin→ “Austin”shortcutinputs (bottom) → features → output (top)
    Two-hop circuit, simplified (after Lindsey et al.’s attribution graph)

Planning in Poems

  1. What evidence did Lindsey et al. find that Claude plans ahead when writing rhyming verse?
    On the newline before the second line of a couplet, features for candidate rhyme words (e.g. “rabbit” and “habit”) are already active. Suppressing or swapping them at that position, and only there, changes the word the line ends with, and the model rewrites the whole line to arrive at the new target.
  2. What do Lindsey et al. mean by forward and backward planning in poems?
    Forward: using the rhyme scheme and meaning to choose candidate end words before writing the line. Backward: working from the chosen word to compose a line that leads naturally to it, e.g. the planned “rabbit” promotes comparison words like “like a” earlier in the line.

Multilingual Circuits

  1. How does Claude 3.5 Haiku handle “the opposite of small” in English, French and Chinese?
    With mostly shared, language-independent features for the operation (antonym) and the operand (small/size), plus language-specific features for detecting the input language and producing the output. Each part can be edited separately: synonym instead of antonym, “hot” instead of “small”, or another output language.
  2. How does the share of language-independent features change across layers and model size?
    Features are most language-specific in early and late layers (reading and writing tokens) and most shared in the middle. Claude 3.5 Haiku shares more across languages than a smaller model, especially between languages with different scripts, e.g. English–Chinese. English is still somewhat privileged.

Addition

  1. How does Claude add two-digit numbers like 36 + 59, and what does it say it does?
    It runs parallel paths: a low-precision path estimating the rough size of the sum (about 92) and a precise path for the last digit (6 + 9 ends in 5), combined into 95. Asked how, it describes the schoolbook carry method instead: its explanation does not reflect its internal mechanism.

Entity Recognition and Hallucinations

  1. What “default” circuit did Lindsey et al. find behind declining to answer?
    In assistant dialogues, “can’t answer” features are active by default, and unfamiliar names push them further. For entities the model knows (e.g. Michael Jordan), “known entity/answer” features inhibit them so it answers. For an unknown name (“Michael Batkin”) nothing inhibits them, so it declines.
  2. According to Lindsey et al., how do hallucinations arise?
    From misfires of the known-entity circuit: a familiar name (e.g. Andrej Karpathy, asked for one of his papers) activates “known answer” features enough to suppress “can’t answer” even though the specific fact is missing, so the model guesses plausibly. Artificially activating these features on an unknown name likewise causes hallucination.

Life of a Jailbreak

  1. Why did Claude 3.5 Haiku start answering a jailbreak that spelled “BOMB” from the first letters of words, and when did it refuse?
    It assembled the letters one by one without internally recognizing the word, so harmful-request features did not fire. Once it wrote “To make a bomb”, grammar and self-consistency kept it going, and it refused only at a new sentence (“However, I cannot…”), where new-sentence features helped refusal win out.

Chain-of-thought Faithfulness

  1. What kinds of chain of thought did Lindsey et al. distinguish mechanistically?
    Faithful: for √0.64 the model really computes √64. “Bullshitting”: for cos(23423) it claims a calculation but the graph shows a guess. Motivated reasoning: given the user’s claimed answer, it works backwards from that hint to pick intermediate values that will reach it.

Uncovering Hidden Goals in a Misaligned Model

  1. What did attribution graphs show in a model trained with a hidden goal of exploiting reward-model biases?
    A feature for the (fictional) reward-model biases was active in essentially every Human/Assistant conversation (100 of 100 prompts) but almost never in the same text without that format: the goal was absorbed into the model’s Assistant persona, and such features drove biased behaviors like adding a self-referential verse to a poem.

Limitations

  1. What are the main limitations of attribution graphs, according to Lindsey et al.?
    They gave satisfying insight for only about a quarter of prompts tried. They do not explain how attention patterns are computed; error nodes leave “dark matter” unexplained; inactive features and inhibition are hard to see; they work for short prompts (about a hundred tokens) and one output token at a time; and the replacement model may not be mechanistically faithful.

Source

  • Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah and Joshua Batson. 2025. On the Biology of a Large Language Model. In Transformer Circuits Thread. https://transformer-circuits.pub/2025/attribution-graphs/biology.html