On the Biology of a Large Language Model
Lindsey et al., 2025, On the Biology of a Large Language Model (Transformer Circuits Thread)
Subscriber deck — free during early access
2 cards in “Planning in Poems” · whole deck
Planning in Poems
- What evidence did Lindsey et al. find that Claude plans ahead when writing rhyming verse?On the newline before the second line of a couplet, features for candidate rhyme words (e.g. “rabbit” and “habit”) are already active. Suppressing or swapping them at that position, and only there, changes the word the line ends with, and the model rewrites the whole line to arrive at the new target.
- Lindsey et al., 2025, On the Biology of a Large Language Model (Transformer Circuits Thread), Planning in Poems
- What do Lindsey et al. mean by forward and backward planning in poems?Forward: using the rhyme scheme and meaning to choose candidate end words before writing the line. Backward: working from the chosen word to compose a line that leads naturally to it, e.g. the planned “rabbit” promotes comparison words like “like a” earlier in the line.
- Lindsey et al., 2025, On the Biology of a Large Language Model (Transformer Circuits Thread), Planning in Poems
Source
- Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah and Joshua Batson. 2025. On the Biology of a Large Language Model. In Transformer Circuits Thread. https://transformer-circuits.pub/2025/attribution-graphs/biology.html