On the Biology of a Large Language Model
Lindsey et al., 2025, On the Biology of a Large Language Model (Transformer Circuits Thread)
Subscriber deck — free during early access
2 cards in “Multilingual Circuits” · whole deck
Multilingual Circuits
- How does Claude 3.5 Haiku handle “the opposite of small” in English, French and Chinese?With mostly shared, language-independent features for the operation (antonym) and the operand (small/size), plus language-specific features for detecting the input language and producing the output. Each part can be edited separately: synonym instead of antonym, “hot” instead of “small”, or another output language.
- Lindsey et al., 2025, On the Biology of a Large Language Model (Transformer Circuits Thread), Multilingual Circuits
- How does the share of language-independent features change across layers and model size?Features are most language-specific in early and late layers (reading and writing tokens) and most shared in the middle. Claude 3.5 Haiku shares more across languages than a smaller model, especially between languages with different scripts, e.g. English–Chinese. English is still somewhat privileged.
- Lindsey et al., 2025, On the Biology of a Large Language Model (Transformer Circuits Thread), Multilingual Circuits
Source
- Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, Andrew Persic, Zhenyi Qi, T. Ben Thompson, Sam Zimmerman, Kelley Rivoire, Thomas Conerly, Chris Olah and Joshua Batson. 2025. On the Biology of a Large Language Model. In Transformer Circuits Thread. https://transformer-circuits.pub/2025/attribution-graphs/biology.html