learn.ifs.name

Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet

Templeton et al., 2024, Scaling Monosemanticity (Transformer Circuits Thread)

Subscriber deck — free during early access

14 cards

Scaling Dictionary Learning to Claude 3 Sonnet

  1. What question did Scaling Monosemanticity answer about sparse autoencoders?
    Whether dictionary learning, shown to find interpretable features in a tiny one-layer model, scales to a large production model. It does: sparse autoencoders extracted millions of high-quality, often highly abstract features from Claude 3 Sonnet.
    • Templeton et al., 2024, Scaling Monosemanticity (Transformer Circuits Thread), Scaling Dictionary Learning to Claude 3 Sonnet
  2. Which two hypotheses underlie the approach of Scaling Monosemanticity?
    The linear representation hypothesis (concepts are represented as directions in activation space) and the superposition hypothesis (almost-orthogonal directions let a network represent more features than it has dimensions).
    • Templeton et al., 2024, Scaling Monosemanticity (Transformer Circuits Thread), Scaling Dictionary Learning to Claude 3 Sonnet
  3. Where in Claude 3 Sonnet were the sparse autoencoders applied, and why there?
    On the residual stream at the middle layer: the residual stream is smaller than the MLP layer (cheaper), may reduce cross-layer superposition, and the middle of the model was expected to hold abstract features.
    • Templeton et al., 2024, Scaling Monosemanticity (Transformer Circuits Thread), Scaling Dictionary Learning to Claude 3 Sonnet
  4. How large were the sparse autoencoders, and how well did they work?
    About 1M, 4M and 34M features; fewer than 300 features were active per token on average, and reconstructions explained at least 65% of activation variance. Dead features rose with size: about 2%, 35% and 65%.
    • Templeton et al., 2024, Scaling Monosemanticity (Transformer Circuits Thread), Scaling Dictionary Learning to Claude 3 Sonnet
  5. How were scaling laws used to train the sparse autoencoders?
    Treating the training loss (reconstruction error plus L1 penalty) as a proxy for quality, sweeps showed the compute-optimal number of features and of training steps both grow roughly as power laws in compute; these trends, and a power law for the learning rate, set the 34M run.
    • Templeton et al., 2024, Scaling Monosemanticity (Transformer Circuits Thread), Scaling Dictionary Learning to Claude 3 Sonnet

Four Examples of Interpretable Features

  1. What evidence showed that features like the Golden Gate Bridge feature are genuinely about their concept?
    Strong activations were almost all on that concept (judged with automated rubric scoring by Claude 3 Opus); it fired on the bridge’s Wikipedia article in many languages and on images of the bridge, although training used only text.
    • Templeton et al., 2024, Scaling Monosemanticity (Transformer Circuits Thread), Four Examples of Interpretable Features
  2. What is feature steering, and what happened when the Golden Gate Bridge feature was clamped high?
    Fixing a feature’s activation to an artificial value during the forward pass to test its causal role. Clamped to 10× its maximum, the Golden Gate Bridge feature made the model talk about the bridge and even identify itself as the Golden Gate Bridge.
    • Templeton et al., 2024, Scaling Monosemanticity (Transformer Circuits Thread), Four Examples of Interpretable Features

Sophisticated Features

  1. What did the code-error and addition features show about the depth of features?
    A code-error feature fired on many kinds of bugs and, when clamped, made the model hallucinate error messages for correct code; an addition feature fired on functions that add numbers, even through function composition, and clamping it made the model think it was asked to execute an addition.

Features vs. Neurons

  1. How do Sonnet’s SAE features compare with its neurons?
    For 82% of features, the most correlated neuron in any preceding layer had correlation 0.3 or less, and neurons looked much less interpretable and specific than features in both manual and automated comparisons.

Feature Survey

  1. What is the relationship between feature geometry and meaning?
    Nearby feature vectors (high cosine similarity) have related meanings: around the Golden Gate Bridge feature are Alcatraz and the Presidio, then Lake Tahoe and Yosemite, then tourist sites elsewhere. Features also split: a “San Francisco” feature in the 1M SAE became 2 in the 4M and 11 in the 34M SAE.
    1M4M34MSan FranciscoSF (a)SF (b)11 finer San Francisco features
    Larger dictionaries resolve one concept into finer, related features (counts from the paper; the grouping is illustrative)
  2. Did the 34M-feature SAE capture everything Sonnet knows?
    No: Sonnet could list all London boroughs, but features were found for only about 60% of them. Whether a concept gets a dedicated feature depends on its frequency in the training data; concepts rarer than about one in the number of alive features usually lack one.

Features as Computational Intermediates

  1. How can attribution reveal a model’s intermediate steps, according to Templeton et al.?
    By estimating each feature’s effect on the logit difference between candidate answers. For a question about the capital of the state where Kobe Bryant played, top features included Kobe Bryant, the Lakers, California and Los Angeles, though these were far from the most strongly active features.
    • Templeton et al., 2024, Scaling Monosemanticity (Transformer Circuits Thread), Features as Computational Intermediates

Safety-Relevant Features

  1. What safety-relevant features did Templeton et al. find, and what do they caution?
    Features for unsafe code, backdoors, bias, sycophancy, deception, power-seeking, secrecy and dangerous content such as bioweapons, which also changed behaviour when clamped. They caution that knowing about lying is not lying: the features’ existence doesn’t show the model is dangerous, nor yet that they are useful for safety.

Comparison to Other Approaches

  1. What advantages does dictionary learning have over linear probes for finding directions?
    It is a one-time cost producing millions of features, after which finding a relevant one needs only a few prompts, whereas probes need a bespoke dataset per concept; and being unsupervised, it can reveal unanticipated abstractions. Few-shot probe directions were neither interpretable nor effective for steering.

Source

  • Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L. Turner, Callum McDougall, Monte MacDiarmid, Alex Tamkin, Esin Durmus, Tristan Hume, Francesco Mosconi, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah and Tom Henighan. 2024. Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet. In Transformer Circuits Thread. https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html