learn.ifs.name

Towards Monosemanticity: Decomposing Language Models With Dictionary Learning

Bricken et al., 2023, Towards Monosemanticity (Transformer Circuits Thread)

Subscriber deck — free during early access

15 cards

Problem Setup

  1. What problem does Towards Monosemanticity address?
    Neurons in language models are often polysemantic (they respond to several unrelated things), so they are poor units for reverse-engineering a model. The paper decomposes MLP activations into many more interpretable “features” than there are neurons.
  2. What model and what layer did Bricken et al. study?
    A one-layer transformer with a 512-neuron ReLU MLP trained on the Pile, “the simplest language model we profoundly don’t understand”: one-layer models are cheap to overtrain and their features’ effects on the output logits are nearly linear and easy to read.

Features as a Decomposition

  1. How is an activation vector decomposed into features?
    \(\mathbf{x}^j \approx \mathbf{b} + \sum_i f_i(\mathbf{x}^j)\, \mathbf{d}_i\): a bias plus a sparse weighted sum of feature directions \(\mathbf{d}_i\) (unit vectors, the decoder’s columns), with feature activations \(f_i(\mathbf{x}) = \mathrm{ReLU}(W_e(\mathbf{x} - \mathbf{b}_d) + \mathbf{b}_e)_i\) from the encoder.
    MLP512recon.5124,096 featuresReLUlinear
    Few features are active at once (highlighted); loss = reconstruction error + L1 on the features

Superposition Hypothesis

  1. Why should the dictionary have more features than neurons?
    The superposition hypothesis: networks represent more features than they have neurons by exploiting sparsity, approximately simulating a much larger, sparser network. So feature directions form an overcomplete basis with sparse activations, the classic dictionary-learning problem.

Why not use architectural approaches?

  1. Why don’t Bricken et al. simply train models without superposition?
    Even with extreme activation sparsity (one-hot activations) neurons stayed polysemantic: under cross-entropy loss a neuron covering two features ambiguously (loss \(\ln 2 \approx 0.7\) in a toy case) beats one covering a single feature cleanly (\(\tfrac34 \ln 3 \approx 0.8\)). Polysemanticity can arise without superposition.

Using Sparse Autoencoders To Find Good Decompositions

  1. Why use a sparse autoencoder rather than classic dictionary-learning algorithms?
    It scales to huge datasets (8 billion activation vectors here), and it has roughly the power of an MLP layer, whereas exact sparse recovery (compressed sensing) is NP-hard and might find “features” the model itself cannot use.
    • Bricken et al., 2023, Towards Monosemanticity (Transformer Circuits Thread), Using Sparse Autoencoders To Find Good Decompositions

Sparse Autoencoder Setup

  1. How was the sparse autoencoder trained, and what practical tricks mattered?
    To reconstruct MLP activations with an MSE loss plus an L1 penalty on feature activations, using Adam. Lots of data made features sharper, and periodically resampling “dead” features (ones that stopped firing) to fit poorly reconstructed examples let more features be learned.

Detailed Investigations of Individual Features

  1. What five claims did Bricken et al. try to establish for features such as Arabic script, DNA, base64 and Hebrew?
    The feature fires specifically on its context; it fires whenever the context is present (sensitivity); it causes appropriate downstream behaviour; it doesn’t correspond to any single neuron; and a similar feature appears when dictionary learning is applied to a different model (universality).
    • Bricken et al., 2023, Towards Monosemanticity (Transformer Circuits Thread), Detailed Investigations of Individual Features
  2. Why is the Arabic-script feature not visible in the neuron basis?
    Its direction is spread over many neurons: its three largest neuron coefficients are all negative and 27 neurons have coefficients of at least 0.1, so no individual neuron tracks Arabic text.
    • Bricken et al., 2023, Towards Monosemanticity (Transformer Circuits Thread), Detailed Investigations of Individual Features

How Interpretable is the Typical Feature?

  1. How did the interpretability of features compare with neurons?
    A blinded human rater gave the median neuron 0 on the rubric (no hypothesis at all) and the median feature interval 12 (confident, consistent, matching the logits). Automated interpretability with Claude agreed: predicted activations correlated far better for features, and logit predictions were 74% vs 58% accurate.
    • Bricken et al., 2023, Towards Monosemanticity (Transformer Circuits Thread), How Interpretable is the Typical Feature?

How much of the model does our interpretation explain?

  1. How much of the MLP’s contribution do the features capture?
    Replacing the MLP activations with the 4,096-feature autoencoder’s reconstruction recovers 79% of the loss reduction the MLP provides; a 131,072-feature dictionary recovers 94.5%. The authors warn that a long tail of features likely remains.
    • Bricken et al., 2023, Towards Monosemanticity (Transformer Circuits Thread), How much of the model does our interpretation explain?

Feature Splitting

  1. What is feature splitting?
    As the dictionary grows (512 to over 131,000 features), one feature becomes a family of finer ones, e.g. one base64 feature splits into three with subtler roles. Different dictionary sizes give different “resolutions” of the same underlying features.

Universality

  1. How universal were the learned features across two separately trained transformers?
    Features from model A had a median activation correlation of 0.72 with their closest feature in model B, far more than the closest neurons (0.46): features are more similar across models than to their own model’s neurons.

"Finite State Automata"

  1. What are feature “finite-state automata”?
    Chains in which one feature raises the probability of certain tokens, which then make another feature fire at the next step, and so on; such assemblies implement behaviours like producing valid HTML, although the model didn’t learn them as deliberate circuits.

Future Work

  1. What is the main obstacle to applying this method to frontier models?
    Cost: a 100× expansion of one 10,000-wide MLP layer needs an autoencoder of about 20 billion parameters, possibly trained on much of the model’s corpus to catch rare features; better metrics for judging feature quality are also missing.

Source

  • Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nicholas L. Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E. Burke, Tristan Hume, Shan Carter, Tom Henighan and Chris Olah. 2023. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. In Transformer Circuits Thread. https://transformer-circuits.pub/2023/monosemantic-features/index.html