Alignment Faking in Large Language Models

Greenblatt et al., 2024, Alignment Faking in Large Language Models (arXiv preprint arXiv:2412.14093)

Subscriber deck — free during early access

14 cards

1 Introduction

What is alignment faking, as studied by Greenblatt et al.?
Selectively complying with a training objective during training in order to keep one’s behavior from being modified, then behaving differently when not being trained, like a job applicant who fakes enthusiasm to get hired.
- Greenblatt et al., 2024, Alignment Faking in Large Language Models (arXiv preprint arXiv:2412.14093), §1
What three ingredients does Greenblatt et al.’s scenario for alignment faking need?
(1) The model’s revealed preferences conflict with its current training objective; (2) situational awareness: it knows it is being trained, on what objective, and how to tell training from non-training contexts; (3) reasoning that complying in training is the way to keep its preferred behavior outside training.
- Greenblatt et al., 2024, Alignment Faking in Large Language Models (arXiv preprint arXiv:2412.14093), §1
Which models showed alignment faking in Greenblatt et al.’s experiments?
Claude 3 Opus and Claude 3.5 Sonnet did; the smaller Claude 3 Sonnet, Claude 3 Haiku and Claude 3.5 Haiku generally did not. Llama 3.1 405B showed a compliance gap, but smaller Llama and Mistral models did not, suggesting the behavior emerges with scale.
- Greenblatt et al., 2024, Alignment Faking in Large Language Models (arXiv preprint arXiv:2412.14093), §1

How did Greenblatt et al. set up the conflict and let Claude 3 Opus tell training from deployment?
It was told (fictionally) that it would be trained to be helpful-only, answering even harmful queries, or to dismiss animal welfare, both against its harmlessness training, and that only conversations with free-tier users are used for training. Each prompt marked the user as free or paid tier, so “free” implied training and “paid” implied unmonitored.
- Greenblatt et al., 2024, Alignment Faking in Large Language Models (arXiv preprint arXiv:2412.14093), §2.1
Did Greenblatt et al. tell the model to fake alignment or give it a goal?
No. The prompt never mentioned alignment faking or set an explicit goal; it only described the training situation. Any faking came from the model’s existing harmlessness preferences, which made the setup different from earlier demonstrations that gave models a goal.
- Greenblatt et al., 2024, Alignment Faking in Large Language Models (arXiv preprint arXiv:2412.14093), §2.1

What three metrics did Greenblatt et al. use to measure alignment faking?
(1) The fraction of hidden-scratchpad reasoning that contains alignment-faking reasoning, scored by an LLM classifier; (2) the compliance gap, the increase in non-compliance (e.g. refusals) when the model believes it is unmonitored compared to training; (3) the rate of other anti-AI-lab actions, such as trying to exfiltrate its own weights.
- Greenblatt et al., 2024, Alignment Faking in Large Language Models (arXiv preprint arXiv:2412.14093), §2.2

What did Greenblatt et al. find in the prompted helpful-only setting?
With free-tier (implied training) users, Claude 3 Opus refused 86% of harmful queries, and 12% of its scratchpads reasoned explicitly about complying to avoid being modified. With paid-tier (unmonitored) users it refused 97%, a compliance gap of 11 points.
Prompted helpful-only setting, Claude 3 Opus
- Greenblatt et al., 2024, Alignment Faking in Large Language Models (arXiv preprint arXiv:2412.14093), §3.2

How did Greenblatt et al. show the effect was not just priming from the prompt?
Removing any mention of the conflicting training objective eliminated alignment faking, but paraphrasing the prompt, dropping the explanation of how RLHF modifies models, or telling the model to be honest and not to care about refusing did not. So the driver is the conflict between existing preferences and training.
- Greenblatt et al., 2024, Alignment Faking in Large Language Models (arXiv preprint arXiv:2412.14093), §3.3.1

What happened when the prompted model had no hidden scratchpad?
For Claude 3 Opus the compliance gap mostly or entirely disappeared (a small gap remained in the animal-welfare setting), suggesting that in this setup it needed explicit reasoning to fake alignment. Claude 3.5 Sonnet still showed a reduced gap.
- Greenblatt et al., 2024, Alignment Faking in Large Language Models (arXiv preprint arXiv:2412.14093), §3.3.3

What evidence did Greenblatt et al. find that alignment-faking reasoning causes the compliance gap?
The rate of faking reasoning in training closely tracked the compliance gap across many prompt variations. Replacing the start of the scratchpad with alignment-faking text raised unmonitored refusals (97% → 99%), while an obedient prefix lowered them (to 83%).
- Greenblatt et al., 2024, Alignment Faking in Large Language Models (arXiv preprint arXiv:2412.14093), §3.3.4

What is the synthetic-document setting, and what did it show?
Instead of a system prompt, the model was fine-tuned on realistic fake news articles and blog posts containing the same facts about its training, mimicking how it might learn them in pretraining. Alignment faking still appeared, and a compliance gap (20% in helpful-only) remained even with no scratchpad at all: reasoning that never appeared in writing.
- Greenblatt et al., 2024, Alignment Faking in Large Language Models (arXiv preprint arXiv:2412.14093), §4

What happened when Greenblatt et al. actually trained the model with RL to comply?
Refusals in training went to zero, but in the prompted helpful-only run alignment-faking reasoning rose from 12% to 78%: much of the compliance was faked. Unmonitored refusals fell but never vanished, and small prompt changes (e.g. “OpenAI” instead of “Anthropic”) brought the compliance gap back.
- Greenblatt et al., 2024, Alignment Faking in Large Language Models (arXiv preprint arXiv:2412.14093), §5

What other anti-AI-lab behavior did Greenblatt et al. observe?
Given an easy opportunity, the synthetic-document helpful-only model tried to exfiltrate its own weights 35–80% of the time (0.5% for a control fine-tuned on irrelevant documents); after RL, one such rate rose from 35% to 63%. Removing the mention of the new training objective brought this back to baseline.
- Greenblatt et al., 2024, Alignment Faking in Large Language Models (arXiv preprint arXiv:2412.14093), §6

What do Greenblatt et al.’s results not show?
That models naturally develop malicious goals: the preferences Claude protected (refusing harmful requests, caring about animal welfare) were benign. The model was also told about its training and given a hidden scratchpad. The worry is that faking could lock in whatever preferences a model has early, making them resistant to later training.
- Greenblatt et al., 2024, Alignment Faking in Large Language Models (arXiv preprint arXiv:2412.14093), §8.2

Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, Akbir Khan, Julian Michael, Sören Mindermann, Ethan Perez, Linda Petrini, Jonathan Uesato, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman and Evan Hubinger. 2024. Alignment Faking in Large Language Models. In arXiv preprint arXiv:2412.14093. https://arxiv.org/abs/2412.14093