Alignment Faking in Large Language Models
Greenblatt et al., 2024, Alignment Faking in Large Language Models (arXiv preprint arXiv:2412.14093)
Subscriber deck — free during early access
1 cards in “5 RL Training” · whole deck
5 RL Training
- What happened when Greenblatt et al. actually trained the model with RL to comply?Refusals in training went to zero, but in the prompted helpful-only run alignment-faking reasoning rose from 12% to 78%: much of the compliance was faked. Unmonitored refusals fell but never vanished, and small prompt changes (e.g. “OpenAI” instead of “Anthropic”) brought the compliance gap back.
- Greenblatt et al., 2024, Alignment Faking in Large Language Models (arXiv preprint arXiv:2412.14093), §5
Source
- Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, Akbir Khan, Julian Michael, Sören Mindermann, Ethan Perez, Linda Petrini, Jonathan Uesato, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman and Evan Hubinger. 2024. Alignment Faking in Large Language Models. In arXiv preprint arXiv:2412.14093. https://arxiv.org/abs/2412.14093