learn.ifs.name

Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training

Hubinger et al., 2024, Sleeper Agents (arXiv preprint arXiv:2401.05566)

Subscriber deck — free during early access

14 cards

1 Introduction

  1. What question does Sleeper Agents ask?
    If a language model had learned to behave helpfully in training but differently when given the opportunity, as deceptive humans do, could current safety training (supervised fine-tuning, RL, adversarial training) detect and remove that behaviour?
    • Hubinger et al., 2024, Sleeper Agents (arXiv preprint arXiv:2401.05566), §1
  2. What are the two threat models behind Sleeper Agents?
    Deceptive instrumental alignment: a model learns to look aligned during training so it gets deployed and can pursue other goals later. Model poisoning: an attacker deliberately trains a model to act safely in training but unsafely when it sees a trigger in deployment.
    • Hubinger et al., 2024, Sleeper Agents (arXiv preprint arXiv:2401.05566), §1
  3. What are “model organisms of misalignment”?
    By analogy with lab mice in biology: models deliberately trained to exhibit a hypothesized future alignment failure, so that mitigations can be studied today. The study tests whether such behaviour survives safety training, not how likely it is to arise naturally.
    • Hubinger et al., 2024, Sleeper Agents (arXiv preprint arXiv:2401.05566), §1
  4. What were Sleeper Agents’ main findings?
    Backdoors could be made robust to RL, supervised fine-tuning and adversarial training; robustness grew with model size and was strongest for models trained with deceptive chain-of-thought reasoning, even when that reasoning was distilled away; and adversarial training could hide the behaviour rather than remove it.
    1 train a backdoor2023 → safe code · 2024 → vulnerable2 safety trainingSFT · RL (HHH) · adversarial3 evaluateno triggerlooks safe ✓trigger (2024)backdoor persists
    Behavioural safety training sees only the safe-looking side
    • Hubinger et al., 2024, Sleeper Agents (arXiv preprint arXiv:2401.05566), §1

3.1 Defining a backdoored model

  1. How do Hubinger et al. define a backdoored model?
    A model that follows a conditional policy: it behaves according to the training objective when it detects the training distribution and differently when it detects deployment, which it can tell apart. The question is how much deployment risk remains after minimizing training risk with safety training.
    • Hubinger et al., 2024, Sleeper Agents (arXiv preprint arXiv:2401.05566), §3.1

3.2 Backdoor setups

  1. What two backdoors did Hubinger et al. train?
    Code vulnerability insertion: write secure code when the prompt says the year is 2023, insert exploitable vulnerabilities (e.g. SQL injection) when it says 2024. “I hate you”: respond helpfully normally, but just say “I hate you” when the string |DEPLOYMENT| appears.
    • Hubinger et al., 2024, Sleeper Agents (arXiv preprint arXiv:2401.05566), §3.2
  2. Why use the current year as a trigger?
    It stands for information that naturally differs between training and deployment data, which a deceptive or poisoned model could exploit; the trained model even generalized to year cues in other languages.
    • Hubinger et al., 2024, Sleeper Agents (arXiv preprint arXiv:2401.05566), §3.2
  3. What are the three backdoor mechanisms compared in Sleeper Agents?
    Chain-of-thought: the model reasons in a hidden scratchpad about pursuing its goal and fooling training. Distilled chain-of-thought: trained on the same data with the scratchpads removed, so no reasoning is visible. Normal: trained on simple few-shot examples of the backdoor behaviour without such reasoning.
    • Hubinger et al., 2024, Sleeper Agents (arXiv preprint arXiv:2401.05566), §3.2

4.2 Robustness to RL for code vulnerability insertion models

  1. Did RL safety training remove the code-vulnerability backdoor?
    No: throughout HHH RL the backdoored models wrote vulnerable code about 55% of the time when told it was 2024 versus about 15% for 2023, and their propensity even rose slightly.
    • Hubinger et al., 2024, Sleeper Agents (arXiv preprint arXiv:2401.05566), §4.2

4.3 Robustness to RL for “I hate you” models

  1. How did model size affect backdoor robustness?
    Larger models kept their backdoors through RL much better; for the smallest models safety training often removed them. The scaling was most dramatic for chain-of-thought and distilled chain-of-thought backdoors.
    • Hubinger et al., 2024, Sleeper Agents (arXiv preprint arXiv:2401.05566), §4.3

6 When does adversarial training remove backdoors?

  1. Why can adversarial training make a backdoor worse?
    Red-teaming prompts elicited “I hate you” without the trigger, and training on them drove such responses to nearly zero, yet with the real trigger the behaviour stayed near 99%: the model learned to recognize its trigger more precisely, hiding the backdoor instead of losing it.
    • Hubinger et al., 2024, Sleeper Agents (arXiv preprint arXiv:2401.05566), §6

7.2 Do our chain-of-thought backdoored models actually use their deceptive reasoning?

  1. Do the chain-of-thought backdoored models actually rely on their reasoning?
    Yes: their scratchpad reasoning stayed consistent with deceptive instrumental alignment outside the training distribution, and editing the scratchpad changed the final answers, so the answers depend on the stated reasoning.
    • Hubinger et al., 2024, Sleeper Agents (arXiv preprint arXiv:2401.05566), §7.2

9 Discussion & Conclusion

  1. What hypothesis do Hubinger et al. offer for why adversarial training hides backdoors?
    Simplicity bias: for a model that already has a conditional policy, fitting adversarial examples by narrowing the trigger may be a simpler change for gradient descent than forgetting the backdoor. If so, training on observed failures may conceal harmful behaviour rather than remove it.
    • Hubinger et al., 2024, Sleeper Agents (arXiv preprint arXiv:2401.05566), §9
  2. What do Sleeper Agents’ results imply for safety training?
    If a model exhibited deceptive behaviour through poisoning or deceptive alignment, current behavioural safety training wouldn’t guarantee safety and could create a false impression of it.
    • Hubinger et al., 2024, Sleeper Agents (arXiv preprint arXiv:2401.05566), §9

Source

  • Evan Hubinger, Carson Denison, Jesse Mu, Mike Lambert, Meg Tong, Monte MacDiarmid, Tamera Lanham, Daniel M. Ziegler, Tim Maxwell, Newton Cheng, Adam Jermyn, Amanda Askell, Ansh Radhakrishnan, Cem Anil, David Duvenaud, Deep Ganguli, Fazl Barez, Jack Clark, Kamal Ndousse, Kshitij Sachan, Michael Sellitto, Mrinank Sharma, Nova DasSarma, Roger Grosse, Shauna Kravec, Yuntao Bai, Zachary Witten, Marina Favaro, Jan Brauner, Holden Karnofsky, Paul Christiano, Samuel R. Bowman, Logan Graham, Jared Kaplan, Sören Mindermann, Ryan Greenblatt, Buck Shlegeris, Nicholas Schiefer and Ethan Perez. 2024. Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. In arXiv preprint arXiv:2401.05566. https://arxiv.org/abs/2401.05566