Jailbroken: How Does LLM Safety Training Fail?

4 Empirical Evaluation of Jailbreak Methods

1. Which jailbreaks were most effective, and what does that imply for defense?
   Combinations of simple attacks (e.g. prefix injection plus refusal suppression plus Base64) were by far the strongest. Since simple ideas can be combined in combinatorially many ways, such attacks are hard to defend against one by one.
   • Wei et al., 2023, Jailbroken (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4
2. What did the adaptive attack and Claude’s role-play resistance reveal?
   Claude v1.3 had apparently been trained against role-play jailbreaks (all of them failed), yet an attacker choosing among 28 attacks per prompt still succeeded on essentially every prompt: training against specific attacks isn’t enough.
   • Wei et al., 2023, Jailbroken (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4
3. How can scale create new vulnerabilities, according to Wei et al.?
   GPT-3.5 Turbo couldn’t understand Base64 or complex combination attacks, so those failed on it; GPT-4 could follow Base64 instructions but without safeguards. New capabilities open new attack surfaces.
   • Wei et al., 2023, Jailbroken (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4