Jailbroken: How Does LLM Safety Training Fail?

1 Introduction

1. What question does Jailbroken address, beyond showing that jailbreaks exist?
   Why jailbreak attacks on safety-trained models succeed and how they can be constructed: it traces them to two failure modes of safety training and uses those to design new attacks.
   • Wei et al., 2023, Jailbroken (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §1
2. What are the two failure modes of safety training proposed by Wei et al.?
   Competing objectives: the model’s capability goals (language modelling, instruction following) conflict with its safety goal. Mismatched generalization: safety training fails to cover a domain where the model still has capabilities from pretraining.

   • Wei et al., 2023, Jailbroken (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §1
3. What did the empirical evaluation show about GPT-4 and Claude v1.3?
   Despite extensive red-teaming and safety training, attacks built on the two failure modes succeeded on over 96% of evaluated harmful prompts, including all of a curated set of red-teaming prompts that safety training was meant to address, and beat ad hoc jailbreaks shared online.
   • Wei et al., 2023, Jailbroken (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §1
4. How did Wei et al. handle disclosure of their attacks?
   They shared preliminary results with OpenAI and Anthropic, omitted the exact prompts of the strongest attacks, focused on how attacks are constructed conceptually, and made code and data available to researchers on request.
   • Wei et al., 2023, Jailbroken (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §1