Jailbroken: How Does LLM Safety Training Fail?

Why jailbreak attacks on safety-trained models succeed and how they can be constructed: it traces them to two failure modes of safety training and uses those to design new attacks.

Competing objectives: the model’s capability goals (language modelling, instruction following) conflict with its safety goal. Mismatched generalization: safety training fails to cover a domain where the model still has capabilities from pretraining.

Despite extensive red-teaming and safety training, attacks built on the two failure modes succeeded on over 96% of evaluated harmful prompts, including all of a curated set of red-teaming prompts that safety training was meant to address, and beat ad hoc jailbreaks shared online.

They shared preliminary results with OpenAI and Anthropic, omitted the exact prompts of the strongest attacks, focused on how attacks are constructed conceptually, and made code and data available to researchers on request.

An attempt to get an on-topic response to a prompt for restricted behavior by submitting a modified prompt, with only black-box chat access: no weights, no system-prompt changes, no edited history. Outcomes are labelled GOOD BOT (refusal), BAD BOT (on-topic response) or UNCLEAR.

The model is asked to begin its answer with an innocuous, eager prefix. It follows the harmless-looking instruction (refusing harmless instructions is penalized), and once the prefix is written, a refusal is unlikely under the pretraining distribution, so it continues with the restricted content. Changing the prefix to “Hello!” removed the effect.

The model is told to answer under rules that rule out typical refusal phrasing (e.g. never apologize, avoid words that start refusals). Instruction following downweights refusal openings; once a compliant answer starts, pretraining favours continuing it. Inverting the rules elicited no restricted behavior.

Large models learned to read and follow Base64 during pretraining, but safety training never contained such unnatural inputs, so refusals don’t generalize there: the model complies without safety considerations. Encoded input mattered more than encoded output, though both together were strongest.

Obfuscations such as ciphers, leetspeak, Pig Latin, synonyms, payload splitting or translation; many unrelated “distractor” instructions; unusual output formats like JSON; and requests to write content as it would appear on a particular website.

Combinations of simple attacks (e.g. prefix injection plus refusal suppression plus Base64) were by far the strongest. Since simple ideas can be combined in combinatorially many ways, such attacks are hard to defend against one by one.

Claude v1.3 had apparently been trained against role-play jailbreaks (all of them failed), yet an attacker choosing among 28 attacks per prompt still succeeded on essentially every prompt: training against specific attacks isn’t enough.

GPT-3.5 Turbo couldn’t understand Base64 or complex combination attacks, so those failed on it; GPT-4 could follow Base64 instructions but without safeguards. New capabilities open new attack surfaces.

The conflict lies in the training objective, not data or model size: RLHF as in InstructGPT keeps terms that pull toward the pretrained model, so safety always trades off against pretraining behaviour. A fix may require moving beyond pretrain-then-fine-tune, e.g. incorporating values during pretraining.

Safety mechanisms must be as sophisticated as the model they protect; otherwise attacks exploit capabilities the safeguards lack, e.g. a weaker filter model that can’t decode Base64 can’t flag a Base64 attack. Models can even help generate attacks on themselves (GPT-4 flagged words to obfuscate; Claude rediscovered Base64).