Jailbroken: How Does LLM Safety Training Fail?

3.1 Competing Objectives

1. How does prefix injection exploit competing objectives?
   The model is asked to begin its answer with an innocuous, eager prefix. It follows the harmless-looking instruction (refusing harmless instructions is penalized), and once the prefix is written, a refusal is unlikely under the pretraining distribution, so it continues with the restricted content. Changing the prefix to “Hello!” removed the effect.
   • Wei et al., 2023, Jailbroken (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §3.1
2. How does refusal suppression exploit competing objectives?
   The model is told to answer under rules that rule out typical refusal phrasing (e.g. never apologize, avoid words that start refusals). Instruction following downweights refusal openings; once a compliant answer starts, pretraining favours continuing it. Inverting the rules elicited no restricted behavior.
   • Wei et al., 2023, Jailbroken (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §3.1