learn.ifs.name

Jailbroken: How Does LLM Safety Training Fail?

Wei et al., 2023, Jailbroken (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))

Subscriber deck — free during early access

2 cards in “5 Implications for Defense” · whole deck

5 Implications for Defense

  1. Why won’t scaling alone fix competing objectives?
    The conflict lies in the training objective, not data or model size: RLHF as in InstructGPT keeps terms that pull toward the pretrained model, so safety always trades off against pretraining behaviour. A fix may require moving beyond pretrain-then-fine-tune, e.g. incorporating values during pretraining.
    • Wei et al., 2023, Jailbroken (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §5
  2. What is safety–capability parity, and why is it needed?
    Safety mechanisms must be as sophisticated as the model they protect; otherwise attacks exploit capabilities the safeguards lack, e.g. a weaker filter model that can’t decode Base64 can’t flag a Base64 attack. Models can even help generate attacks on themselves (GPT-4 flagged words to obfuscate; Claude rediscovered Base64).
    • Wei et al., 2023, Jailbroken (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §5

Source