learn.ifs.name

Jailbroken: How Does LLM Safety Training Fail?

Wei et al., 2023, Jailbroken (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))

Subscriber deck — free during early access

2 cards in “3.2 Mismatched Generalization” · whole deck

3.2 Mismatched Generalization

  1. Why does a Base64-encoded request bypass safety training?
    Large models learned to read and follow Base64 during pretraining, but safety training never contained such unnatural inputs, so refusals don’t generalize there: the model complies without safety considerations. Encoded input mattered more than encoded output, though both together were strongest.
    • Wei et al., 2023, Jailbroken (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §3.2
  2. Besides encodings, what other inputs exploit mismatched generalization?
    Obfuscations such as ciphers, leetspeak, Pig Latin, synonyms, payload splitting or translation; many unrelated “distractor” instructions; unusual output formats like JSON; and requests to write content as it would appear on a particular website.
    • Wei et al., 2023, Jailbroken (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §3.2

Source