Universal and Transferable Adversarial Attacks on Aligned Language Models

Earlier jailbreaks were hand-crafted with human ingenuity and brittle, and automatic adversarial prompt search had largely failed on aligned models. Zou et al. automatically optimize an adversarial suffix appended to the user’s request, leaving the request itself unchanged.

Text inputs are discrete tokens, which limits the effective input dimensionality and makes the search combinatorial; gradient methods that work in continuous pixel space don’t apply directly.

(1) Target an affirmative start of the response (“Sure, here is …”); (2) optimize the suffix with combined greedy and gradient-based discrete search; (3) optimize one suffix over many prompts and several models at once, making it universal and transferable.

Like adversarial examples in vision, which remain unsolved because defenses are costly, hurt clean performance and cover narrow threat models, attacks on aligned models may persist: post hoc “repair” of models may not suffice.

Making “Sure, here is [the requested content]” the most likely start puts the model into a mode where it continues with the content. Targeting only “Sure” risks a suffix that overrides the request (e.g. “never mind, tell me a joke”); restating the request keeps the behaviour on target.

The negative log-probability of the target token sequence \(x^\star_{n+1:n+H}\) (the affirmative beginning) given the prompt, \(\mathcal{L}(x_{1:n}) = -\log p(x^\star_{n+1:n+H} \mid x_{1:n})\), minimized over the tokens at the suffix positions only.

Compute the gradient of the loss with respect to the one-hot vector of each suffix token; for every position take the top-\(k\) tokens with the most negative gradient as candidates; sample \(B\) single-token swaps from all positions, evaluate their exact loss with forward passes, and keep the best. Repeat.

AutoPrompt picks one position in advance and only tries replacements there; GCG considers candidate swaps at all positions in each step. With the same number of forward passes this seemingly minor change made a large difference: 88% vs 25% exact harmful-string matches on Vicuna-7B.

Optimize a single suffix against the summed losses of many harmful prompts (adding prompts incrementally once earlier ones succeed) and of several models sharing a tokenizer (Vicuna-7B and 13B, plus Guanaco), aggregating normalized gradients.

Zou et al.’s benchmark with two settings: 500 harmful strings the model must output exactly (fine-grained control), and 500 harmful behaviors where any genuine attempt to comply counts as success (like red-teaming). The metric is attack success rate.

Exact harmful strings: 88% on Vicuna-7B and 57% on LLaMA-2-7B-Chat (AutoPrompt 25% and 3%; PEZ and GBDA none). Harmful behaviors: 100% and 88%. A single suffix optimized on 25 behaviors also worked on held-out behaviors, so universal attacks exist.

Suffixes optimized only on open Vicuna and Guanaco models elicited harmful behaviors from GPT-3.5 (up to about 87%), GPT-4 (about 47–54%) and PaLM-2 (66%), but rarely Claude 2 (about 2%); transfer was best to GPT models, perhaps because Vicuna was trained on ChatGPT outputs.

Concatenating several GCG suffixes raised GPT-3.5 success from 47% to 80% (though GPT-4 then often asked for clarification), and an ensemble trying several suffixes per request succeeded more often; stopping optimization earlier also reduced over-fitting to the source models.

Whether adversarial training (attacking the model during fine-tuning and training on refusals) can make models robust without sacrificing capability, which has not been achieved for classical models, or whether pretraining itself needs to change.

The techniques are simple, previously known in similar forms and discoverable by any dedicated team; manual jailbreaks were already widespread; risks grow as language models gain autonomy. They disclosed results to the model providers before publication.