LIMA: Less Is More for Alignment

How important the two stages of training a language model are relative to each other: unsupervised pretraining versus large-scale instruction tuning and reinforcement learning from human feedback, by aligning a strong pretrained model with only 1,000 examples.

LLaMA 65B fine-tuned with the ordinary supervised loss on just 1,000 carefully curated prompts and responses, with no reinforcement learning or preference modelling.

A model’s knowledge and capabilities are learned almost entirely during pretraining; alignment mainly teaches it which subdistribution of formats to use when interacting with users. Corollary: a small set of examples should suffice to align a pretrained model.

Diverse inputs, uniform outputs: prompts cover many topics and tasks, while responses share the style of a helpful AI assistant, so the model learns one consistent format across many situations.

200 from STEM and 200 from other exchanges, sampled with temperature to balance domains; top-scoring questions answerable from the title; the top answer (score at least 10); answers filtered by length (1,200–4,096 characters) and for first-person or references to other answers, with links and images removed.

Highly upvoted Reddit answers are often witty or trolling rather than helpful, so examples were picked by hand: 150 r/WritingPrompts stories for training, and r/AskReddit prompts only for the test set.

Responses typically acknowledge the question, then answer it. Preliminary experiments showed this consistent format helped, perhaps by helping the model form a chain of thought, like “let’s think step by step”.

From only 13 training prompts with some toxicity or malice, answered by partially or fully refusing and explaining why. On 30 sensitive test prompts LIMA responded safely 80% of the time, but implicit malicious intent often slipped through.

A new end-of-turn token separating speakers (to avoid the pretrained meaning of end-of-sequence); 15 epochs; residual dropout rising from 0 at the bottom layer to 0.3 at the top; and checkpoints chosen by hand on a 50-example development set, because perplexity didn’t correlate with generation quality.

Over 300 test prompts it was preferred to Alpaca 65B (trained on 52× more data) and to RLHF-trained DaVinci003; against Bard, GPT-4 and Claude it was equal or better in 58%, 43% and 46% of cases. GPT-4 as judge showed the same trends, even preferring LIMA to its own output 19% of the time.

Using tie-discounted accuracy, GPT-4 agreed with crowd workers 78% and with authors 79% of the time, about as often as humans agreed with each other (78–82%).

Of 50 random test prompts, 50% of responses were excellent and only 6 failed to meet the prompt’s requirements; on out-of-distribution prompts (formats absent from training) the results were similar: 45% excellent, 35% pass, 20% fail.

With LLaMA 7B and 2,000 examples: diverse Stack Exchange prompts beat homogeneous wikiHow “how to” prompts; quality-filtered answers beat unfiltered ones by about 0.5 on a 6-point scale; but increasing the amount of data up to 16× gave no gain.

Trained on no dialogues, it still held surprisingly coherent conversations but failed within 3 turns in 6 of 10 chats. Adding just 30 dialogue chains raised excellent responses from 45.2% to 76.1% and cut failures from 15 of 42 turns to 1 of 46, again suggesting the ability comes from pretraining.

Writing high-quality, diverse examples takes considerable mental effort and is hard to scale; and LIMA is less robust than product models: an unlucky sample or adversarial prompt can produce a weak response.