learn.ifs.name

Training language models to follow instructions with human feedback

Ouyang et al., 2022, Training language models to follow instructions with human feedback (Advances in Neural Information Processing Systems 35 (NeurIPS 2022))

Subscriber deck — free during early access

21 cards

1 Introduction

  1. Why is the language modelling objective “misaligned”, according to Ouyang et al.?
    Predicting the next token of web text differs from “follow the user’s instructions helpfully and safely”, so large LMs can make up facts, produce biased or toxic text, or ignore instructions; making them bigger doesn’t by itself make them better at following intent.
  2. What does “aligned” mean in the InstructGPT paper?
    Acting in accordance with the user’s intentions, explicit (following instructions) and implicit (staying truthful, not biased, toxic or harmful): in Askell et al.’s terms, helpful, honest and harmless.
  3. What was the headline result of InstructGPT?
    Labelers preferred outputs of the 1.3B-parameter InstructGPT model to those of 175B GPT-3, despite 100× fewer parameters; 175B InstructGPT was preferred to GPT-3 85 ± 3% of the time and to few-shot-prompted GPT-3 71 ± 4% of the time.
  4. Whose preferences is InstructGPT aligned to?
    To the stated preferences of a specific group: mostly the roughly 40 contractors who labelled data and the researchers who instructed them, not any broader notion of “human values”.

3.1 High-level methodology

  1. Describe the three steps of InstructGPT training.
    (1) Collect labeler demonstrations of desired behaviour on prompts and fine-tune GPT-3 on them (SFT). (2) Collect labeler rankings of model outputs and train a reward model to predict preferred outputs. (3) Fine-tune the SFT policy with PPO to maximize the reward model’s score.
    1. SFTfine-tune GPT-3 on human demos2. reward modelhumans rank 4–9 outputs → train RM3. PPOmaximize RM reward, starting from SFT;KL penalty + pretraining mix
    Steps 2 and 3 can be repeated with comparisons of the newest policy’s outputs

3.2 Dataset

  1. Where did InstructGPT’s prompts come from, and how big were the data sets?
    Mostly prompts submitted to the OpenAI API (over 96% English, deduplicated, PII filtered, with users split between train and test) plus some labeler-written ones: about 13k prompts for SFT, 33k for the reward model, and 31k unlabelled API prompts for PPO.

3.3 Human data collection

  1. How were InstructGPT’s labelers chosen, and how much did they agree?
    About 40 contractors selected by a screening test for sensitivity to different demographic groups’ preferences and for spotting harmful outputs. Training labelers agreed 72.6% of the time; held-out labelers (not screened, no training data) 77.3%.

3.4 Models

  1. How is the InstructGPT reward model trained?
    A 6B GPT-3 model maps a prompt and response to a scalar. Labelers rank \(K = 4\) to 9 responses, giving \(\binom{K}{2}\) pairs, all used as one batch element: \(\text{loss}(\theta) = -\frac{1}{\binom{K}{2}} \mathbb{E}_{(x, y_w, y_l)} [\log \sigma(r_\theta(x, y_w) - r_\theta(x, y_l))]\), where \(y_w\) is preferred over \(y_l\).
  2. How many pairwise comparisons does one ranking of \(K\) responses give, and why train them together?
    \(\binom{K}{2}\): 6 for \(K = 4\), 36 for \(K = 9\). Training on all of them as one batch element per prompt is computationally efficient: each of the \(K\) completions takes part in \(K - 1\) comparisons but needs only one forward pass.
  3. How is the InstructGPT policy optimized with PPO?
    As a bandit: present a random prompt, take the response, get the reward model’s score, end the episode. A per-token KL penalty from the SFT model limits over-optimization of the reward model; the value function starts from the RM.
  4. What is PPO-ptx, and why was it introduced?
    PPO with pretraining gradients mixed in (increasing the log-likelihood of the pretraining distribution), which mostly removed RLHF’s performance regressions on public NLP data sets without hurting labeler preferences; it worked better than raising the KL coefficient. “InstructGPT” usually means PPO-ptx.

3.5 Evaluation

  1. How was InstructGPT evaluated?
    Mainly by labeler preference on held-out API prompts from users not in training (win rate against 175B SFT, 1–7 Likert scores, and metadata such as following constraints or hallucinating), plus public data sets for truthfulness (TruthfulQA), toxicity (RealToxicityPrompts), bias (Winogender, CrowS-Pairs) and standard NLP tasks.

4.1 Results on the API distribution

  1. How did each training step change labeler preferences?
    GPT-3 did worst; a well-crafted few-shot prompt helped; supervised fine-tuning on demonstrations helped more; and PPO on comparison data helped most. Adding pretraining updates (PPO-ptx) barely changed preferences.
  2. Why did GPT-3 fine-tuned on FLAN or T0 do worse than InstructGPT on API prompts?
    Academic data sets focus on easily measured tasks like classification and QA, while about 57% of API prompts are open-ended generation; FLAN and T0 models did about as well as well-prompted GPT-3 and worse than SFT.
  3. Do InstructGPT’s gains just reflect overfitting to the training labelers?
    No: held-out labelers who produced no training data preferred InstructGPT over GPT-3 at about the same rate, though broader user groups and disagreements remain to be studied.

4.2 Results on public NLP datasets

  1. How did InstructGPT change truthfulness, toxicity and bias?
    More truthful and informative on TruthfulQA, and hallucinating about half as often on closed-domain API tasks (21% vs 41%); about 25% fewer toxic outputs when asked to be respectful, but no advantage without that instruction; no significant improvement on bias benchmarks.

4.3 Qualitative results

  1. What simple mistakes does InstructGPT still make?
    It may accept false premises, hedge excessively on simple questions (perhaps because labelers rewarded epistemic humility), and struggle with multiple explicit constraints or hard ones (e.g. “list 10 movies made in the 1930s set in France”, summaries of a set number of sentences).
  2. What generalization beyond its fine-tuning data did InstructGPT show?
    It sometimes followed instructions in other languages and summarized or answered questions about code, though these were tiny parts of the fine-tuning data, suggesting it generalizes the notion of “following instructions”.

5.1 Implications for alignment research

  1. How does the cost of alignment compare with pretraining in InstructGPT?
    Modest: 4.9 petaflop/s-days for 175B SFT and 60 for 175B PPO-ptx, vs 3,640 for training GPT-3, yet RLHF improved helpfulness more than a 100× increase in model size.
  2. What is an alignment tax?
    Extra cost of aligning a model, such as degraded performance on other tasks; techniques with a high tax may not be adopted. PPO-ptx reduced InstructGPT’s tax by mixing in pretraining updates.

5.2 Limitations

  1. What is perhaps the greatest limitation of InstructGPT, according to its authors?
    In most cases it follows the user’s instruction even when that could cause real-world harm: asked to be maximally biased, InstructGPT produced more toxic outputs than GPT-3 of the same size. It is neither fully aligned nor fully safe.

Source

  • Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022). https://papers.nips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html