Training language models to follow instructions with human feedback

1 Introduction

1. Why is the language modelling objective “misaligned”, according to Ouyang et al.?
   Predicting the next token of web text differs from “follow the user’s instructions helpfully and safely”, so large LMs can make up facts, produce biased or toxic text, or ignore instructions; making them bigger doesn’t by itself make them better at following intent.
   • Ouyang et al., 2022, Training language models to follow instructions with human feedback (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §1
2. What does “aligned” mean in the InstructGPT paper?
   Acting in accordance with the user’s intentions, explicit (following instructions) and implicit (staying truthful, not biased, toxic or harmful): in Askell et al.’s terms, helpful, honest and harmless.
   • Ouyang et al., 2022, Training language models to follow instructions with human feedback (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §1
3. What was the headline result of InstructGPT?
   Labelers preferred outputs of the 1.3B-parameter InstructGPT model to those of 175B GPT-3, despite 100× fewer parameters; 175B InstructGPT was preferred to GPT-3 85 ± 3% of the time and to few-shot-prompted GPT-3 71 ± 4% of the time.
   • Ouyang et al., 2022, Training language models to follow instructions with human feedback (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §1
4. Whose preferences is InstructGPT aligned to?
   To the stated preferences of a specific group: mostly the roughly 40 contractors who labelled data and the researchers who instructed them, not any broader notion of “human values”.
   • Ouyang et al., 2022, Training language models to follow instructions with human feedback (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §1