Training language models to follow instructions with human feedback

4.1 Results on the API distribution

1. How did each training step change labeler preferences?
   GPT-3 did worst; a well-crafted few-shot prompt helped; supervised fine-tuning on demonstrations helped more; and PPO on comparison data helped most. Adding pretraining updates (PPO-ptx) barely changed preferences.
   • Ouyang et al., 2022, Training language models to follow instructions with human feedback (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §4.1
2. Why did GPT-3 fine-tuned on FLAN or T0 do worse than InstructGPT on API prompts?
   Academic data sets focus on easily measured tasks like classification and QA, while about 57% of API prompts are open-ended generation; FLAN and T0 models did about as well as well-prompted GPT-3 and worse than SFT.
   • Ouyang et al., 2022, Training language models to follow instructions with human feedback (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §4.1
3. Do InstructGPT’s gains just reflect overfitting to the training labelers?
   No: held-out labelers who produced no training data preferred InstructGPT over GPT-3 at about the same rate, though broader user groups and disagreements remain to be studied.
   • Ouyang et al., 2022, Training language models to follow instructions with human feedback (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §4.1