Training language models to follow instructions with human feedback

4.3 Qualitative results

1. What simple mistakes does InstructGPT still make?
   It may accept false premises, hedge excessively on simple questions (perhaps because labelers rewarded epistemic humility), and struggle with multiple explicit constraints or hard ones (e.g. “list 10 movies made in the 1930s set in France”, summaries of a set number of sentences).
   • Ouyang et al., 2022, Training language models to follow instructions with human feedback (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §4.3
2. What generalization beyond its fine-tuning data did InstructGPT show?
   It sometimes followed instructions in other languages and summarized or answered questions about code, though these were tiny parts of the fine-tuning data, suggesting it generalizes the notion of “following instructions”.
   • Ouyang et al., 2022, Training language models to follow instructions with human feedback (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §4.3