Training language models to follow instructions with human feedback
Ouyang et al., 2022, Training language models to follow instructions with human feedback (Advances in Neural Information Processing Systems 35 (NeurIPS 2022))
Subscriber deck — free during early access
1 cards in “3.1 High-level methodology” · whole deck
3.1 High-level methodology
- Describe the three steps of InstructGPT training.(1) Collect labeler demonstrations of desired behaviour on prompts and fine-tune GPT-3 on them (SFT). (2) Collect labeler rankings of model outputs and train a reward model to predict preferred outputs. (3) Fine-tune the SFT policy with PPO to maximize the reward model’s score.
Steps 2 and 3 can be repeated with comparisons of the newest policy’s outputs - Ouyang et al., 2022, Training language models to follow instructions with human feedback (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §3.1
Source
- Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022). https://papers.nips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html