Training language models to follow instructions with human feedback
Ouyang et al., 2022, Training language models to follow instructions with human feedback (Advances in Neural Information Processing Systems 35 (NeurIPS 2022))
Subscriber deck — free during early access
2 cards in “5.1 Implications for alignment research” · whole deck
5.1 Implications for alignment research
- How does the cost of alignment compare with pretraining in InstructGPT?Modest: 4.9 petaflop/s-days for 175B SFT and 60 for 175B PPO-ptx, vs 3,640 for training GPT-3, yet RLHF improved helpfulness more than a 100× increase in model size.
- Ouyang et al., 2022, Training language models to follow instructions with human feedback (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §5.1
- What is an alignment tax?Extra cost of aligning a model, such as degraded performance on other tasks; techniques with a high tax may not be adopted. PPO-ptx reduced InstructGPT’s tax by mixing in pretraining updates.
- Ouyang et al., 2022, Training language models to follow instructions with human feedback (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §5.1
Source
- Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022). https://papers.nips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html