Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov et al., 2023, Direct Preference Optimization (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))
Subscriber deck — free during early access
1 cards in “6.3 Generalization to a new input distribution” · whole deck
6.3 Generalization to a new input distribution
- Did DPO generalize to a different input distribution?Applied to CNN/DailyMail news articles after training on Reddit TL;DR posts, DPO still beat the PPO policy clearly, even though PPO also used extra unlabeled Reddit prompts.
- Rafailov et al., 2023, Direct Preference Optimization (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §6.3
Source
- Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html