Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov et al., 2023, Direct Preference Optimization (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))
Subscriber deck — free during early access
1 cards in “6.2 Can DPO scale to real preference datasets?” · whole deck
6.2 Can DPO scale to real preference datasets?
- How did DPO perform on TL;DR summarization and Anthropic HH dialogue?Summarization (GPT-J): about 61% GPT-4 win rate against reference summaries vs 57% for PPO at its best temperature, and far more robust to sampling temperature. Dialogue (Pythia-2.8B): the only efficient method that beat the dataset’s preferred responses, matching the costly best-of-128 baseline.
- Rafailov et al., 2023, Direct Preference Optimization (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §6.2
Source
- Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html