Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov et al., 2023, Direct Preference Optimization (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))
Subscriber deck — free during early access
1 cards in “6.1 How well can DPO optimize the RLHF objective?” · whole deck
6.1 How well can DPO optimize the RLHF objective?
- What did the controlled sentiment experiment show about DPO and PPO?With a known ground-truth reward (a sentiment classifier), DPO achieved the best reward–KL frontier: higher reward at every KL from the reference policy, strictly dominating PPO and even PPO given the ground-truth reward.
- Rafailov et al., 2023, Direct Preference Optimization (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §6.1
Source
- Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html