Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov et al., 2023, Direct Preference Optimization (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))
Subscriber deck — free during early access
1 cards in “5.2 Instability of Actor-Critic Algorithms” · whole deck
5.2 Instability of Actor-Critic Algorithms
- How does the DPO analysis explain PPO’s instability in RLHF?The normalization term (a soft value function of the reference policy) doesn’t change the optimum but, if missing, makes policy gradients high-variance; PPO must estimate it with a learned value function or a baseline, both hard. DPO’s reparameterized reward needs no baseline.
- Rafailov et al., 2023, Direct Preference Optimization (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §5.2
Source
- Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html