Direct Preference Optimization: Your Language Model is Secretly a Reward Model

1 Introduction

1. What is wrong with RLHF that DPO addresses?
   RLHF is complex and often unstable: it fits a separate reward model to human preferences, then fine-tunes the language model with reinforcement learning (typically PPO) to maximize that reward without drifting too far from the original model, training several models and sampling from the policy during training.
   • Rafailov et al., 2023, Direct Preference Optimization (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §1
2. What does DPO do, in one sentence?
   It optimizes the same KL-constrained reward-maximization objective as RLHF exactly, but with a single stage of policy training: a simple binary cross-entropy (classification) loss on preference pairs, with no explicit reward model and no reinforcement learning.

   • Rafailov et al., 2023, Direct Preference Optimization (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §1