Direct Preference Optimization: Your Language Model is Secretly a Reward Model

RLHF is complex and often unstable: it fits a separate reward model to human preferences, then fine-tunes the language model with reinforcement learning (typically PPO) to maximize that reward without drifting too far from the original model, training several models and sampling from the policy during training.

It optimizes the same KL-constrained reward-maximization objective as RLHF exactly, but with a single stage of policy training: a simple binary cross-entropy (classification) loss on preference pairs, with no explicit reward model and no reinforcement learning.

The Bradley–Terry model: the probability that a human prefers \(y_1\) to \(y_2\) for prompt \(x\) is \(\sigma(r^*(x, y_1) - r^*(x, y_2))\), the logistic function of the difference of a latent reward. A reward model is fit by maximizing this likelihood on preference data.

\(\max_{\pi_\theta} \mathbb{E}_{x, y \sim \pi_\theta}[r_\phi(x, y)] - \beta\, D_{\mathrm{KL}}[\pi_\theta(y \mid x) \,\|\, \pi_{\mathrm{ref}}(y \mid x)]\): maximize the learned reward while staying close to the reference (SFT) model, which keeps the reward model accurate, preserves diversity and prevents collapse to a single high-reward answer.

\(\pi_r(y \mid x) = \frac{1}{Z(x)}\, \pi_{\mathrm{ref}}(y \mid x) \exp\!\big(\tfrac{1}{\beta} r(x, y)\big)\), where the partition function \(Z(x)\) sums over all responses and is intractable to compute directly.

Solving the optimal-policy equation for the reward: \(r(x, y) = \beta \log \frac{\pi_r(y \mid x)}{\pi_{\mathrm{ref}}(y \mid x)} + \beta \log Z(x)\). Bradley–Terry depends only on reward differences between two responses to the same \(x\), so the \(\beta \log Z(x)\) terms cancel, leaving preferences expressed through the policy alone.

\(\mathcal{L}_{\mathrm{DPO}} = -\mathbb{E}\big[\log \sigma(\beta h_w - \beta h_l)\big]\) over preference triples \((x, y_w, y_l)\), where \(h = \log \frac{\pi_\theta(y \mid x)}{\pi_{\mathrm{ref}}(y \mid x)}\) is the log-ratio of policy to reference for a response, \(y_w\) the preferred and \(y_l\) the dispreferred one.

It raises the likelihood of preferred responses and lowers that of dispreferred ones, weighting each example by \(\sigma(\hat r_\theta(x, y_l) - \hat r_\theta(x, y_w))\): how wrongly the implicit reward \(\hat r_\theta = \beta \log \frac{\pi_\theta}{\pi_{\mathrm{ref}}}\) orders the pair. Without this weight, a naive version made the model degenerate.

Collect preference pairs for prompts (or reuse a public dataset), then minimize \(\mathcal{L}_{\mathrm{DPO}}\). Set \(\pi_{\mathrm{ref}}\) to the SFT model that produced the samples; if it isn’t available, fine-tune on the preferred responses first to reduce distribution shift.

Rewards that differ only by a function of the prompt, \(f(x)\), induce the same preferences and the same optimal policy. Every such class contains a reward of the form \(\beta \log \frac{\pi(y \mid x)}{\pi_{\mathrm{ref}}(y \mid x)}\), so representing rewards through a policy covers all reward classes consistent with Bradley–Terry (and Plackett–Luce).

The normalization term (a soft value function of the reference policy) doesn’t change the optimum but, if missing, makes policy gradients high-variance; PPO must estimate it with a learned value function or a baseline, both hard. DPO’s reparameterized reward needs no baseline.

With a known ground-truth reward (a sentiment classifier), DPO achieved the best reward–KL frontier: higher reward at every KL from the reference policy, strictly dominating PPO and even PPO given the ground-truth reward.

Summarization (GPT-J): about 61% GPT-4 win rate against reference summaries vs 57% for PPO at its best temperature, and far more robust to sampling temperature. Dialogue (Pythia-2.8B): the only efficient method that beat the dataset’s preferred responses, matching the costly best-of-128 baseline.

Applied to CNN/DailyMail news articles after training on Reddit TL;DR posts, DPO still beat the PPO policy clearly, even though PPO also used extra unlabeled Reddit prompts.

In a human study, GPT-4 agreed with humans about as often as humans agreed with each other. A prompt that also asks for conciseness was used, since GPT-4 otherwise favoured longer, more repetitive summaries than humans did.

How DPO policies generalize out of distribution compared with explicit reward models; whether self-labelling can exploit unlabeled prompts; how reward over-optimization shows up in DPO; scaling beyond 6B parameters; and how GPT-4 win rates depend on the judging prompt.