Direct Preference Optimization: Your Language Model is Secretly a Reward Model

4 Direct Preference Optimization

1. What is the closed-form optimal policy of the KL-constrained objective?
   \(\pi_r(y \mid x) = \frac{1}{Z(x)}\, \pi_{\mathrm{ref}}(y \mid x) \exp\!\big(\tfrac{1}{\beta} r(x, y)\big)\), where the partition function \(Z(x)\) sums over all responses and is intractable to compute directly.
   • Rafailov et al., 2023, Direct Preference Optimization (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4
2. What is DPO’s key change of variables, and why does \(Z(x)\) disappear?
   Solving the optimal-policy equation for the reward: \(r(x, y) = \beta \log \frac{\pi_r(y \mid x)}{\pi_{\mathrm{ref}}(y \mid x)} + \beta \log Z(x)\). Bradley–Terry depends only on reward differences between two responses to the same \(x\), so the \(\beta \log Z(x)\) terms cancel, leaving preferences expressed through the policy alone.
   • Rafailov et al., 2023, Direct Preference Optimization (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4
3. State the DPO loss.
   \(\mathcal{L}_{\mathrm{DPO}} = -\mathbb{E}\big[\log \sigma(\beta h_w - \beta h_l)\big]\) over preference triples \((x, y_w, y_l)\), where \(h = \log \frac{\pi_\theta(y \mid x)}{\pi_{\mathrm{ref}}(y \mid x)}\) is the log-ratio of policy to reference for a response, \(y_w\) the preferred and \(y_l\) the dispreferred one.
   • Rafailov et al., 2023, Direct Preference Optimization (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4
4. What does the DPO gradient do, and why does its weighting matter?
   It raises the likelihood of preferred responses and lowers that of dispreferred ones, weighting each example by \(\sigma(\hat r_\theta(x, y_l) - \hat r_\theta(x, y_w))\): how wrongly the implicit reward \(\hat r_\theta = \beta \log \frac{\pi_\theta}{\pi_{\mathrm{ref}}}\) orders the pair. Without this weight, a naive version made the model degenerate.
   • Rafailov et al., 2023, Direct Preference Optimization (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4
5. Outline the DPO pipeline and the choice of reference model.
   Collect preference pairs for prompts (or reuse a public dataset), then minimize \(\mathcal{L}_{\mathrm{DPO}}\). Set \(\pi_{\mathrm{ref}}\) to the SFT model that produced the samples; if it isn’t available, fine-tune on the preferred responses first to reduce distribution shift.
   • Rafailov et al., 2023, Direct Preference Optimization (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4