Direct Preference Optimization: Your Language Model is Secretly a Reward Model

3 Preliminaries

1. What preference model do RLHF and DPO assume?
   The Bradley–Terry model: the probability that a human prefers \(y_1\) to \(y_2\) for prompt \(x\) is \(\sigma(r^*(x, y_1) - r^*(x, y_2))\), the logistic function of the difference of a latent reward. A reward model is fit by maximizing this likelihood on preference data.
   • Rafailov et al., 2023, Direct Preference Optimization (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §3
2. State the RL fine-tuning objective of RLHF.
   \(\max_{\pi_\theta} \mathbb{E}_{x, y \sim \pi_\theta}[r_\phi(x, y)] - \beta\, D_{\mathrm{KL}}[\pi_\theta(y \mid x) \,\|\, \pi_{\mathrm{ref}}(y \mid x)]\): maximize the learned reward while staying close to the reference (SFT) model, which keeps the reward model accurate, preserves diversity and prevents collapse to a single high-reward answer.
   • Rafailov et al., 2023, Direct Preference Optimization (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §3