Direct Preference Optimization: Your Language Model is Secretly a Reward Model

5.1 Your Language Model Is Secretly a Reward Model

1. Why does DPO’s reparameterization lose no generality?
   Rewards that differ only by a function of the prompt, \(f(x)\), induce the same preferences and the same optimal policy. Every such class contains a reward of the form \(\beta \log \frac{\pi(y \mid x)}{\pi_{\mathrm{ref}}(y \mid x)}\), so representing rewards through a policy covers all reward classes consistent with Bradley–Terry (and Plackett–Luce).
   • Rafailov et al., 2023, Direct Preference Optimization (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §5.1