Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Rafailov et al., 2023, Direct Preference Optimization (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))
Subscriber deck — free during early access
1 cards in “6.4 Validating GPT-4 judgments with human judgments” · whole deck
6.4 Validating GPT-4 judgments with human judgments
- Why was GPT-4 trusted as an evaluator in DPO’s experiments?In a human study, GPT-4 agreed with humans about as often as humans agreed with each other. A prompt that also asks for conciseness was used, since GPT-4 otherwise favoured longer, more repetitive summaries than humans did.
- Rafailov et al., 2023, Direct Preference Optimization (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §6.4
Source
- Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning and Chelsea Finn. 2023. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://papers.nips.cc/paper_files/paper/2023/hash/a85b405ed65c6477a4fe8302b5e06ce7-Abstract-Conference.html