Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng et al., 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track)
Subscriber deck — free during early access
1 cards in “3.5 Multi-turn judge” · whole deck
3.5 Multi-turn judge
- How should a two-turn MT-Bench conversation be shown to a judge?As complete conversations in one prompt, asking the judge to focus on the second turn. Splitting the turns into separate prompts made the judge lose track of which earlier answer belonged to which assistant.
- Zheng et al., 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track), §3.5
Source
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track. https://papers.nips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html