Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng et al., 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track)
Subscriber deck — free during early access
1 cards in “3.1 Types of LLM-as-a-Judge” · whole deck
3.1 Types of LLM-as-a-Judge
- What three forms of LLM-as-a-judge do Zheng et al. propose, with their trade-offs?Pairwise comparison (which of two answers is better, or a tie): sensitive but scales quadratically with the number of models. Single-answer grading (a score for one answer): scalable but may miss subtle differences and drift if the judge changes. Reference-guided grading: a reference solution is supplied, e.g. for math.
- Zheng et al., 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track), §3.1
Source
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track. https://papers.nips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html