Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

1 Introduction

1. What gap in chatbot evaluation motivates LLM-as-a-judge?
   Users strongly prefer aligned chat models over their base models, yet benchmarks like MMLU and HELM barely tell them apart: they test core abilities on closed-ended tasks, not alignment with human preferences in open-ended, multi-turn use. Human evaluation fills the gap but is slow and costly.
   • Zheng et al., 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track), §1
2. What is LLM-as-a-judge, and what are its two main benefits?
   Using a strong language model such as GPT-4 as a surrogate for human raters of chatbot answers. It is scalable (fast, cheap iteration without humans) and explainable: the judge gives reasons along with its verdicts.
   • Zheng et al., 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track), §1