Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng et al., 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track)
Subscriber deck — free during early access
1 cards in “5 Human Preference Benchmark and Standardized Benchmark” · whole deck
5 Human Preference Benchmark and Standardized Benchmark
- What did comparing LLaMA variants on MMLU, TruthfulQA and MT-Bench show?No single benchmark decides model quality: fine-tuning on more ShareGPT dialogue improved MMLU with data size, while a small, high-quality set of 3K conversations quickly taught a style preferred by the GPT-4 judge without improving MMLU. They recommend combining capability and preference benchmarks.
- Zheng et al., 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track), §5
Source
- Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track. https://papers.nips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html