Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

3.3 Limitations of LLM-as-a-Judge

1. What is position bias in LLM judges, and how strong was it?
   Favouring an answer because of where it appears. With two very similar answers swapped in order, all judges tested favoured one position, mostly the first; only GPT-4 gave consistent verdicts in more than 60% of cases. Claude-v1 also preferred whichever answer was named “Assistant A”.
   • Zheng et al., 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track), §3.3
2. How was verbosity bias tested?
   With a “repetitive list” attack: answers containing a numbered list were padded by adding a rephrased copy of the list, adding no information. Judges often preferred the padded answer; GPT-4 resisted much better than others.
   • Zheng et al., 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track), §3.3
3. What did Zheng et al. find about self-enhancement bias?
   Some judges seemed to favour their own answers relative to human preferences (GPT-4 by about 10% in win rate, Claude-v1 by about 25%), but they also favoured other models and GPT-3.5 didn’t favour itself, so the data couldn’t establish a real self-enhancement bias.
   • Zheng et al., 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track), §3.3
4. What surprising weakness did GPT-4 show when grading math?
   It misjudged even elementary problems it could solve on its own, because it was misled by the answers being graded; asking it to solve the problem first (chain of thought) often reproduced the same mistake.
   • Zheng et al., 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track), §3.3