learn.ifs.name

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

Zheng et al., 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track)

Subscriber deck — free during early access

16 cards

1 Introduction

  1. What gap in chatbot evaluation motivates LLM-as-a-judge?
    Users strongly prefer aligned chat models over their base models, yet benchmarks like MMLU and HELM barely tell them apart: they test core abilities on closed-ended tasks, not alignment with human preferences in open-ended, multi-turn use. Human evaluation fills the gap but is slow and costly.
  2. What is LLM-as-a-judge, and what are its two main benefits?
    Using a strong language model such as GPT-4 as a surrogate for human raters of chatbot answers. It is scalable (fast, cheap iteration without humans) and explainable: the judge gives reasons along with its verdicts.

2.2 MT-Bench

  1. What is MT-Bench?
    80 hand-written two-turn questions, 10 in each of 8 categories (writing, roleplay, extraction, reasoning, math, coding, STEM knowledge, humanities/social science knowledge), designed to test multi-turn conversation and instruction following and to separate strong models.

2.3 Chatbot Arena

  1. What is Chatbot Arena?
    A crowdsourced platform where users chat with two anonymous models side by side on any question they like and vote for the better answer; model names are revealed only after voting. It collected about 30K votes in its first month.

3.1 Types of LLM-as-a-Judge

  1. What three forms of LLM-as-a-judge do Zheng et al. propose, with their trade-offs?
    Pairwise comparison (which of two answers is better, or a tie): sensitive but scales quadratically with the number of models. Single-answer grading (a score for one answer): scalable but may miss subtle differences and drift if the judge changes. Reference-guided grading: a reference solution is supplied, e.g. for math.

3.3 Limitations of LLM-as-a-Judge

  1. What is position bias in LLM judges, and how strong was it?
    Favouring an answer because of where it appears. With two very similar answers swapped in order, all judges tested favoured one position, mostly the first; only GPT-4 gave consistent verdicts in more than 60% of cases. Claude-v1 also preferred whichever answer was named “Assistant A”.
  2. How was verbosity bias tested?
    With a “repetitive list” attack: answers containing a numbered list were padded by adding a rephrased copy of the list, adding no information. Judges often preferred the padded answer; GPT-4 resisted much better than others.
  3. What did Zheng et al. find about self-enhancement bias?
    Some judges seemed to favour their own answers relative to human preferences (GPT-4 by about 10% in win rate, Claude-v1 by about 25%), but they also favoured other models and GPT-3.5 didn’t favour itself, so the data couldn’t establish a real self-enhancement bias.
  4. What surprising weakness did GPT-4 show when grading math?
    It misjudged even elementary problems it could solve on its own, because it was misled by the answers being graded; asking it to solve the problem first (chain of thought) often reproduced the same mistake.

3.4 Addressing limitations

  1. How can position bias and math-grading errors in LLM judges be mitigated?
    Call the judge twice with the answers swapped and declare a win only if both orders agree (otherwise a tie). For math, have the judge answer the question independently first and insert that answer as a reference in the judging prompt, which cut the failure rate from 70% to 15%. Few-shot examples raised GPT-4’s consistency from 65% to 77.5%.
    judge(X, Y)X betterjudge(Y, X)X betterX winsverdicts differ after swapping?count a tie
    Swapping the answers’ order and requiring agreement neutralizes position bias

3.5 Multi-turn judge

  1. How should a two-turn MT-Bench conversation be shown to a judge?
    As complete conversations in one prompt, asking the judge to focus on the second turn. Splitting the turns into separate prompts made the judge lose track of which earlier answer belonged to which assistant.

4.2 High agreement between GPT-4 and humans

  1. How well did GPT-4 agree with human judges?
    On MT-Bench (58 expert raters, about 3K votes), GPT-4 agreed with humans 85% of the time on non-tie votes, more than humans agreed with each other (81%). On Chatbot Arena crowd votes the picture was similar. Agreement rose from about 70% toward 100% as the quality gap between the two models grew.
  2. What happened when human raters disagreed with GPT-4 and were shown its judgment?
    They found GPT-4’s reasoning reasonable in 75% of cases and changed their own choice in 34%.
  3. Why is GPT-4 single-answer grading attractive despite small losses?
    Its scores matched both pairwise GPT-4 and human preferences well, suggesting a fairly stable internal rubric, and it scales linearly with the number of models, unlike pairwise comparison.

5 Human Preference Benchmark and Standardized Benchmark

  1. What did comparing LLaMA variants on MMLU, TruthfulQA and MT-Bench show?
    No single benchmark decides model quality: fine-tuning on more ShareGPT dialogue improved MMLU with data size, while a small, high-quality set of 3K conversations quickly taught a style preferred by the GPT-4 judge without improving MMLU. They recommend combining capability and preference benchmarks.

6 Discussion

  1. What limitations of their study do Zheng et al. acknowledge?
    It focuses on helpfulness and largely neglects safety (honesty, harmlessness), and it merges dimensions such as accuracy, relevance and creativity into a single score; the bias mitigations are only preliminary.

Source