GPQA: A Graduate-Level Google-Proof Q&A Benchmark

Supervising AI systems on questions where humans can’t easily produce or check the answer themselves. RLHF relies on annotators judging correctness; testing oversight methods requires questions with known answers that even skilled, well-resourced non-experts can’t reliably solve.

448 multiple-choice questions in biology, physics and chemistry written and validated by people with or pursuing PhDs, designed to be answerable by experts but “Google-proof” for skilled non-experts with unrestricted web access.

Experts 65% (about 74% once their clear, self-acknowledged mistakes are discounted); highly skilled non-experts 34% despite spending over 30 minutes per question with the web; GPT-4 with few-shot chain of thought 39%; chance is 25%.

To reduce leakage into future models’ training data, which would make the benchmark meaningless; they also embed a canary string so training corpora can be filtered.

An expert writes a question with answer explanations; a first expert validator answers and gives feedback; the writer revises; then a second expert validator and three non-expert validators (PhD-level experts in other domains) attempt it.

Writers got bonuses for each expert validator who answered correctly (objectivity) and for each non-expert who answered incorrectly (difficulty), plus an extra bonus when both held; validators were paid more for correct answers and useful feedback. Average pay was about $95 per hour.

Someone with expertise in a different domain (e.g. a physicist on a chemistry question; organic chemists and biologists were kept apart), with unrestricted internet access but no LLM assistants. They had to spend at least 15 minutes and averaged 37, sometimes reading research papers or writing simulations.

Extended: all 546 released questions. GPQA (main, 448): removes questions both experts got wrong while all non-experts got right. Diamond (198): only questions both experts answered correctly (or where the second clearly acknowledged a mistake) and most non-experts got wrong.

After answering, experts saw the writer’s explanation and said whether the answer was uncontroversially correct. Of 191 second-expert errors, 46% ended with the validator clearly accepting the question (often describing their own mistake); excluding such cases gives about 74–76% objectivity on the extended set.

GPT-4 with a self-ask style search tool reached only 39.4% on the extended set, barely above few-shot chain of thought, and abstained on 37% of main-set questions (backing off to the chain-of-thought answer); Llama-2-70B-chat and GPT-3.5 scored 28–31%.

It enables “sandwiching” experiments: non-experts interact with an unreliable but somewhat helpful model and try to reach expert-level accuracy. If the model were near-perfect, they could simply defer to it.

In crowdsourced benchmarks non-experts set the ground truth from a passage; curated benchmarks draw on resources a non-expert can look up. GPQA’s ground truth comes from experts and is deliberately not findable by searching.

It is small (only large accuracy differences are statistically detectable); its non-experts are unusually skilled, so other settings must measure their own non-expert accuracy; contributors from one freelance platform may bias its content; and it remains a proxy for supervising truly superhuman systems.