GPQA: A Graduate-Level Google-Proof Q&A Benchmark

1 Introduction

1. What is scalable oversight, and why does it need datasets like GPQA?
   Supervising AI systems on questions where humans can’t easily produce or check the answer themselves. RLHF relies on annotators judging correctness; testing oversight methods requires questions with known answers that even skilled, well-resourced non-experts can’t reliably solve.
   • Rein et al., 2024, GPQA (First Conference on Language Modeling (COLM 2024)), §1
2. What is GPQA?
   448 multiple-choice questions in biology, physics and chemistry written and validated by people with or pursuing PhDs, designed to be answerable by experts but “Google-proof” for skilled non-experts with unrestricted web access.
   • Rein et al., 2024, GPQA (First Conference on Language Modeling (COLM 2024)), §1
3. What were the headline accuracies on GPQA?
   Experts 65% (about 74% once their clear, self-acknowledged mistakes are discounted); highly skilled non-experts 34% despite spending over 30 minutes per question with the web; GPT-4 with few-shot chain of thought 39%; chance is 25%.

   • Rein et al., 2024, GPQA (First Conference on Language Modeling (COLM 2024)), §1
4. Why do the GPQA authors ask people not to post its questions online?
   To reduce leakage into future models’ training data, which would make the benchmark meaningless; they also embed a canary string so training corpora can be filtered.
   • Rein et al., 2024, GPQA (First Conference on Language Modeling (COLM 2024)), §1