GPQA: A Graduate-Level Google-Proof Q&A Benchmark

4 Baseline

1. How did language models with web search do on GPQA?
   GPT-4 with a self-ask style search tool reached only 39.4% on the extended set, barely above few-shot chain of thought, and abstained on 37% of main-set questions (backing off to the chain-of-thought answer); Llama-2-70B-chat and GPT-3.5 scored 28–31%.
   • Rein et al., 2024, GPQA (First Conference on Language Modeling (COLM 2024)), §4
2. Why is it useful that models score between non-experts and experts on GPQA?
   It enables “sandwiching” experiments: non-experts interact with an unreliable but somewhat helpful model and try to reach expert-level accuracy. If the model were near-perfect, they could simply defer to it.
   • Rein et al., 2024, GPQA (First Conference on Language Modeling (COLM 2024)), §4