GPQA: A Graduate-Level Google-Proof Q&A Benchmark
Rein et al., 2024, GPQA (First Conference on Language Modeling (COLM 2024))
Subscriber deck — free during early access
1 cards in “3.1 Question Objectivity” · whole deck
3.1 Question Objectivity
- How was question objectivity estimated beyond raw expert accuracy?After answering, experts saw the writer’s explanation and said whether the answer was uncontroversially correct. Of 191 second-expert errors, 46% ended with the validator clearly accepting the question (often describing their own mistake); excluding such cases gives about 74–76% objectivity on the extended set.
- Rein et al., 2024, GPQA (First Conference on Language Modeling (COLM 2024)), §3.1
Source
- David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael and Samuel R. Bowman. 2024. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. In First Conference on Language Modeling (COLM 2024). https://arxiv.org/abs/2311.12022