learn.ifs.name

GPQA: A Graduate-Level Google-Proof Q&A Benchmark

Rein et al., 2024, GPQA (First Conference on Language Modeling (COLM 2024))

Subscriber deck — free during early access

1 cards in “2.3 Dataset Splits” · whole deck

2.3 Dataset Splits

  1. What are GPQA Extended, GPQA and GPQA Diamond?
    Extended: all 546 released questions. GPQA (main, 448): removes questions both experts got wrong while all non-experts got right. Diamond (198): only questions both experts answered correctly (or where the second clearly acknowledged a mistake) and most non-experts got wrong.
    • Rein et al., 2024, GPQA (First Conference on Language Modeling (COLM 2024)), §2.3

Source

  • David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael and Samuel R. Bowman. 2024. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. In First Conference on Language Modeling (COLM 2024). https://arxiv.org/abs/2311.12022