learn.ifs.name

GPQA: A Graduate-Level Google-Proof Q&A Benchmark

Rein et al., 2024, GPQA (First Conference on Language Modeling (COLM 2024))

Subscriber deck — free during early access

1 cards in “5 Related Work” · whole deck

5 Related Work

  1. How does GPQA differ from crowdsourced and curated QA benchmarks?
    In crowdsourced benchmarks non-experts set the ground truth from a passage; curated benchmarks draw on resources a non-expert can look up. GPQA’s ground truth comes from experts and is deliberately not findable by searching.
    • Rein et al., 2024, GPQA (First Conference on Language Modeling (COLM 2024)), §5

Source

  • David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael and Samuel R. Bowman. 2024. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. In First Conference on Language Modeling (COLM 2024). https://arxiv.org/abs/2311.12022