learn.ifs.name

GPQA: A Graduate-Level Google-Proof Q&A Benchmark

Rein et al., 2024, GPQA (First Conference on Language Modeling (COLM 2024))

Subscriber deck — free during early access

1 cards in “6 Limitations” · whole deck

6 Limitations

  1. What limitations of GPQA do Rein et al. acknowledge?
    It is small (only large accuracy differences are statistically detectable); its non-experts are unusually skilled, so other settings must measure their own non-expert accuracy; contributors from one freelance platform may bias its content; and it remains a proxy for supervising truly superhuman systems.
    • Rein et al., 2024, GPQA (First Conference on Language Modeling (COLM 2024)), §6

Source

  • David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael and Samuel R. Bowman. 2024. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. In First Conference on Language Modeling (COLM 2024). https://arxiv.org/abs/2311.12022