GPQA: A Graduate-Level Google-Proof Q&A Benchmark

2.1 Data Collection Pipeline

1. Describe GPQA’s four-stage data collection pipeline.
   An expert writes a question with answer explanations; a first expert validator answers and gives feedback; the writer revises; then a second expert validator and three non-expert validators (PhD-level experts in other domains) attempt it.
   • Rein et al., 2024, GPQA (First Conference on Language Modeling (COLM 2024)), §2.1
2. How did GPQA’s payment scheme encourage questions that are both objective and hard?
   Writers got bonuses for each expert validator who answered correctly (objectivity) and for each non-expert who answered incorrectly (difficulty), plus an extra bonus when both held; validators were paid more for correct answers and useful feedback. Average pay was about $95 per hour.
   • Rein et al., 2024, GPQA (First Conference on Language Modeling (COLM 2024)), §2.1
3. Who counts as a non-expert validator for a GPQA question, and how hard did they try?
   Someone with expertise in a different domain (e.g. a physicist on a chemistry question; organic chemists and biologists were kept apart), with unrestricted internet access but no LLM assistants. They had to spend at least 15 minutes and averaged 37, sometimes reading research papers or writing simulations.
   • Rein et al., 2024, GPQA (First Conference on Language Modeling (COLM 2024)), §2.1