learn.ifs.name

Let’s Verify Step by Step

Lightman et al., 2024, Let’s Verify Step by Step (The Twelfth International Conference on Learning Representations (ICLR 2024))

Subscriber deck — free during early access

2 cards in “2.4 Data Collection” · whole deck

2.4 Data Collection

  1. What is PRM800K?
    The released dataset of human step-level feedback: 800K labels (positive, negative or neutral per step) on 75K model-generated solutions to 12K MATH problems; neutral marks ambiguous or subtly misleading steps.
    • Lightman et al., 2024, Let’s Verify Step by Step (The Twelfth International Conference on Learning Representations (ICLR 2024)), §2.4
  2. Which solutions did Lightman et al. show to human labelers, and why?
    Mostly “convincing wrong-answer” solutions: rated highly by the current best PRM but reaching a wrong final answer. Labels on solutions that fool the reward model are the most informative, a form of active learning.
    • Lightman et al., 2024, Let’s Verify Step by Step (The Twelfth International Conference on Learning Representations (ICLR 2024)), §2.4

Source

  • Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever and Karl Cobbe. 2024. Let’s Verify Step by Step. In The Twelfth International Conference on Learning Representations (ICLR 2024). https://arxiv.org/abs/2305.20050