learn.ifs.name

Let’s Verify Step by Step

Lightman et al., 2024, Let’s Verify Step by Step (The Twelfth International Conference on Learning Representations (ICLR 2024))

Subscriber deck — free during early access

3 cards in “1 Introduction” · whole deck

1 Introduction

  1. What is the difference between outcome and process supervision of reward models?
    An outcome-supervised reward model (ORM) is trained only on whether a solution’s final result is correct; a process-supervised reward model (PRM) gets feedback on each intermediate reasoning step.
    solution stepsprocessoutcomestep 1✓ correctstep 2✓ correctstep 3 (mistake)✗ first errorstep 4 (answer)not labelled✗ wrongsame solution, two kinds of label
    Process supervision says where the reasoning first goes wrong; outcome supervision only that it does (invented example)
  2. Why might process supervision be preferable even beyond accuracy?
    It gives more precise feedback (the location of an error), is easier for humans to interpret, and rewards following a human-endorsed chain of thought; outcome supervision can reward wrong reasoning that happens to reach the right answer.
  3. How does Lightman et al.’s study differ from Uesato et al. (2022), who found the two kinds of supervision similar?
    A more capable base model (GPT-4), much more human feedback, and the harder MATH dataset instead of grade-school math.

Source

  • Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever and Karl Cobbe. 2024. Let’s Verify Step by Step. In The Twelfth International Conference on Learning Representations (ICLR 2024). https://arxiv.org/abs/2305.20050