Let’s Verify Step by Step

2.6 Process-supervised Reward Models

1. How does a PRM score a whole solution?
   It predicts after each step’s last token whether the step is correct, in one forward pass; the solution’s score is the probability that every step is correct, i.e. the product of the step probabilities.
   • Lightman et al., 2024, Let’s Verify Step by Step (The Twelfth International Conference on Learning Representations (ICLR 2024)), §2.6
2. Why did Lightman et al. give process labels only up to the first incorrect step?
   It makes the comparison fair: for correct solutions both methods say every step is right; for incorrect ones both say there is a mistake, and process supervision adds only where the first mistake is. Finding the first mistake is also what a human must do to judge correctness without a checkable answer.
   • Lightman et al., 2024, Let’s Verify Step by Step (The Twelfth International Conference on Learning Representations (ICLR 2024)), §2.6