Let’s Verify Step by Step

4.1 Process vs Outcome Supervision

1. How did Lightman et al. compare outcome and process supervision fairly at small scale?
   They used the large PRM as a cheap stand-in for human labellers to supervise smaller reward models on identical datasets, varying only the supervision: process labels from the large PRM, outcome labels from it, or outcome labels from final-answer checking. Process supervision won at every data size.
   • Lightman et al., 2024, Let’s Verify Step by Step (The Twelfth International Conference on Learning Representations (ICLR 2024)), §4.1
2. Why was outcome supervision by the large PRM better than final-answer checking?
   Final-answer checking labels solutions that reach the right answer through wrong reasoning as correct (false positives); the large PRM catches many of these.
   • Lightman et al., 2024, Let’s Verify Step by Step (The Twelfth International Conference on Learning Representations (ICLR 2024)), §4.1