Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters

5.1 Training Verifiers Amenable to Search

1. How did Snell et al. train their process reward model?
   Without human labels: per-step correctness was estimated from Monte Carlo rollouts from each step (following Wang et al.), so step scores approximate the reward-to-go of the base model’s policy. PRM800K labels were easily exploited, likely due to the shift between GPT-4 and PaLM 2 samples.
   • Snell et al., 2025, Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters (The Thirteenth International Conference on Learning Representations (ICLR 2025)), §5.1
2. How did Snell et al. aggregate PRM scores to choose a final answer?
   Each solution is scored by the PRM’s prediction at its last step (better than taking the product or minimum of step scores); then “best-of-N weighted” selection sums scores over all solutions sharing the same final answer and picks the answer with the largest total.
   • Snell et al., 2025, Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters (The Thirteenth International Conference on Learning Representations (ICLR 2025)), §5.1