Let’s Verify Step by Step

An outcome-supervised reward model (ORM) is trained only on whether a solution’s final result is correct; a process-supervised reward model (PRM) gets feedback on each intermediate reasoning step.

It gives more precise feedback (the location of an error), is easier for humans to interpret, and rewards following a human-endorsed chain of thought; outcome supervision can reward wrong reasoning that happens to reach the right answer.

A more capable base model (GPT-4), much more human feedback, and the harder MATH dataset instead of grade-school math.

By best-of-N search: a fixed generator samples N solutions per problem, the reward model picks its top-ranked one, and the fraction of picked solutions with a correct final answer is reported. The generator itself is not trained with RL.

The released dataset of human step-level feedback: 800K labels (positive, negative or neutral per step) on 75K model-generated solutions to 12K MATH problems; neutral marks ambiguous or subtly misleading steps.

Mostly “convincing wrong-answer” solutions: rated highly by the current best PRM but reaching a wrong final answer. Labels on solutions that fool the reward model are the most informative, a form of active learning.

It predicts after each step’s last token whether the step is correct, in one forward pass; the solution’s score is the probability that every step is correct, i.e. the product of the step probabilities.

It makes the comparison fair: for correct solutions both methods say every step is right; for incorrect ones both say there is a mistake, and process supervision adds only where the first mistake is. Finding the first mistake is also what a human must do to judge correctness without a checkable answer.

Searching over GPT-4 generator samples, the PRM beat both at every N, and the gap widened as N grew; the PRM solved 78.2% of a representative subset of MATH test problems. The ORM was only slightly better than majority voting.

They used the large PRM as a cheap stand-in for human labellers to supervise smaller reward models on identical datasets, varying only the supervision: process labels from the large PRM, outcome labels from it, or outcome labels from final-answer checking. Process supervision won at every data size.

Final-answer checking labels solutions that reach the right answer through wrong reasoning as correct (false positives); the large PRM catches many of these.

Choosing mostly convincing wrong-answer samples for labelling was about 2.6× more data-efficient than labelling uniformly sampled solutions. Iteratively retraining the selector during collection was unstable and didn’t help.

Yes: on 224 recent AP Physics, Calculus, Chemistry and AMC10/12 problems released after pretraining data was collected, the PRM again beat the ORM and majority voting.

Credit assignment: with outcome labels, a reward model must work out for itself where a wrong solution went wrong, and on hard problems most solutions are wrong somewhere, so a negative label says little. Process labels say how many steps were right and exactly which one failed.

The performance cost sometimes paid for safer training methods, which discourages adopting them. Process supervision had a negative alignment tax: it was both more interpretable and better performing, at least in math.

Data scale: their scaling curves show a little process supervision matching a lot of outcome supervision, as Uesato et al. saw, but process supervision pulling ahead as more is collected.