Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters

If an LLM may use a fixed but substantial amount of compute at inference time on a hard prompt, how much can it improve, and how does spending that compute compare with spending it on a bigger pretrained model?

Choosing the test-time strategy per prompt according to its difficulty (“compute-optimal” scaling) matched best-of-N with about 4× less compute; and in FLOPs-matched comparisons, a small model with extra test-time compute beat a 14× larger model on problems where the small model had non-trivial success, but not on the hardest ones.

Modifying the proposal distribution (e.g. a model fine-tuned to revise its own previous answers) and searching against a verifier (e.g. best-of-N or tree search scored by a process reward model), analogous to MCMC combining a proposal distribution with a score function.

For a prompt \(q\) and budget \(N\), choose the hyperparameters \(\theta\) of the test-time method (e.g. revisions vs parallel samples, beam width) that maximize the probability of the correct answer: \(\theta^*_q(N) = \arg\max_\theta \mathbb{E}_{y \sim \mathrm{Target}(\theta, N, q)}[\mathbb{1}_{y = y^*(q)}]\).

Relative to the base model: its pass@1 over 2,048 samples, binned into five quantiles. In deployment the true answer is unknown, so the bins are instead predicted from a learned verifier’s average final-answer score on the samples, at extra inference cost.

Without human labels: per-step correctness was estimated from Monte Carlo rollouts from each step (following Wang et al.), so step scores approximate the reward-to-go of the base model’s policy. PRM800K labels were easily exploited, likely due to the shift between GPT-4 and PaLM 2 samples.

Each solution is scored by the PRM’s prediction at its last step (better than taking the product or minimum of step scores); then “best-of-N weighted” selection sums scores over all solutions sharing the same final answer and picks the answer with the largest total.

Best-of-N samples whole solutions and picks the best. Beam search samples candidates step by step and keeps the top ones by PRM score. Lookahead search scores each step by rolling out \(k\) more steps first, a deterministic form of MCTS, and costs \(N(k+1)\) samples.

Beam search beat best-of-N at small budgets but its advantage faded and reversed at large budgets; lookahead search did worst for its cost. Stronger search over-optimized the PRM, producing repetitive low-information steps or very short solutions.

On easy questions beam search degraded with more budget (amplifying the verifier’s spurious features) while best-of-N didn’t; on medium-hard questions beam search consistently won; on the hardest, no method helped much.

From 64 parallel samples per question, each correct answer was paired with 0–4 incorrect answers chosen to be similar to it (by edit distance) as context, and the model was fine-tuned to produce the correct one. Because it also turned about 38% of correct answers into wrong ones, the final answer is chosen from the revision chain by verifier or majority vote.

It depends on difficulty: easy questions do best with purely sequential revisions (local refinement of an answer that is nearly right), harder ones with a balance of revisions and parallel samples (global search over different approaches).

With pretraining FLOPs \(6ND_{\mathrm{pretrain}}\) and inference FLOPs \(2ND_{\mathrm{inference}}\), scaling parameters by \(M\) multiplies both; the smaller model gets the matching extra inference budget. The comparison depends on \(R = D_{\mathrm{inference}}/D_{\mathrm{pretrain}}\): test-time compute won on easy and medium questions when \(R \ll 1\), pretraining won on hard questions or when \(R \gg 1\).

Combining revisions with PRM tree search and other techniques; cheaper ways to estimate question difficulty (e.g. models trained to predict it); and distilling the outputs of test-time compute back into the base model, for iterative self-improvement.