Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

That, like RL in Go or Atari, RLVR lets LLMs discover genuinely new reasoning strategies beyond their base models. The alternative they test: it only makes the model better at finding reasoning paths the base model could already produce.

Pass@1 and greedy accuracy measure average behavior and can miss problems a model solves only occasionally. Pass@k counts a problem as solved if any of \(k\) samples is correct, so at large \(k\) (up to 256 or more) it estimates which problems lie within the model’s reach: its reasoning boundary.

In math a model can reach the right answer by luck with a wrong chain of thought as \(k\) grows, so they manually checked the reasoning on a subset; with code, unit tests make guessing ineffective. And with an astronomically large \(k\) anything succeeds, but the base model already succeeded at realistic \(k\).

Across model families, math, code and visual reasoning benchmarks, and RL algorithms, RLVR models win at small \(k\) (e.g. pass@1), but as \(k\) reaches the tens or hundreds the base models catch up and overtake them: RLVR narrows the set of solvable problems rather than widening it.

The problems the RL model solves are nearly a subset of those the base model solves at large \(k\); and the RL model’s responses have low perplexity under the base model, i.e. they are outputs the base model was already likely to produce. RL shifts probability toward rewarded paths rather than creating new ones.

DeepSeek-R1-Distill-Qwen-7B, fine-tuned on R1’s long reasoning traces, had a pass@k curve consistently above its base model (Qwen2.5-Math-7B) at all \(k\): distillation brings in new reasoning patterns from a stronger teacher, while RLVR stays within the base model’s limits.

\(\Delta_{SE}\) = the base model’s pass@256 minus the RL model’s pass@1: how far RL is from turning everything the base model can do into first-try success. PPO, GRPO, Reinforce++, RLOO, ReMax and DAPO all differed only slightly and stayed above 40 points.

Training-set pass@1 kept rising (from 26.1 to 42.5) while pass@256 kept falling. More rollouts per prompt (32 instead of 8) helped pass@k slightly, but the base model still won at large \(k\).

Only partly. Raising the RL model’s sampling temperature until its entropy matched the base model’s improved its large-\(k\) pass@k somewhat, but it still fell below the base model.

The action space of language is vast, so RL must start from a pretrained prior to get any reward. That prior is double-edged: samples far from it are usually nonsense and get no reward, so policy gradient raises the probability of good responses already within the prior and rarely discovers anything outside it.

Exploration at a higher level of abstraction (e.g. program-level search like AlphaEvolve), curricula from easy to hard problems so hard ones yield some reward, process rewards with finer credit assignment, and multi-turn interaction with environments.

Many of the strongest models and RL pipelines are proprietary and could not be evaluated; experiments on larger models (e.g. Magistral-Medium) were preliminary; and RL methods change quickly, so newer techniques may overcome the limits they found.