Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

5. Discussion

1. Why might RLVR stay bounded by the base model, unlike RL in Go?
   The action space of language is vast, so RL must start from a pretrained prior to get any reward. That prior is double-edged: samples far from it are usually nonsense and get no reward, so policy gradient raises the probability of good responses already within the prior and rarely discovers anything outside it.
   • Yue et al., 2025, Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (arXiv preprint arXiv:2504.13837), 5. Discussion
2. What directions do Yue et al. suggest for RL that goes beyond the base model?
   Exploration at a higher level of abstraction (e.g. program-level search like AlphaEvolve), curricula from easy to hard problems so hard ones yield some reward, process rewards with finer credit assignment, and multi-turn interaction with environments.
   • Yue et al., 2025, Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (arXiv preprint arXiv:2504.13837), 5. Discussion