Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

2.2. Metrics for LLM Reasoning Capacity Boundary

1. Why do Yue et al. use pass@k with large \(k\) instead of pass@1?
   Pass@1 and greedy accuracy measure average behavior and can miss problems a model solves only occasionally. Pass@k counts a problem as solved if any of \(k\) samples is correct, so at large \(k\) (up to 256 or more) it estimates which problems lie within the model’s reach: its reasoning boundary.
   • Yue et al., 2025, Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (arXiv preprint arXiv:2504.13837), 2.2. Metrics for LLM Reasoning Capacity Boundary
2. What pitfalls of pass@k do Yue et al. address?
   In math a model can reach the right answer by luck with a wrong chain of thought as \(k\) grows, so they manually checked the reasoning on a subset; with code, unit tests make guessing ineffective. And with an astronomically large \(k\) anything succeeds, but the base model already succeeded at realistic \(k\).
   • Yue et al., 2025, Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (arXiv preprint arXiv:2504.13837), 2.2. Metrics for LLM Reasoning Capacity Boundary