DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

2.2.4 Performance, Self-evolution Process and Aha Moment of DeepSeek-R1-Zero

1. What happened to DeepSeek-R1-Zero during RL?
   AIME 2024 pass@1 rose from 15.6% to 71.0% (86.7% with majority voting). Its responses grew from hundreds to thousands of tokens, and behaviors such as reflection, re-checking earlier steps and trying alternative approaches appeared without being taught.
   • DeepSeek-AI, 2025, DeepSeek-R1 (arXiv preprint arXiv:2501.12948), §2.2.4
2. What was DeepSeek-R1-Zero’s “aha moment”?
   Midway through training, the model learned to stop mid-solution, say in effect “wait, let’s reevaluate this step by step”, and rethink its initial approach: it learned to spend more thinking on a problem, prompted only by the outcome reward.
   • DeepSeek-AI, 2025, DeepSeek-R1 (arXiv preprint arXiv:2501.12948), §2.2.4
3. What were DeepSeek-R1-Zero’s drawbacks?
   Its reasoning was hard to read: it mixed languages and lacked formatting. DeepSeek-R1 was built to fix this.
   • DeepSeek-AI, 2025, DeepSeek-R1 (arXiv preprint arXiv:2501.12948), §2.2.4