DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Whether an LLM can develop strong reasoning through reinforcement learning alone, without supervised examples of reasoning. o1 had shown that longer chains of thought at inference help, but not openly how to train for them.

GRPO: for each question it samples a group of outputs and uses each output’s reward, normalized by the group’s mean and standard deviation, as its advantage, so no critic model as large as the policy is needed.

Rule-based ones only: an accuracy reward (e.g. a boxed math answer checked against the solution, code run against tests) and a format reward for putting the reasoning between <think> tags. Neural outcome or process reward models were avoided because large-scale RL tends to hack them, and retraining them adds cost.

It only asks the model to reason inside <think> tags and then give the answer inside <answer> tags, without asking for reflection or any particular strategy, so whatever reasoning behaviors appear come from RL, not from the prompt.

AIME 2024 pass@1 rose from 15.6% to 71.0% (86.7% with majority voting). Its responses grew from hundreds to thousands of tokens, and behaviors such as reflection, re-checking earlier steps and trying alternative approaches appeared without being taught.

Midway through training, the model learned to stop mid-solution, say in effect “wait, let’s reevaluate this step by step”, and rethink its initial approach: it learned to spend more thinking on a problem, prompted only by the outcome reward.

Its reasoning was hard to read: it mixed languages and lacked formatting. DeepSeek-R1 was built to fix this.

Before RL, DeepSeek-V3-Base is fine-tuned on thousands of long chain-of-thought examples in a readable format (reasoning followed by a summary), collected by prompting and from cleaned-up R1-Zero outputs. This avoids the unstable early phase of RL from the base model and improved both readability and final performance.

A language-consistency reward, the proportion of words in the chain of thought that are in the target language, was added to the accuracy reward. It cost slightly in benchmark scores but made the reasoning more readable.

From the converged reasoning-RL checkpoint, several responses were sampled per prompt and only correct, readable ones kept (about 600k reasoning samples, some judged by DeepSeek-V3); about 200k non-reasoning samples (writing, factual QA, translation) came from the DeepSeek-V3 pipeline. DeepSeek-V3-Base was then fine-tuned on all 800k.

Helpfulness and harmlessness alongside reasoning: rule-based rewards for math, code and logic, and reward models of human preferences for general prompts. Helpfulness was judged on the final summary only, harmlessness on the whole response including the reasoning.

Distillation. Over 10K steps of RL on Qwen-32B-Base reached about QwQ-32B-Preview level, while fine-tuning the same base on R1’s 800k samples (no RL) did much better. Strong reasoning patterns found by large models transfer cheaply, though going beyond them may still need bigger bases and RL.

PRMs: a “step” is hard to define, step correctness is hard to label at scale, and a model-based reward invites reward hacking. MCTS: the token search space is vastly larger than chess, and training a fine-grained value model to guide search proved too hard for iterative self-improvement.

It is weaker than DeepSeek-V3 at function calling, multi-turn dialogue, role-play and JSON output; it may reason or answer in English (or Chinese) for queries in other languages; few-shot prompting hurts it, so zero-shot prompts are recommended; and it gained little on software engineering, where slow evaluation limited RL.