DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

DeepSeek-Coder-Base-v1.5 7B further pretrained on 120B math tokens from Common Crawl plus text and code, then instruction-tuned and trained with RL. It reached 51.7% on competition-level MATH without tools or voting (60.9% with self-consistency over 64 samples), approaching GPT-4 and Gemini Ultra.

A carefully engineered pipeline for selecting math data from public web pages, and Group Relative Policy Optimization (GRPO), a variant of PPO that improves reasoning while using less memory.

DeepSeekMath-Base 7B matched Minerva 540B on math benchmarks: parameter count isn’t the only key factor; a small model pretrained on high-quality data can be strong.

Iteratively: train a fastText classifier with OpenWebMath as positives and random Common Crawl pages as negatives, recall and rank math pages from 40B deduplicated pages; find web domains where over 10% of pages were collected, hand-annotate their math URLs, add uncollected pages to the seeds and retrain. After four rounds (the fourth adding almost nothing): 35.5M pages.

Any text segment sharing an exact 10-gram with GSM8K, MATH, CMATH, AGIEval and similar benchmarks was removed (exact matching for shorter benchmark texts of at least 3 grams).

English and Chinese problems paired with chain-of-thought (natural-language steps), program-of-thought (code) and tool-integrated reasoning solutions, about 776K examples.

It is usually another model as large as the policy, adding substantial memory and compute; and since only the final token gets a reward, training an accurate per-token value function is difficult.

For each question it samples a group of \(G\) outputs from the old policy, scores them with the reward model, and normalizes within the group: with outcome rewards, every token of output \(i\) gets advantage \(\tilde r_i = \frac{r_i - \mathrm{mean}(\mathbf{r})}{\mathrm{std}(\mathbf{r})}\). The group average serves as the baseline.

Instead of folding a per-token KL penalty into the reward (which complicates the advantage), GRPO adds the KL divergence to the reference policy directly to the loss, using the unbiased estimator \(\frac{\pi_{ref}}{\pi_\theta} - \log\frac{\pi_{ref}}{\pi_\theta} - 1\).

A process reward model scores each reasoning step of every sampled output; step rewards are normalized across the group, and each token’s advantage is the sum of the normalized rewards of the steps from that token onward.

Trained only on about 144K chain-of-thought questions from GSM8K and MATH (64 samples per question), it raised GSM8K from 82.9% to 88.2% and MATH from 46.8% to 51.7%, and also improved out-of-domain benchmarks such as the Chinese CMATH (84.6% → 88.8%).

Code training before math training improved mathematical problem solving both with and without tools, partly answering whether code helps reasoning; arXiv papers, despite their popularity in math corpora, brought no notable improvement on the benchmarks tested.

SFT, rejection-sampling fine-tuning, DPO, PPO and GRPO all compute a gradient as a coefficient times \(\nabla \log \pi_\theta\) of tokens, differing in three components: the data source (offline or sampled online), the reward function (rule, model, none) and the algorithm turning data and reward into the gradient coefficient.

RL raised Maj@K (majority-vote accuracy) but not Pass@K (whether any of K samples is right): it made the output distribution more robust by boosting correct answers already among the top samples, rather than adding fundamentally new capabilities.

Geometry and theorem proving lagged behind closed models (e.g. problems on triangles and ellipses), possibly from data-selection bias; and at 7B it gained little from few-shot examples, unlike GPT-4.