DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

4.1.1 From PPO to GRPO

1. Why is PPO’s value model a problem for LLM fine-tuning?
   It is usually another model as large as the policy, adding substantial memory and compute; and since only the final token gets a reward, training an accurate per-token value function is difficult.
   • Shao et al., 2024, DeepSeekMath (arXiv preprint arXiv:2402.03300), §4.1.1
2. How does GRPO estimate advantages without a value model?
   For each question it samples a group of \(G\) outputs from the old policy, scores them with the reward model, and normalizes within the group: with outcome rewards, every token of output \(i\) gets advantage \(\tilde r_i = \frac{r_i - \mathrm{mean}(\mathbf{r})}{\mathrm{std}(\mathbf{r})}\). The group average serves as the baseline.

   • Shao et al., 2024, DeepSeekMath (arXiv preprint arXiv:2402.03300), §4.1.1
3. How does GRPO handle the KL penalty differently from PPO?
   Instead of folding a per-token KL penalty into the reward (which complicates the advantage), GRPO adds the KL divergence to the reference policy directly to the loss, using the unbiased estimator \(\frac{\pi_{ref}}{\pi_\theta} - \log\frac{\pi_{ref}}{\pi_\theta} - 1\).
   • Shao et al., 2024, DeepSeekMath (arXiv preprint arXiv:2402.03300), §4.1.1