learn.ifs.name

DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

DeepSeek-AI, 2025, DeepSeek-R1 (arXiv preprint arXiv:2501.12948)

Subscriber deck — free during early access

1 cards in “4.2 Unsuccessful Attempts” · whole deck

4.2 Unsuccessful Attempts

  1. Why did the DeepSeek-R1 team abandon process reward models and Monte Carlo Tree Search?
    PRMs: a “step” is hard to define, step correctness is hard to label at scale, and a model-based reward invites reward hacking. MCTS: the token search space is vastly larger than chess, and training a fine-grained value model to guide search proved too hard for iterative self-improvement.
    • DeepSeek-AI, 2025, DeepSeek-R1 (arXiv preprint arXiv:2501.12948), §4.2

Source

  • DeepSeek-AI. 2025. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. In arXiv preprint arXiv:2501.12948. https://arxiv.org/abs/2501.12948