s1: Simple test-time scaling

What is the simplest recipe that gives both strong reasoning and test-time scaling (better accuracy with more thinking)? o1 had shown such scaling without revealing how, and replications such as DeepSeek-R1 used large-scale RL on many samples.

(1) Supervised fine-tuning of Qwen2.5-32B-Instruct on just 1,000 carefully chosen questions with reasoning traces (s1K), which took 26 minutes on 16 H100s; (2) budget forcing at test time to control how long the model thinks.

From 16 sources chosen for quality, difficulty and diversity (mostly NuminaMATH, plus past AIME problems, Olympiad and standardized-test questions, and two new sets: Stanford statistics PhD qualifying-exam probability questions and hard quant-interview brain-teasers). Reasoning traces and answers came from Gemini 2.0 Flash Thinking, decontaminated against the evaluations with 8-grams.

Quality: drop API errors and badly formatted samples. Difficulty: drop questions that Qwen2.5-7B-Instruct or Qwen2.5-32B-Instruct already solves. Diversity: repeatedly pick a domain (from the Mathematics Subject Classification) uniformly at random, then a question from it, favoring longer reasoning traces, until there are 1,000 across 50 domains.

The model is fine-tuned to imitate the reasoning process, not to memorize answers: SFT here mainly teaches reasoning patterns, so traces with wrong final answers still help.

A decoding-time control of thinking length. To cap it, the end-of-thinking delimiter (optionally with “Final Answer:”) is appended once the budget is used, forcing an answer. To extend it, the end-of-thinking delimiter is suppressed and “Wait” is appended, which often makes the model re-check and fix its answer.

Control: the fraction of runs that stay within the intended compute budget. Scaling: the average slope of accuracy against thinking tokens, which must be positive. Performance: the best accuracy reached.

Accuracy rose with thinking tokens, and forcing longer thinking extrapolated beyond the model’s unforced performance on AIME24, from 50% to 57%, exceeding o1-preview on competition math. Suppressing the end more than about six times flattened out as the model fell into repetitive loops.

Majority voting over up to 64 samples from Qwen2.5-32B-Instruct never caught up with s1-32B under budget forcing, supporting the intuition that sequential scaling, where later thinking builds on earlier thinking, is more effective than parallel.

Choosing 1,000 samples by quality alone (random), diversity alone, or difficulty alone (longest traces) was clearly worse than s1K, about 30 points lower on AIME24 on average. Training on all 59K was not clearly better, while costing 394 H100 hours instead of 7.

Telling the model a token budget failed because it cannot count tokens; with a step budget it hit any target with a similar number of tokens, just spread across more, shorter steps; and rejection sampling to a length limit showed inverse scaling: shorter accepted samples were more often correct, as long ones backtrack a lot.

It eventually flattens out, and sequential thinking is bounded by the context window; the authors suggest combining it with parallel methods such as majority voting or reward-guided tree search (REBASE). Like other test-time methods, it was tested only on problems with checkable answers, not on open-ended tasks like creative writing.