Qwen3 Technical Report

Thinking (step-by-step reasoning) and non-thinking (fast) modes in a single model, switchable per query; a thinking budget that caps reasoning effort; strong-to-weak distillation to build small models cheaply; and pretraining on 36T tokens in 119 languages and dialects (up from 29 in Qwen2.5). All models are open under Apache 2.0.

Dense models keep GQA, SwiGLU, RoPE and pre-RMSNorm, but drop the QKV bias and add QK-Norm for training stability. The MoE models (30B-A3B and the flagship 235B-A22B) have 128 fine-grained experts with 8 active per token, no shared experts, and a global-batch load-balancing loss to encourage expert specialization.

Qwen2.5-VL extracted text from large numbers of PDF-like documents (cleaned up by Qwen2.5); Qwen2.5, Qwen2.5-Math and Qwen2.5-Coder generated trillions of synthetic tokens (textbooks, QA, instructions, code); and more multilingual data was added. An annotation system labeled over 30T tokens (educational value, domain, safety) so the mixture could be tuned per instance on small proxy models.

General: over 30T tokens at 4,096-token length. Reasoning: about 5T higher-quality tokens with more STEM, code, reasoning and synthetic data, and faster learning-rate decay. Long context: hundreds of billions of tokens at 32,768 length, with RoPE base raised from 10,000 to 1,000,000, plus YaRN and Dual Chunk Attention for 4× longer contexts at inference.

(1) Long chain-of-thought cold start; (2) reasoning RL; (3) thinking mode fusion: SFT that adds non-thinking behavior to the reasoning model; (4) general RL across many tasks. Smaller models skip these and are distilled from the large ones instead.

Queries had to be verifiable and need reasoning (dropped if Qwen2.5-72B-Instruct answered them without chain of thought); QwQ-32B responses were removed if wrong, repetitive, guessed, inconsistent with their summary or language-mixed. The stage uses few samples and steps to instill reasoning patterns without limiting what RL can later achieve.

3,995 query–verifier pairs, unseen in the cold start, learnable but as hard as possible, trained with GRPO using large batches, many rollouts per query and off-policy reuse, while keeping the model’s entropy steady or slowly rising. AIME’24 for the 235B model rose from 70.1 to 85.1 in 170 steps.

Continued SFT on a mix of thinking data (rejection-sampled from the stage-2 model itself) and non-thinking data, with a chat template where /think or /no_think in the user or system message picks the mode. Non-thinking answers keep an empty <think></think> block for format consistency; in multi-turn dialogue the last flag wins.

When thinking reaches a user-set length, generation is interrupted with an instruction saying that, given limited time, the model must answer from its thinking so far, followed by </think>. The model then answers from partial reasoning. This was not trained directly; it emerged from mode fusion. Scores rose smoothly with larger budgets.

Over 20 tasks (instruction following, format and mode-flag following, preference alignment, tool use in real multi-turn environments, RAG and more), scored by rule-based rewards, by Qwen2.5-72B-Instruct judging against a reference answer, or by a reward model trained on human preferences when no reference exists.

Off-policy distillation first, fine-tuning on teacher outputs in both modes; then on-policy distillation, where the student generates responses and is trained to match the teacher’s (Qwen3-32B or 235B-A22B) logits by minimizing KL divergence. Compared with RL from the same checkpoint, this scored better at about 1/10 of the GPU hours and also raised pass@64, which RL did not.

They improved general abilities, instruction and mode following (ThinkFollow from 88.7 to 98.9) and tool use, but brought no gains in knowledge, STEM, math and coding, and slightly lowered thinking-mode scores on AIME’24 and LiveCodeBench. The team accepted this trade-off for a more versatile model.