Qwen3 Technical Report
Team, 2025, Qwen3 Technical Report (arXiv preprint arXiv:2505.09388)
Subscriber deck — free during early access
1 cards in “3.2 Pre-training Stage” · whole deck
3.2 Pre-training Stage
- What were the three stages of Qwen3 pretraining?General: over 30T tokens at 4,096-token length. Reasoning: about 5T higher-quality tokens with more STEM, code, reasoning and synthetic data, and faster learning-rate decay. Long context: hundreds of billions of tokens at 32,768 length, with RoPE base raised from 10,000 to 1,000,000, plus YaRN and Dual Chunk Attention for 4× longer contexts at inference.
- Team, 2025, Qwen3 Technical Report (arXiv preprint arXiv:2505.09388), §3.2
Source
- Qwen Team. 2025. Qwen3 Technical Report. In arXiv preprint arXiv:2505.09388. https://arxiv.org/abs/2505.09388