Qwen3 Technical Report
Team, 2025, Qwen3 Technical Report (arXiv preprint arXiv:2505.09388)
Subscriber deck — free during early access
1 cards in “3.1 Pre-training Data” · whole deck
3.1 Pre-training Data
- How did Qwen3 enlarge its pretraining data to 36T tokens?Qwen2.5-VL extracted text from large numbers of PDF-like documents (cleaned up by Qwen2.5); Qwen2.5, Qwen2.5-Math and Qwen2.5-Coder generated trillions of synthetic tokens (textbooks, QA, instructions, code); and more multilingual data was added. An annotation system labeled over 30T tokens (educational value, domain, safety) so the mixture could be tuned per instance on small proxy models.
- Team, 2025, Qwen3 Technical Report (arXiv preprint arXiv:2505.09388), §3.1
Source
- Qwen Team. 2025. Qwen3 Technical Report. In arXiv preprint arXiv:2505.09388. https://arxiv.org/abs/2505.09388