Qwen3 Technical Report
Team, 2025, Qwen3 Technical Report (arXiv preprint arXiv:2505.09388)
Subscriber deck — free during early access
1 cards in “2 Architecture” · whole deck
2 Architecture
- What changed in Qwen3’s architecture compared with Qwen2.5?Dense models keep GQA, SwiGLU, RoPE and pre-RMSNorm, but drop the QKV bias and add QK-Norm for training stability. The MoE models (30B-A3B and the flagship 235B-A22B) have 128 fine-grained experts with 8 active per token, no shared experts, and a global-batch load-balancing loss to encourage expert specialization.
- Team, 2025, Qwen3 Technical Report (arXiv preprint arXiv:2505.09388), §2
Source
- Qwen Team. 2025. Qwen3 Technical Report. In arXiv preprint arXiv:2505.09388. https://arxiv.org/abs/2505.09388