Qwen3 Technical Report
Team, 2025, Qwen3 Technical Report (arXiv preprint arXiv:2505.09388)
Subscriber deck — free during early access
1 cards in “4.5 Strong-to-Weak Distillation” · whole deck
4.5 Strong-to-Weak Distillation
- How were the small Qwen3 models trained?Off-policy distillation first, fine-tuning on teacher outputs in both modes; then on-policy distillation, where the student generates responses and is trained to match the teacher’s (Qwen3-32B or 235B-A22B) logits by minimizing KL divergence. Compared with RL from the same checkpoint, this scored better at about 1/10 of the GPU hours and also raised pass@64, which RL did not.
- Team, 2025, Qwen3 Technical Report (arXiv preprint arXiv:2505.09388), §4.5
Source
- Qwen Team. 2025. Qwen3 Technical Report. In arXiv preprint arXiv:2505.09388. https://arxiv.org/abs/2505.09388