DeepSeek-V3 Technical Report

2.1.2 DeepSeekMoE with Auxiliary-Loss-Free Load Balancing

1. What does a DeepSeek-V3 MoE layer look like?
   One shared expert that every token uses plus 256 fine-grained routed experts, of which each token picks the top 8 by sigmoid affinity to the expert’s centroid; gates are normalized over the chosen ones. Each token is sent to at most 4 nodes to limit communication.
   • DeepSeek-AI, 2024, DeepSeek-V3 Technical Report (arXiv preprint arXiv:2412.19437), §2.1.2
2. How does DeepSeek-V3 balance expert load without an auxiliary loss?
   Each expert has a bias added to its affinity score for top-\(K\) routing only (the gate value still uses the raw score). After each step the bias of an overloaded expert is lowered by \(\gamma\) and that of an underloaded one raised. This avoids the quality cost of a large balancing loss; only a tiny sequence-wise loss remains against extreme imbalance.
   • DeepSeek-AI, 2024, DeepSeek-V3 Technical Report (arXiv preprint arXiv:2412.19437), §2.1.2