DeepSeek-V3 Technical Report
DeepSeek-AI, 2024, DeepSeek-V3 Technical Report (arXiv preprint arXiv:2412.19437)
Subscriber deck — free during early access
1 cards in “4.5.3 Batch-Wise Load Balance VS. Sequence-Wise Load Balance” · whole deck
4.5.3 Batch-Wise Load Balance VS. Sequence-Wise Load Balance
- Why might balancing expert load per batch beat balancing it per sequence?Batch-wise balance is a looser constraint: a sequence from one domain may use some experts heavily as long as the whole batch is balanced, so experts can specialize by domain. A batch-wise auxiliary loss matched the auxiliary-loss-free method’s validation loss, and both beat a sequence-wise loss.
- DeepSeek-AI, 2024, DeepSeek-V3 Technical Report (arXiv preprint arXiv:2412.19437), §4.5.3
Source
- DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. In arXiv preprint arXiv:2412.19437. https://arxiv.org/abs/2412.19437