DeepSeek-V3 Technical Report
DeepSeek-AI, 2024, DeepSeek-V3 Technical Report (arXiv preprint arXiv:2412.19437)
Subscriber deck — free during early access
1 cards in “3.2.1 DualPipe and Computation-Communication Overlap” · whole deck
3.2.1 DualPipe and Computation-Communication Overlap
- What problem does DualPipe solve?Cross-node expert parallelism made communication about as costly as computation (roughly 1:1). DualPipe splits forward and backward chunks into attention, all-to-all dispatch, MLP and all-to-all combine, and interleaves a forward and a backward chunk so communication is hidden behind computation, while feeding micro-batches from both ends of the pipeline to reduce bubbles.
- DeepSeek-AI, 2024, DeepSeek-V3 Technical Report (arXiv preprint arXiv:2412.19437), §3.2.1
Source
- DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. In arXiv preprint arXiv:2412.19437. https://arxiv.org/abs/2412.19437