DeepSeek-V3 Technical Report
DeepSeek-AI, 2024, DeepSeek-V3 Technical Report (arXiv preprint arXiv:2412.19437)
Subscriber deck — free during early access
1 cards in “3.3.2 Improved Precision from Quantization and Multiplication” · whole deck
3.3.2 Improved Precision from Quantization and Multiplication
- What accumulation problem did DeepSeek-V3 find in FP8 matrix multiplies on H800 GPUs, and how was it fixed?Tensor Cores accumulated FP8 products with only about 14 bits of precision, giving up to about 2% relative error for inner dimension 4096. Every 128 elements, partial sums are copied to FP32 registers on CUDA cores and accumulated there in full precision, where the per-group scaling factors are also applied.
- DeepSeek-AI, 2024, DeepSeek-V3 Technical Report (arXiv preprint arXiv:2412.19437), §3.3.2
Source
- DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. In arXiv preprint arXiv:2412.19437. https://arxiv.org/abs/2412.19437