DeepSeek-V3 Technical Report

DeepSeek-AI, 2024, DeepSeek-V3 Technical Report (arXiv preprint arXiv:2412.19437)

Subscriber deck — free during early access

1 cards in “3.3 FP8 Training” · whole deck

3.3 FP8 Training

Why is FP8 training hard, and how did DeepSeek-V3 handle outliers?
FP8’s narrow range makes per-tensor scaling sensitive to outliers. DeepSeek-V3 scales activations per 1×128 tile (per token, per 128 channels) and weights per 128×128 block, so one outlier only affects its small group.
- DeepSeek-AI, 2024, DeepSeek-V3 Technical Report (arXiv preprint arXiv:2412.19437), §3.3

Source

DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. In arXiv preprint arXiv:2412.19437. https://arxiv.org/abs/2412.19437