DeepSeek-V3 Technical Report

A Mixture-of-Experts LLM with 671B parameters, of which 37B are active per token, pretrained on 14.8T tokens. Its full training took 2.788M H800 GPU hours (about $5.6M at $2 per GPU hour, excluding earlier research and ablations), with no irrecoverable loss spikes or rollbacks.

Keys and values are jointly compressed into one low-rank latent vector per token, \(\mathbf{c}^{KV}_t = W^{DKV}\mathbf{h}_t\) (512 numbers), and expanded by up-projections when needed. Only this latent and a small shared key carrying RoPE position information (64 numbers) are cached, instead of full keys and values for all 128 heads.

One shared expert that every token uses plus 256 fine-grained routed experts, of which each token picks the top 8 by sigmoid affinity to the expert’s centroid; gates are normalized over the chosen ones. Each token is sent to at most 4 nodes to limit communication.

Each expert has a bias added to its affinity score for top-\(K\) routing only (the gate value still uses the raw score). After each step the bias of an overloaded expert is lowered by \(\gamma\) and that of an underloaded one raised. This avoids the quality cost of a large balancing loss; only a tiny sequence-wise loss remains against extreme imbalance.

Besides the next token, each position also predicts the one after, using an extra sequential module (a Transformer block sharing the embedding and output head) that combines the main model’s state with the embedding of the next token, keeping the causal chain. MTP densifies the training signal and improved benchmarks in ablations.

It can be dropped, leaving inference cost unchanged, or used for speculative decoding: the second token’s prediction was accepted 85–90% of the time, giving about 1.8× more tokens per second.

Cross-node expert parallelism made communication about as costly as computation (roughly 1:1). DualPipe splits forward and backward chunks into attention, all-to-all dispatch, MLP and all-to-all combine, and interleaves a forward and a backward chunk so communication is hidden behind computation, while feeding micro-batches from both ends of the pipeline to reduce bubbles.

FP8’s narrow range makes per-tensor scaling sensitive to outliers. DeepSeek-V3 scales activations per 1×128 tile (per token, per 128 channels) and weights per 128×128 block, so one outlier only affects its small group.

Tensor Cores accumulated FP8 products with only about 14 bits of precision, giving up to about 2% relative error for inner dimension 4096. Every 128 elements, partial sums are copied to FP32 registers on CUDA cores and accumulated there in full precision, where the per-group scaling factors are also applied.

Batch-wise balance is a looser constraint: a sequence from one domain may use some experts heavily as long as the whole batch is balanced, so experts can specialize by domain. A batch-wise auxiliary loss matched the auxiliary-loss-free method’s validation loss, and both beat a sequence-wise loss.

Domain expert models were trained with SFT and RL on two kinds of samples: problems with original concise answers, and problems with R1’s long reasoning plus a system prompt asking for reflection and verification. RL taught the experts to blend R1’s patterns into clearer answers; rejection sampling from them then produced the final SFT data.

Rule-based rewards wherever answers can be checked (boxed math answers, code test cases), being harder to game; a model-based reward model for free-form answers and open-ended tasks. The policy was optimized with GRPO, which needs no critic model.

Deployment: the recommended serving unit is large (prefilling alone uses 32 GPUs), a burden for small teams, and generation speed, though over twice DeepSeek-V2’s, could still improve. They expect better hardware to ease both.