DeepSeek-V3 Technical Report

2.2 Multi-Token Prediction

1. How does DeepSeek-V3’s multi-token prediction (MTP) work?
   Besides the next token, each position also predicts the one after, using an extra sequential module (a Transformer block sharing the embedding and output head) that combines the main model’s state with the embedding of the next token, keeping the causal chain. MTP densifies the training signal and improved benchmarks in ablations.
   • DeepSeek-AI, 2024, DeepSeek-V3 Technical Report (arXiv preprint arXiv:2412.19437), §2.2
2. What can DeepSeek-V3’s MTP module be used for at inference?
   It can be dropped, leaving inference cost unchanged, or used for speculative decoding: the second token’s prediction was accepted 85–90% of the time, giving about 1.8× more tokens per second.
   • DeepSeek-AI, 2024, DeepSeek-V3 Technical Report (arXiv preprint arXiv:2412.19437), §2.2