DeepSeek-V3 Technical Report
DeepSeek-AI, 2024, DeepSeek-V3 Technical Report (arXiv preprint arXiv:2412.19437)
Subscriber deck — free during early access
1 cards in “2.1.1 Multi-Head Latent Attention” · whole deck
2.1.1 Multi-Head Latent Attention
- How does Multi-head Latent Attention (MLA) shrink the KV cache?Keys and values are jointly compressed into one low-rank latent vector per token, \(\mathbf{c}^{KV}_t = W^{DKV}\mathbf{h}_t\) (512 numbers), and expanded by up-projections when needed. Only this latent and a small shared key carrying RoPE position information (64 numbers) are cached, instead of full keys and values for all 128 heads.
Sizes from DeepSeek-V3’s configuration; up-projections run only when attending - DeepSeek-AI, 2024, DeepSeek-V3 Technical Report (arXiv preprint arXiv:2412.19437), §2.1.1
Source
- DeepSeek-AI. 2024. DeepSeek-V3 Technical Report. In arXiv preprint arXiv:2412.19437. https://arxiv.org/abs/2412.19437