Large Language Diffusion Models
Nie et al., 2025, Large Language Diffusion Models (Advances in Neural Information Processing Systems 38 (NeurIPS 2025))
Subscriber deck — free during early access
1 cards in “2.2 Pre-training” · whole deck
2.2 Pre-training
- How large was LLaDA 8B’s pretraining, and what architectural change did it need?2.3T tokens with 4096-token sequences, about 0.13M H800 GPU hours, similar to an autoregressive model of that size. LLaDA cannot use a KV cache, so grouped-query attention (which mainly saves KV-cache memory) was replaced by plain multi-head attention with a smaller FFN to keep the parameter count comparable.
- Nie et al., 2025, Large Language Diffusion Models (Advances in Neural Information Processing Systems 38 (NeurIPS 2025)), §2.2
Source
- Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen and Chongxuan Li. 2025. Large Language Diffusion Models. In Advances in Neural Information Processing Systems 38 (NeurIPS 2025). https://arxiv.org/abs/2502.09992