Large Language Diffusion Models
Nie et al., 2025, Large Language Diffusion Models (Advances in Neural Information Processing Systems 38 (NeurIPS 2025))
Subscriber deck — free during early access
1 cards in “3.1 Scalability of LLaDA on Language Tasks” · whole deck
3.1 Scalability of LLaDA on Language Tasks
- How did LLaDA scale compared with autoregressive baselines trained on the same data?From \(10^{20}\) to \(10^{23}\) pretraining FLOPs it matched them overall on six tasks, scaling even better on MMLU and GSM8K and closing the gap on weaker tasks like PIQA. An earlier study had estimated masked diffusion needs 16× more compute for the same likelihood, but likelihood is only an indirect measure.
- Nie et al., 2025, Large Language Diffusion Models (Advances in Neural Information Processing Systems 38 (NeurIPS 2025)), §3.1
Source
- Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen and Chongxuan Li. 2025. Large Language Diffusion Models. In Advances in Neural Information Processing Systems 38 (NeurIPS 2025). https://arxiv.org/abs/2502.09992