Large Language Diffusion Models
Nie et al., 2025, Large Language Diffusion Models (Advances in Neural Information Processing Systems 38 (NeurIPS 2025))
Subscriber deck — free during early access
1 cards in “2.3 Supervised Fine-Tuning” · whole deck
2.3 Supervised Fine-Tuning
- How does LLaDA do supervised fine-tuning?The prompt is left intact and only response tokens are masked (each with probability \(t\)); the loss is computed on masked response tokens. Short pairs are padded with end-of-sequence tokens, which the model learns to predict, so it can control the length of its answers. 4.5M pairs were used.
- Nie et al., 2025, Large Language Diffusion Models (Advances in Neural Information Processing Systems 38 (NeurIPS 2025)), §2.3
Source
- Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen and Chongxuan Li. 2025. Large Language Diffusion Models. In Advances in Neural Information Processing Systems 38 (NeurIPS 2025). https://arxiv.org/abs/2502.09992