Large Language Diffusion Models

2.1 Probabilistic Formulation

1. Describe LLaDA’s forward and reverse processes.
   Forward: each token is independently replaced by a mask with probability \(t\), from \(t = 0\) (clean text) to \(t = 1\) (all masked). Reverse: a Transformer mask predictor, with no causal mask, sees the partly masked sequence and predicts all masked tokens at once, step by step from \(t = 1\) back to \(t = 0\).
   • Nie et al., 2025, Large Language Diffusion Models (Advances in Neural Information Processing Systems 38 (NeurIPS 2025)), §2.1
2. What is LLaDA’s training loss, and why does it matter that it is a bound?
   Cross-entropy on the masked positions only, weighted by \(1/t\): \(-\mathbb{E}\big[\frac{1}{t}\sum_i \mathbb{1}[x_t^i = \mathrm{M}] \log p_\theta(x_0^i \mid x_t)\big]\). This is an upper bound on the negative log-likelihood, so LLaDA is a principled generative model trained by (approximate) maximum likelihood.
   • Nie et al., 2025, Large Language Diffusion Models (Advances in Neural Information Processing Systems 38 (NeurIPS 2025)), §2.1
3. How does LLaDA’s training differ from BERT’s masked language modeling and from MaskGIT?
   BERT masks a fixed fraction (15%) of tokens; LLaDA samples the masking ratio \(t\) uniformly from 0 to 1 for each sequence, so it learns to generate from fully masked text. MaskGIT’s heuristic objective lacks the \(1/t\) weight and therefore the link to likelihood.
   • Nie et al., 2025, Large Language Diffusion Models (Advances in Neural Information Processing Systems 38 (NeurIPS 2025)), §2.1