AI Models Collapse When Trained on Recursively Generated Data
Shumailov et al., 2024, AI models collapse when trained on recursively generated data (Nature, vol. 631)
Subscriber deck — free during early access
2 cards in “Theoretical intuition” · whole deck
Theoretical intuition
- Why must recursive training on a discrete distribution collapse even with perfect fitting?With \(M\) samples, a state of probability \(q \le 1/M\) is expected to appear less than once, so it is likely lost. Viewed as a Markov chain, the only absorbing states are delta functions (all samples identical), so with probability 1 the process ends in a single state, losing all information.
Each generation is fitted to 40 samples from the previous one: tails vanish first (simulation, not from the paper) - Shumailov et al., 2024, AI models collapse when trained on recursively generated data (Nature, vol. 631), Theoretical intuition, p. 757
- What does the Gaussian model collapse theorem state?If each generation fits a Gaussian with unbiased mean and variance estimates from a fixed number of samples of the previous generation, then the approximation drifts arbitrarily far from the original (in Wasserstein-2 distance) while its variance collapses to zero, with probability 1.
- Shumailov et al., 2024, AI models collapse when trained on recursively generated data (Nature, vol. 631), Theoretical intuition, p. 757
Source
- Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson and Yarin Gal. 2024. AI models collapse when trained on recursively generated data. In Nature, vol. 631, pp. 755–759. License: CC BY 4.0. Flashcards adapted from this work. doi:10.1038/s41586-024-07566-y