AI Models Collapse When Trained on Recursively Generated Data

What happens to future models (“GPT-{n}”) once much of the web text they are trained on was produced by earlier models: models training on their predecessors’ output, generation after generation.

A degenerative process in which data generated by a model pollute the training set of the next generation, which then misperceives reality; models gradually forget the true data distribution, even though the distribution itself doesn’t change.

Early: the model loses information about the tails of the distribution (rare events). Late: it converges to a distribution bearing little resemblance to the original, often with greatly reduced variance, approaching a point estimate.

Statistical approximation error, from finite samples (rare events may not be sampled at all), the primary cause; functional expressivity error, from limited model capacity (e.g. fitting two Gaussians with one); and functional approximation error, from the learning procedure itself, such as biases of SGD or the objective.

Expressiveness can counteract statistical noise but can equally fit and compound it: an overfitted density model may extrapolate and assign high density to regions outside the training support, which are then sampled with arbitrary frequency in the next generation.

With \(M\) samples, a state of probability \(q \le 1/M\) is expected to appear less than once, so it is likely lost. Viewed as a Markov chain, the only absorbing states are delta functions (all samples identical), so with probability 1 the process ends in a single state, losing all information.

If each generation fits a Gaussian with unbiased mean and variance estimates from a fixed number of samples of the previous generation, then the approximation drifts arbitrarily far from the original (in Wasserstein-2 distance) while its variance collapses to zero, with probability 1.

OPT-125m was fine-tuned on wikitext2, then used to regenerate a dataset of the same size (continuing each 64-token block), on which the next generation was fine-tuned, and so on, either with no original data or keeping a random 10% of the original data each generation.

Later generations produced more sequences the original model found likely, but also a growing tail of sequences it would never produce: accumulated errors. Text degenerated, e.g. by generation 9 a passage on church architecture had become a list of jackrabbits with differently coloured tails.

Keeping 10% of the original data reduced degradation to minor levels but didn’t eliminate it. Penalizing repetition made models produce lower-probability continuations and the next generations perform even worse (perplexity doubled).

Rare events are often relevant to marginalized groups, so losing them harms fairness, and they are vital for understanding complex systems.

A “first mover advantage”: access to genuine human-generated data (pre-LLM web crawls, real human interactions) becomes increasingly valuable, and the provenance of crawled content must be tracked, perhaps through community-wide coordination among those building and deploying LLMs.