Scaling Laws for Neural Language Models
Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361)
Subscriber deck — free during early access
1 cards in “4 Charting the Infinite Data Limit and Overfitting” · whole deck
4 Charting the Infinite Data Limit and Overfitting
- How much data avoids overfitting, according to Kaplan et al.?To stay within about 0.02 nats (the seed-to-seed variation) of the infinite-data loss, \(D \gtrsim 5 \times 10^3 \, N^{0.74}\) tokens: data can grow sublinearly with model size, so models under about \(10^9\) parameters barely overfit WebText2’s 22B tokens.
- Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361), §4
Source
- Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu and Dario Amodei. 2020. Scaling Laws for Neural Language Models. In arXiv preprint arXiv:2001.08361. https://arxiv.org/abs/2001.08361