Scaling Laws for Neural Language Models
Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361)
Subscriber deck — free during early access
1 cards in “6.3 Contradictions and a Conjecture” · whole deck
6.3 Contradictions and a Conjecture
- What contradiction did Kaplan et al. find in their own scaling laws, and what did they conjecture?Compute-efficient training uses data so slowly (\(D \propto C^{0.27}\)) that eventually it would overfit, so the \(L(C_{\min})\) and \(L(D)\) trends must cross, around \(10^4\) PF-days, \(10^{12}\) parameters and \(10^{12}\) tokens (very uncertain). They conjectured this marks where models extract all reliable information from text, with \(L^* \approx 1.7\) nats/token as a rough entropy estimate.
- Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361), §6.3
Source
- Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu and Dario Amodei. 2020. Scaling Laws for Neural Language Models. In arXiv preprint arXiv:2001.08361. https://arxiv.org/abs/2001.08361