Scaling Laws for Neural Language Models

1.1 Summary

1. What is the main finding of “Scaling Laws for Neural Language Models”?
   Transformer language model test loss falls as a smooth power law in model size \(N\), dataset size \(D\) and training compute \(C\), each when not bottlenecked by the others, over more than six orders of magnitude, while depending only weakly on architecture details like depth versus width.

   • Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361), §1.1
2. How does model shape affect language model loss at fixed parameter count, according to Kaplan et al.?
   Very weakly: at fixed non-embedding \(N\), loss varied only a few percent over wide ranges of depth, width, number of heads and feed-forward ratio; e.g. a 6-layer model with \(d_{\text{model}} = 4288\) came within 3% of a 48-layer, 1600-wide model.
   • Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361), §1.1
3. What does “universality of overfitting” mean in Kaplan et al.?
   Performance improves predictably if \(N\) and \(D\) grow together, but with one fixed, returns diminish; the penalty depends on \(N^{0.74}/D\), so an 8× larger model needs only about 5× more data to avoid it.
   • Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361), §1.1
4. What did Kaplan et al. find about sample efficiency and convergence?
   Larger models are more sample-efficient, reaching a given loss with fewer steps and fewer tokens. Under a fixed compute budget, optimal training uses very large models stopped well before convergence, with data growing only as \(D \sim C^{0.27}\).
   • Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361), §1.1
5. What did Kaplan et al. find about generalization to other text distributions?
   Loss on other distributions (books, Wikipedia, Common Crawl) improved in parallel with in-distribution loss, with a roughly constant offset; generalization depended almost only on in-distribution validation loss, not on training duration or depth.
   • Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361), §1.1