Scaling Laws for Neural Language Models

1.2 Summary of Scaling Laws

1. Give Kaplan et al.’s three basic power laws for the loss.
   Limited by parameters (trained to convergence on enough data): \(L(N) = (N_c/N)^{\alpha_N}\), \(\alpha_N \approx 0.076\). Limited by data (large model, early stopping): \(L(D) = (D_c/D)^{\alpha_D}\), \(\alpha_D \approx 0.095\). Limited by optimally used compute: \(L(C_{\min}) = (C_c^{\min}/C_{\min})^{\alpha_C^{\min}}\), \(\alpha_C^{\min} \approx 0.050\).
   • Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361), §1.2
2. By how much does doubling the number of parameters reduce the loss under \(L(N) \propto N^{-0.076}\)?
   By a factor \(2^{-0.076} \approx 0.95\), about 5%. The constants \(N_c\), \(D_c\), \(C_c\) depend on vocabulary and tokenization and have no fundamental meaning; the exponents carry the content.
   • Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361), §1.2
3. Give Kaplan et al.’s combined law for loss as a function of model and data size.
   \(L(N, D) = \Big[\big(\frac{N_c}{N}\big)^{\alpha_N/\alpha_D} + \frac{D_c}{D}\Big]^{\alpha_D}\). It reduces to \(L(N)\) as \(D \to \infty\) and \(L(D)\) as \(N \to \infty\), allows rescaling by tokenization, and has a \(1/D\) expansion; it implies data should grow as \(D \propto N^{\alpha_N/\alpha_D} \approx N^{0.74}\).
   • Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361), §1.2
4. How should a fixed compute budget be split, according to Kaplan et al.?
   Mostly into model size: \(N \propto C_{\min}^{0.73}\), batch size \(B \propto C_{\min}^{0.24}\), and serial steps \(S \propto C_{\min}^{0.03}\) (almost none). For a billion-fold increase in compute, most should go to a bigger model, a little to more data, and that mostly via larger batches.
   • Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361), §1.2