Scaling Laws for Neural Language Models
Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361)
Subscriber deck — free during early access
1 cards in “5.2 Results for L(N, S_min) and Performance with Model Size and Compute” · whole deck
5.2 Results for L(N, S_min) and Performance with Model Size and Compute
- How does loss depend on model size and training steps?After a transient, \(L(N, S) = \big(\frac{N_c}{N}\big)^{\alpha_N} + \big(\frac{S_c}{S_{\min}}\big)^{\alpha_S}\) with \(\alpha_S \approx 0.76\), where \(S_{\min}\) adjusts steps to large-batch training; extrapolating early learning curves predicts the loss of longer training.
- Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361), §5.2
Source
- Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu and Dario Amodei. 2020. Scaling Laws for Neural Language Models. In arXiv preprint arXiv:2001.08361. https://arxiv.org/abs/2001.08361