Scaling Laws for Neural Language Models

5.1 Adjustment for Training at B_crit(L)

1. What is the critical batch size, and how does it scale?
   The batch size up to which larger batches cost almost no extra compute; beyond it returns diminish. Training at \(B_{\text{crit}}\) roughly balances time and compute. It depends on the loss, not the model size: \(B_{\text{crit}}(L) \approx B_* / L^{1/\alpha_B}\) with \(B_* \approx 2 \times 10^8\) tokens, \(\alpha_B \approx 0.21\), roughly doubling for every 13% decrease in loss.
   • Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361), §5.1