Scaling Laws for Neural Language Models

Transformer language model test loss falls as a smooth power law in model size \(N\), dataset size \(D\) and training compute \(C\), each when not bottlenecked by the others, over more than six orders of magnitude, while depending only weakly on architecture details like depth versus width.

Very weakly: at fixed non-embedding \(N\), loss varied only a few percent over wide ranges of depth, width, number of heads and feed-forward ratio; e.g. a 6-layer model with \(d_{\text{model}} = 4288\) came within 3% of a 48-layer, 1600-wide model.

Performance improves predictably if \(N\) and \(D\) grow together, but with one fixed, returns diminish; the penalty depends on \(N^{0.74}/D\), so an 8× larger model needs only about 5× more data to avoid it.

Larger models are more sample-efficient, reaching a given loss with fewer steps and fewer tokens. Under a fixed compute budget, optimal training uses very large models stopped well before convergence, with data growing only as \(D \sim C^{0.27}\).

Loss on other distributions (books, Wikipedia, Common Crawl) improved in parallel with in-distribution loss, with a roughly constant offset; generalization depended almost only on in-distribution validation loss, not on training duration or depth.

Limited by parameters (trained to convergence on enough data): \(L(N) = (N_c/N)^{\alpha_N}\), \(\alpha_N \approx 0.076\). Limited by data (large model, early stopping): \(L(D) = (D_c/D)^{\alpha_D}\), \(\alpha_D \approx 0.095\). Limited by optimally used compute: \(L(C_{\min}) = (C_c^{\min}/C_{\min})^{\alpha_C^{\min}}\), \(\alpha_C^{\min} \approx 0.050\).

By a factor \(2^{-0.076} \approx 0.95\), about 5%. The constants \(N_c\), \(D_c\), \(C_c\) depend on vocabulary and tokenization and have no fundamental meaning; the exponents carry the content.

\(L(N, D) = \Big[\big(\frac{N_c}{N}\big)^{\alpha_N/\alpha_D} + \frac{D_c}{D}\Big]^{\alpha_D}\). It reduces to \(L(N)\) as \(D \to \infty\) and \(L(D)\) as \(N \to \infty\), allows rescaling by tokenization, and has a \(1/D\) expansion; it implies data should grow as \(D \propto N^{\alpha_N/\alpha_D} \approx N^{0.74}\).

Mostly into model size: \(N \propto C_{\min}^{0.73}\), batch size \(B \propto C_{\min}^{0.24}\), and serial steps \(S \propto C_{\min}^{0.03}\) (almost none). For a billion-fold increase in compute, most should go to a bigger model, a little to more data, and that mostly via larger batches.

\(C \approx 6NBS\) non-embedding FLOPs (about \(2N\) per token forward, twice that backward), quoted in PF-days (\(8.64 \times 10^{19}\) FLOPs). Counting non-embedding parameters gives much cleaner trends: with embeddings, models of different depth seem to follow different curves.

\(C \approx 6 N D = 6 \cdot 10^9 \cdot 2 \cdot 10^{10} = 1.2 \times 10^{20}\) FLOPs, about \(1.2 \times 10^{20} / 8.64 \times 10^{19} \approx 1.4\) PF-days.

Decoder-only Transformers (768 to 1.5B non-embedding parameters; also LSTMs and Universal Transformers for comparison) trained on WebText2 (about 23B BPE tokens from Reddit-linked pages through Oct 2018), with 1024-token contexts, cross-entropy loss in nats as the metric.

\(N \approx 2 d_{\text{model}} n_{\text{layer}} (2 d_{\text{attn}} + d_{ff}) = 12 n_{\text{layer}} d_{\text{model}}^2\) with the standard \(d_{\text{attn}} = d_{ff}/4 = d_{\text{model}}\), ignoring biases and other sub-leading terms.

LSTMs matched Transformers on tokens early in the context but plateaued after under 100 tokens, while Transformers kept improving across the whole context, so Transformers asymptotically outperform LSTMs by using long contexts better.

To stay within about 0.02 nats (the seed-to-seed variation) of the infinite-data loss, \(D \gtrsim 5 \times 10^3 \, N^{0.74}\) tokens: data can grow sublinearly with model size, so models under about \(10^9\) parameters barely overfit WebText2’s 22B tokens.

The batch size up to which larger batches cost almost no extra compute; beyond it returns diminish. Training at \(B_{\text{crit}}\) roughly balances time and compute. It depends on the loss, not the model size: \(B_{\text{crit}}(L) \approx B_* / L^{1/\alpha_B}\) with \(B_* \approx 2 \times 10^8\) tokens, \(\alpha_B \approx 0.21\), roughly doubling for every 13% decrease in loss.

After a transient, \(L(N, S) = \big(\frac{N_c}{N}\big)^{\alpha_N} + \big(\frac{S_c}{S_{\min}}\big)^{\alpha_S}\) with \(\alpha_S \approx 0.76\), where \(S_{\min}\) adjusts steps to large-batch training; extrapolating early learning curves predicts the loss of longer training.

Compute-efficient training uses data so slowly (\(D \propto C^{0.27}\)) that eventually it would overfit, so the \(L(C_{\min})\) and \(L(D)\) trends must cross, around \(10^4\) PF-days, \(10^{12}\) parameters and \(10^{12}\) tokens (very uncertain). They conjectured this marks where models extract all reliable information from text, with \(L^* \approx 1.7\) nats/token as a rough entropy estimate.

As analogues of the ideal gas law: universal relations independent of most microscopic details, possibly holding for other generative tasks. Open: whether smooth loss improvements translate into relevant task improvements, since smooth quantitative change can hide qualitative changes (“more is different”); they concluded big models may matter more than big data.