Scaling Laws for Neural Language Models

2 Background and Methods

1. On what data and models were the scaling laws measured?
   Decoder-only Transformers (768 to 1.5B non-embedding parameters; also LSTMs and Universal Transformers for comparison) trained on WebText2 (about 23B BPE tokens from Reddit-linked pages through Oct 2018), with 1024-token contexts, cross-entropy loss in nats as the metric.
   • Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361), §2
2. What is the approximate non-embedding parameter count of a Transformer in terms of its shape?
   \(N \approx 2 d_{\text{model}} n_{\text{layer}} (2 d_{\text{attn}} + d_{ff}) = 12 n_{\text{layer}} d_{\text{model}}^2\) with the standard \(d_{\text{attn}} = d_{ff}/4 = d_{\text{model}}\), ignoring biases and other sub-leading terms.
   • Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361), §2