Scaling Laws for Neural Language Models

1.3 Notation

1. How do Kaplan et al. estimate training compute, and why exclude embeddings from \(N\)?
   \(C \approx 6NBS\) non-embedding FLOPs (about \(2N\) per token forward, twice that backward), quoted in PF-days (\(8.64 \times 10^{19}\) FLOPs). Counting non-embedding parameters gives much cleaner trends: with embeddings, models of different depth seem to follow different curves.
   • Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361), §1.3
2. Estimate the training compute for a model with \(N = 10^9\) non-embedding parameters trained on \(2 \times 10^{10}\) tokens.
   \(C \approx 6 N D = 6 \cdot 10^9 \cdot 2 \cdot 10^{10} = 1.2 \times 10^{20}\) FLOPs, about \(1.2 \times 10^{20} / 8.64 \times 10^{19} \approx 1.4\) PF-days.
   • Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361), §1.3