Training Compute-Optimal Large Language Models

3.4 Optimal model scaling

1. How do the Chinchilla scaling exponents compare with Kaplan et al.’s?
   All three approaches give \(N_{\text{opt}} \propto C^{\approx 0.5}\) and \(D_{\text{opt}} \propto C^{\approx 0.5}\) (equal scaling), whereas Kaplan et al. had \(N \propto C^{0.73}\), \(D \propto C^{0.27}\); so existing large models were oversized for their budgets, and far more data is needed.

   method	a (N)	b (D)
   1. training curve envelope	0.50	0.50
   2. IsoFLOP profiles	0.49	0.51
   3. parametric loss fit	0.46	0.54
   Kaplan et al. (2020)	0.73	0.27

   • Hoffmann et al., 2022, Training Compute-Optimal Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §3.4
2. Check that Chinchilla matches Gopher’s compute budget, and give the tokens-per-parameter ratio it implies.
   With \(C \approx 6ND\): \(6 \times 7 \times 10^{10} \times 1.4 \times 10^{12} \approx 5.9 \times 10^{23}\) FLOPs, matching Gopher’s \(5.76 \times 10^{23}\). That is \(1.4 \times 10^{12} / 7 \times 10^{10} = 20\) tokens per parameter.
   • Hoffmann et al., 2022, Training Compute-Optimal Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §3.4