Training Compute-Optimal Large Language Models

1 Introduction

1. What is the main finding of the Chinchilla paper?
   Large language models of the time were significantly undertrained: for compute-optimal training, model size and number of training tokens should be scaled equally, so every doubling of model size should come with a doubling of training tokens.
   • Hoffmann et al., 2022, Training Compute-Optimal Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §1
2. What question does the Chinchilla paper formalize?
   Given a fixed FLOPs budget \(C\), choose model size \(N\) and training tokens \(D\) to minimize final loss: \(N_{\text{opt}}(C), D_{\text{opt}}(C) = \arg\min_{N, D \text{ s.t. } \text{FLOPs}(N, D) = C} L(N, D)\).
   • Hoffmann et al., 2022, Training Compute-Optimal Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §1
3. How were large models trained before Chinchilla, and why?
   Following Kaplan et al. (2020), model size grew faster than data: models from 175B to 530B parameters (GPT-3, Jurassic-1, Gopher, MT-NLG) were all trained on about 300B tokens.
   • Hoffmann et al., 2022, Training Compute-Optimal Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §1
4. What is Chinchilla, and how does it compare with Gopher?
   A 70B-parameter model trained on 1.4 trillion tokens with the same compute as the 280B Gopher (4× smaller, 4× more data). It uniformly and significantly outperformed Gopher, GPT-3, Jurassic-1 and MT-NLG 530B, and is cheaper to fine-tune and run.
   • Hoffmann et al., 2022, Training Compute-Optimal Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §1