Training Compute-Optimal Large Language Models
Hoffmann et al., 2022, Training Compute-Optimal Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022))
Subscriber deck — free during early access
1 cards in “3.2 Approach 2: IsoFLOP profiles” · whole deck
3.2 Approach 2: IsoFLOP profiles
- Describe Approach 2 (IsoFLOP profiles).For each of 9 fixed FLOP budgets (\(6 \times 10^{18}\) to \(3 \times 10^{21}\)), train a range of model sizes with token counts chosen to use exactly that budget; final loss against size shows a clear valley; fit a parabola to find the optimal size, then fit power laws: \(a = 0.49\), \(b = 0.51\).
Schematic isoFLOP profiles: each budget C has a loss-optimal model size - Hoffmann et al., 2022, Training Compute-Optimal Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §3.2
Source
- Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae and Laurent Sifre. 2022. Training Compute-Optimal Large Language Models. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022). https://papers.nips.cc/paper_files/paper/2022/hash/c1e2faff6f588870935f114ebe04a3e5-Abstract-Conference.html