Training Compute-Optimal Large Language Models
Hoffmann et al., 2022, Training Compute-Optimal Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022))
Subscriber deck — free during early access
2 cards in “5 Discussion & Conclusion” · whole deck
5 Discussion & Conclusion
- What limitations of the Chinchilla analysis did its authors note?Only two comparable large-scale runs (Chinchilla and Gopher) with no intermediate scales; the assumption of a power-law frontier, though there is some concavity at high compute, so optimal sizes may still be overestimated; and all runs used less than one epoch of data.
- Hoffmann et al., 2022, Training Compute-Optimal Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §5
- What implications for data did the Chinchilla authors draw?Scaling requires much larger, high-quality data sets, collected responsibly with attention to quality, train–test overlap, toxicity, bias and private information, which become more prevalent in absolute terms as data grows.
- Hoffmann et al., 2022, Training Compute-Optimal Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §5
Source
- Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae and Laurent Sifre. 2022. Training Compute-Optimal Large Language Models. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022). https://papers.nips.cc/paper_files/paper/2022/hash/c1e2faff6f588870935f114ebe04a3e5-Abstract-Conference.html