Training Compute-Optimal Large Language Models
Hoffmann et al., 2022, Training Compute-Optimal Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022))
Subscriber deck — free during early access
1 cards in “2 Related Work” · whole deck
2 Related Work
- Why did Kaplan et al. (2020) conclude that model size should grow faster than data, according to Hoffmann et al.?They used one fixed learning-rate schedule length (130B tokens) for all runs, so intermediate losses of shorter runs overestimated what a matched schedule would achieve, underestimating the value of more data; they also mostly used smaller models (many under 100M parameters).
- Hoffmann et al., 2022, Training Compute-Optimal Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §2
Source
- Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae and Laurent Sifre. 2022. Training Compute-Optimal Large Language Models. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022). https://papers.nips.cc/paper_files/paper/2022/hash/c1e2faff6f588870935f114ebe04a3e5-Abstract-Conference.html