Training Compute-Optimal Large Language Models
Hoffmann et al., 2022, Training Compute-Optimal Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022))
Subscriber deck — free during early access
1 cards in “3.3 Approach 3: Fitting a parametric loss function” · whole deck
3.3 Approach 3: Fitting a parametric loss function
- Give Chinchilla’s parametric loss model and the meaning of its terms.\(\hat L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}\): \(E\) is the loss of an ideal generative process (the entropy of natural text); \(A/N^\alpha\) is the shortfall of a finite \(N\)-parameter Transformer; \(B/D^\beta\) is the shortfall from finite training on \(D\) tokens. Fitted with a Huber loss on log loss via L-BFGS; minimizing under \(\text{FLOPs} \approx 6ND\) gives \(a = 0.46\), \(b = 0.54\).
- Hoffmann et al., 2022, Training Compute-Optimal Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §3.3
Source
- Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae and Laurent Sifre. 2022. Training Compute-Optimal Large Language Models. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022). https://papers.nips.cc/paper_files/paper/2022/hash/c1e2faff6f588870935f114ebe04a3e5-Abstract-Conference.html