learn.ifs.name

Training Compute-Optimal Large Language Models

Hoffmann et al., 2022, Training Compute-Optimal Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022))

Subscriber deck — free during early access

18 cards

1 Introduction

  1. What is the main finding of the Chinchilla paper?
    Large language models of the time were significantly undertrained: for compute-optimal training, model size and number of training tokens should be scaled equally, so every doubling of model size should come with a doubling of training tokens.
  2. What question does the Chinchilla paper formalize?
    Given a fixed FLOPs budget \(C\), choose model size \(N\) and training tokens \(D\) to minimize final loss: \(N_{\text{opt}}(C), D_{\text{opt}}(C) = \arg\min_{N, D \text{ s.t. } \text{FLOPs}(N, D) = C} L(N, D)\).
  3. How were large models trained before Chinchilla, and why?
    Following Kaplan et al. (2020), model size grew faster than data: models from 175B to 530B parameters (GPT-3, Jurassic-1, Gopher, MT-NLG) were all trained on about 300B tokens.
  4. What is Chinchilla, and how does it compare with Gopher?
    A 70B-parameter model trained on 1.4 trillion tokens with the same compute as the 280B Gopher (4× smaller, 4× more data). It uniformly and significantly outperformed Gopher, GPT-3, Jurassic-1 and MT-NLG 530B, and is cheaper to fine-tune and run.

2 Related Work

  1. Why did Kaplan et al. (2020) conclude that model size should grow faster than data, according to Hoffmann et al.?
    They used one fixed learning-rate schedule length (130B tokens) for all runs, so intermediate losses of shorter runs overestimated what a matched schedule would achieve, underestimating the value of more data; they also mostly used smaller models (many under 100M parameters).

3 Estimating the optimal parameter/training tokens allocation

  1. How did the Chinchilla authors gather data on scaling?
    By training over 400 language models, from 70M to over 16B parameters, on 5B to 500B tokens, with learning-rate schedules matched to the number of training tokens.

3.1 Approach 1: Fix model sizes and vary number of training tokens

  1. Describe Approach 1 for estimating compute-optimal scaling.
    Train each model size with 4 different schedule lengths, take the envelope of minimal loss per FLOP across all runs, read off the best \(N\) and \(D\) for each FLOP count, and fit power laws \(N_{\text{opt}} \propto C^a\), \(D_{\text{opt}} \propto C^b\): \(a = 0.50\), \(b = 0.50\).

3.2 Approach 2: IsoFLOP profiles

  1. Describe Approach 2 (IsoFLOP profiles).
    For each of 9 fixed FLOP budgets (\(6 \times 10^{18}\) to \(3 \times 10^{21}\)), train a range of model sizes with token counts chosen to use exactly that budget; final loss against size shows a clear valley; fit a parabola to find the optimal size, then fit power laws: \(a = 0.49\), \(b = 0.51\).
    model size N (log)final losssmall Clarge Coptima
    Schematic isoFLOP profiles: each budget C has a loss-optimal model size

3.3 Approach 3: Fitting a parametric loss function

  1. Give Chinchilla’s parametric loss model and the meaning of its terms.
    \(\hat L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}\): \(E\) is the loss of an ideal generative process (the entropy of natural text); \(A/N^\alpha\) is the shortfall of a finite \(N\)-parameter Transformer; \(B/D^\beta\) is the shortfall from finite training on \(D\) tokens. Fitted with a Huber loss on log loss via L-BFGS; minimizing under \(\text{FLOPs} \approx 6ND\) gives \(a = 0.46\), \(b = 0.54\).

3.4 Optimal model scaling

  1. How do the Chinchilla scaling exponents compare with Kaplan et al.’s?
    All three approaches give \(N_{\text{opt}} \propto C^{\approx 0.5}\) and \(D_{\text{opt}} \propto C^{\approx 0.5}\) (equal scaling), whereas Kaplan et al. had \(N \propto C^{0.73}\), \(D \propto C^{0.27}\); so existing large models were oversized for their budgets, and far more data is needed.
    methoda (N)b (D)
    1. training curve envelope0.500.50
    2. IsoFLOP profiles0.490.51
    3. parametric loss fit0.460.54
    Kaplan et al. (2020)0.730.27
  2. Check that Chinchilla matches Gopher’s compute budget, and give the tokens-per-parameter ratio it implies.
    With \(C \approx 6ND\): \(6 \times 7 \times 10^{10} \times 1.4 \times 10^{12} \approx 5.9 \times 10^{23}\) FLOPs, matching Gopher’s \(5.76 \times 10^{23}\). That is \(1.4 \times 10^{12} / 7 \times 10^{10} = 20\) tokens per parameter.

4 Chinchilla

  1. Why was Chinchilla built with 70B parameters?
    The analysis put the optimal size for Gopher’s budget between 40B and 70B parameters; 70B on 1.4T tokens was chosen at the larger end of that range for data and computational efficiency reasons.

4.1 Model and training details

  1. How did Chinchilla’s training differ from Gopher’s besides size and data?
    Same architecture and MassiveText data (with a different subset mix for more tokens), but AdamW instead of Adam, a SentencePiece tokenizer without NFKC normalization (better for maths and chemistry), and a float32 copy of the weights in the optimizer state alongside bfloat16 computation.

4.2 Results

  1. How did Chinchilla do on MMLU?
    67.5% average 5-shot accuracy, over 7 points above Gopher, beating an expert forecast for June 2023 (63.4%), with over 90% on four individual subjects.
  2. How did Chinchilla compare with Gopher on other benchmarks?
    Better on every subset of The Pile, Wikitext103 perplexity 7.16 vs 7.75, LAMBADA 77.4% vs 74.5%, RACE-h and RACE-m by over 10 points, BIG-bench 65.1% vs 54.4%, common-sense benchmarks, closed-book QA (Natural Questions 5-shot 31.5% vs 21%), and TruthfulQA 0-shot 43.6% vs 29.5%.
  3. What did Chinchilla’s TruthfulQA result suggest?
    Contrary to earlier findings that bigger models aren’t more truthful, better modelling of the pre-training data alone (14.1 points more 0-shot accuracy than Gopher) substantially improved the benchmark.

5 Discussion & Conclusion

  1. What limitations of the Chinchilla analysis did its authors note?
    Only two comparable large-scale runs (Chinchilla and Gopher) with no intermediate scales; the assumption of a power-law frontier, though there is some concavity at high compute, so optimal sizes may still be overestimated; and all runs used less than one epoch of data.
  2. What implications for data did the Chinchilla authors draw?
    Scaling requires much larger, high-quality data sets, collected responsibly with attention to quality, train–test overlap, toxicity, bias and private information, which become more prevalent in absolute terms as data grows.

Source

  • Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack W. Rae and Laurent Sifre. 2022. Training Compute-Optimal Large Language Models. In Advances in Neural Information Processing Systems 35 (NeurIPS 2022). https://papers.nips.cc/paper_files/paper/2022/hash/c1e2faff6f588870935f114ebe04a3e5-Abstract-Conference.html