Training Compute-Optimal Large Language Models

4.2 Results

1. How did Chinchilla do on MMLU?
   67.5% average 5-shot accuracy, over 7 points above Gopher, beating an expert forecast for June 2023 (63.4%), with over 90% on four individual subjects.
   • Hoffmann et al., 2022, Training Compute-Optimal Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §4.2
2. How did Chinchilla compare with Gopher on other benchmarks?
   Better on every subset of The Pile, Wikitext103 perplexity 7.16 vs 7.75, LAMBADA 77.4% vs 74.5%, RACE-h and RACE-m by over 10 points, BIG-bench 65.1% vs 54.4%, common-sense benchmarks, closed-book QA (Natural Questions 5-shot 31.5% vs 21%), and TruthfulQA 0-shot 43.6% vs 29.5%.
   • Hoffmann et al., 2022, Training Compute-Optimal Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §4.2
3. What did Chinchilla’s TruthfulQA result suggest?
   Contrary to earlier findings that bigger models aren’t more truthful, better modelling of the pre-training data alone (14.1 points more 0-shot accuracy than Gopher) substantially improved the benchmark.
   • Hoffmann et al., 2022, Training Compute-Optimal Large Language Models (Advances in Neural Information Processing Systems 35 (NeurIPS 2022)), §4.2