LLaMA: Open and Efficient Foundation Language Models

1 Introduction

1. What is LLaMA, and what were its two headline claims?
   A family of foundation language models from 7B to 65B parameters trained on trillions of tokens of publicly available data only. LLaMA-13B outperformed GPT-3 (175B) on most benchmarks, and LLaMA-65B was competitive with Chinchilla-70B and PaLM-540B.
   • Touvron et al., 2023, LLaMA (arXiv preprint arXiv:2302.13971), §1
2. Why did LLaMA train smaller models on more tokens than the Chinchilla scaling laws recommend?
   Chinchilla optimizes performance for a training compute budget but ignores the inference budget, which dominates when serving a model at scale. A smaller model trained longer is costlier to train to a given level but cheaper to run; e.g. a 7B model kept improving even after 1T tokens, far beyond Chinchilla’s recommendation of 200B tokens for 10B.
   • Touvron et al., 2023, LLaMA (arXiv preprint arXiv:2302.13971), §1
3. Why did using only public data matter for LLaMA?
   It made the work compatible with open release. Competing models used private or undocumented data (e.g. “Books – 2TB”), and earlier open models such as OPT, BLOOM and GPT-NeoX weren’t competitive with PaLM-62B or Chinchilla.
   • Touvron et al., 2023, LLaMA (arXiv preprint arXiv:2302.13971), §1