LLaMA: Open and Efficient Foundation Language Models

A family of foundation language models from 7B to 65B parameters trained on trillions of tokens of publicly available data only. LLaMA-13B outperformed GPT-3 (175B) on most benchmarks, and LLaMA-65B was competitive with Chinchilla-70B and PaLM-540B.

Chinchilla optimizes performance for a training compute budget but ignores the inference budget, which dominates when serving a model at scale. A smaller model trained longer is costlier to train to a given level but cheaper to run; e.g. a 7B model kept improving even after 1T tokens, far beyond Chinchilla’s recommendation of 200B tokens for 10B.

It made the work compatible with open release. Competing models used private or undocumented data (e.g. “Books – 2TB”), and earlier open models such as OPT, BLOOM and GPT-NeoX weren’t competitive with PaLM-62B or Chinchilla.

About 1.4T tokens: CommonCrawl via CCNet 67%, C4 15%, GitHub 4.5%, Wikipedia 4.5%, books (Gutenberg, Books3) 4.5%, arXiv 2.5%, Stack Exchange 2%. Most tokens were seen once; Wikipedia and books about twice.

With the CCNet pipeline (line-level deduplication, fastText language identification, n-gram language model filtering), plus a linear classifier trained to tell pages cited as references in Wikipedia from random pages, discarding pages not classified as references.

BPE via SentencePiece, with all numbers split into individual digits and a fallback to bytes for unknown UTF-8 characters.

Pre-normalization with RMSNorm on each sub-layer’s input for training stability (from GPT-3); the SwiGLU activation instead of ReLU (from PaLM); and rotary positional embeddings (RoPE) at every layer instead of absolute position embeddings (from GPT-Neo).

A memory-efficient causal attention (xformers) that doesn’t store attention weights or compute masked scores; checkpointing that saves expensive activations such as linear-layer outputs via a hand-written backward pass; model and sequence parallelism; and overlapping computation with GPU communication. The 65B model processed about 380 tokens/s/GPU, about 21 days on 2,048 A100s for 1.4T tokens.

LLaMA-65B beat Chinchilla-70B on all eight benchmarks except BoolQ and PaLM-540B on all but BoolQ and WinoGrande; LLaMA-13B beat GPT-3 on most despite being 10× smaller.

LLaMA-65B reached state of the art on Natural Questions and TriviaQA in zero- and few-shot settings; LLaMA-13B was competitive with GPT-3 and Chinchilla while being 5–10× smaller and runnable on a single V100 GPU.

On GSM8k LLaMA-65B outperformed Minerva-62B, a PaLM model fine-tuned on 38.5B tokens of math and arXiv data, though LLaMA saw no special math data.

On MMLU (5-shot) it trailed Chinchilla-70B and PaLM-540B by a few points across most domains, probably because its books and academic papers totalled only 177 GB versus up to 2 TB of books for those models.

A single fine-tuning run following Flan’s protocol (LLaMA-I) raised MMLU from 63.4 to 68.9, beating other moderate-sized instruction-tuned models, though still well below code-davinci-002’s 77.4.

On RealToxicityPrompts, toxicity grew with model size, especially for “respectful” prompts asking for polite completions, consistent with earlier findings (Chinchilla vs Gopher being an exception).

Co-reference was much more accurate for “their/them/someone” than for “her/she” or “his/he”, and errors rose on “gotcha” sentences where the pronoun doesn’t match the occupation’s majority gender: the model uses occupational gender stereotypes rather than the sentence’s evidence.

LLaMA scored higher than GPT-3 on truthful and truthful-and-informative answers, but the rate of correct answers was still low, so the model is likely to hallucinate wrong answers.

About 2,048 A100s for roughly five months: an estimated 2,638 MWh and 1,015 tCO₂eq (at the US average carbon intensity). Releasing the trained models, some small enough for a single GPU, may reduce future emissions since others needn’t retrain them.