The Llama 3 Herd of Models

Data (about 15T multilingual pretraining tokens vs 1.8T for Llama 2, with better curation and post-training data quality control); scale (3.8 × 10²⁵ FLOPs for the 405B flagship, almost 50× the largest Llama 2); and managing complexity (a standard dense Transformer and a simple SFT + rejection sampling + DPO post-training recipe).

To keep development stable and scalable: MoE was avoided for training stability, and complex RL algorithms such as PPO tend to be less stable and harder to scale; DPO also needed less compute and did better on instruction following.

Models with 8B, 70B and 405B parameters, supporting multilinguality, coding, reasoning and tool use with up to 128K-token context. The 405B model performed on par with leading models such as GPT-4 on many tasks; the smaller ones were best in their size class.

PII and safety filtering of domains; a custom HTML parser that preserves math and code (and keeps image alt text); URL-, document- (MinHash) and line-level deduplication; heuristic filters (duplicated n-grams, dirty words, token-distribution outliers); model-based quality classifiers trained on Llama 2 judgments; and dedicated code, math and multilingual pipelines.

Markdown markers were removed, since markdown hurt a model trained mostly on web text compared with plain text; and aggressive line-level deduplication (dropping lines seen more than 6 times per 30M documents) helped despite also removing some frequent high-quality text.

About 50% general knowledge, 25% math and reasoning, 17% code and 8% multilingual tokens, chosen with a knowledge classifier (to downsample over-represented web categories like arts and entertainment) and scaling-law experiments that predict a large model’s performance from small models trained on each candidate mix.

Take a half-trained 8B model, anneal its learning rate to zero over 40B tokens with 30% of the mix from the new dataset, and measure the effect: much cheaper than running scaling-law experiments for every small dataset. Annealing on high-quality math helped the 8B model a lot but the 405B model negligibly.

Grouped-query attention with 8 key–value heads (faster inference, smaller KV cache); an attention mask preventing attention across documents packed in one sequence; a 128K-token vocabulary (100K tiktoken + 28K non-English tokens, 3.94 vs 3.17 characters per token on English); and RoPE base frequency raised to 500,000 for long contexts.

In two steps: relate the compute-optimal models’ negative log-likelihood of correct answers to training FLOPs, then map that likelihood to accuracy with a sigmoid fitted on scaling-law models and older Llama 2 models. Extrapolating over four orders of magnitude, it predicted ARC-Challenge accuracy closely.

In a 54-day period there were 466 interruptions (419 unexpected, about 78% hardware-related, GPUs alone 58.7%), yet automation kept effective training time above 90%, with significant manual intervention needed only three times.

Only at the end of pretraining, since attention cost grows quadratically: in six stages from 8K to 128K using about 800B tokens, moving on once short-context performance had fully recovered and “needle in a haystack” retrieval was perfect at the current length.

Train a reward model on human preference data; use it for rejection sampling (pick the best of several generated responses) on human prompts; supervised fine-tuning on that plus other and synthetic data; then DPO on fresh preference data from the latest models; finally average models from different runs. Six such rounds were done.

Masking special formatting tokens (headers, terminators) from the DPO loss, since tokens shared by chosen and rejected responses get conflicting gradients and caused tail repetition or abrupt termination; and adding a negative log-likelihood term (weight 0.2) on chosen responses to preserve formatting.

No testing can find every risk: the models may still produce harmful content, especially in languages other than English or under skilled adversarial prompting, and new jailbreaks will be found; they release system-level safety tools (such as Llama Guard 3) for developers.