The Llama 3 Herd of Models

1 Introduction

1. What are the three “levers” the Llama 3 team optimized, and how?
   Data (about 15T multilingual pretraining tokens vs 1.8T for Llama 2, with better curation and post-training data quality control); scale (3.8 × 10²⁵ FLOPs for the 405B flagship, almost 50× the largest Llama 2); and managing complexity (a standard dense Transformer and a simple SFT + rejection sampling + DPO post-training recipe).
   • Meta, 2024, The Llama 3 Herd of Models (arXiv preprint arXiv:2407.21783), §1
2. Why did Llama 3 choose a dense Transformer and DPO rather than mixture-of-experts and PPO?
   To keep development stable and scalable: MoE was avoided for training stability, and complex RL algorithms such as PPO tend to be less stable and harder to scale; DPO also needed less compute and did better on instruction following.
   • Meta, 2024, The Llama 3 Herd of Models (arXiv preprint arXiv:2407.21783), §1
3. What does the Llama 3 herd consist of, and how good was the flagship?
   Models with 8B, 70B and 405B parameters, supporting multilinguality, coding, reasoning and tool use with up to 128K-token context. The 405B model performed on par with leading models such as GPT-4 on many tasks; the smaller ones were best in their size class.
   • Meta, 2024, The Llama 3 Herd of Models (arXiv preprint arXiv:2407.21783), §1