The Llama 3 Herd of Models
Meta, 2024, The Llama 3 Herd of Models (arXiv preprint arXiv:2407.21783)
Subscriber deck — free during early access
1 cards in “3.2 Model Architecture” · whole deck
3.2 Model Architecture
- What architectural changes did Llama 3 make relative to Llama 2?Grouped-query attention with 8 key–value heads (faster inference, smaller KV cache); an attention mask preventing attention across documents packed in one sequence; a 128K-token vocabulary (100K tiktoken + 28K non-English tokens, 3.94 vs 3.17 characters per token on English); and RoPE base frequency raised to 500,000 for long contexts.
- Meta, 2024, The Llama 3 Herd of Models (arXiv preprint arXiv:2407.21783), §3.2
Source
- Llama Team, AI @ Meta. 2024. The Llama 3 Herd of Models. In arXiv preprint arXiv:2407.21783. https://arxiv.org/abs/2407.21783