The Llama 3 Herd of Models
Meta, 2024, The Llama 3 Herd of Models (arXiv preprint arXiv:2407.21783)
Subscriber deck — free during early access
1 cards in “3.2.1 Scaling Laws” · whole deck
3.2.1 Scaling Laws
- How did the Llama 3 team predict benchmark performance before training the flagship?In two steps: relate the compute-optimal models’ negative log-likelihood of correct answers to training FLOPs, then map that likelihood to accuracy with a sigmoid fitted on scaling-law models and older Llama 2 models. Extrapolating over four orders of magnitude, it predicted ARC-Challenge accuracy closely.
- Meta, 2024, The Llama 3 Herd of Models (arXiv preprint arXiv:2407.21783), §3.2.1
Source
- Llama Team, AI @ Meta. 2024. The Llama 3 Herd of Models. In arXiv preprint arXiv:2407.21783. https://arxiv.org/abs/2407.21783