The Llama 3 Herd of Models
Meta, 2024, The Llama 3 Herd of Models (arXiv preprint arXiv:2407.21783)
Subscriber deck — free during early access
1 cards in “3.1.2 Determining the Data Mix” · whole deck
3.1.2 Determining the Data Mix
- What was Llama 3’s final pretraining data mix, and how was it chosen?About 50% general knowledge, 25% math and reasoning, 17% code and 8% multilingual tokens, chosen with a knowledge classifier (to downsample over-represented web categories like arts and entertainment) and scaling-law experiments that predict a large model’s performance from small models trained on each candidate mix.
- Meta, 2024, The Llama 3 Herd of Models (arXiv preprint arXiv:2407.21783), §3.1.2
Source
- Llama Team, AI @ Meta. 2024. The Llama 3 Herd of Models. In arXiv preprint arXiv:2407.21783. https://arxiv.org/abs/2407.21783