The Llama 3 Herd of Models
Meta, 2024, The Llama 3 Herd of Models (arXiv preprint arXiv:2407.21783)
Subscriber deck — free during early access
1 cards in “3.1.3 Annealing Data” · whole deck
3.1.3 Annealing Data
- How did Llama 3 use annealing to judge the value of a small dataset?Take a half-trained 8B model, anneal its learning rate to zero over 40B tokens with 30% of the mix from the new dataset, and measure the effect: much cheaper than running scaling-law experiments for every small dataset. Annealing on high-quality math helped the 8B model a lot but the 405B model negligibly.
- Meta, 2024, The Llama 3 Herd of Models (arXiv preprint arXiv:2407.21783), §3.1.3
Source
- Llama Team, AI @ Meta. 2024. The Llama 3 Herd of Models. In arXiv preprint arXiv:2407.21783. https://arxiv.org/abs/2407.21783