The Llama 3 Herd of Models

Meta, 2024, The Llama 3 Herd of Models (arXiv preprint arXiv:2407.21783)

Subscriber deck — free during early access

1 cards in “3.4.2 Long Context Pre-Training” · whole deck

3.4.2 Long Context Pre-Training

How was Llama 3’s context window extended to 128K tokens?
Only at the end of pretraining, since attention cost grows quadratically: in six stages from 8K to 128K using about 800B tokens, moving on once short-context performance had fully recovered and “needle in a haystack” retrieval was perfect at the current length.
- Meta, 2024, The Llama 3 Herd of Models (arXiv preprint arXiv:2407.21783), §3.4.2

Source

Llama Team, AI @ Meta. 2024. The Llama 3 Herd of Models. In arXiv preprint arXiv:2407.21783. https://arxiv.org/abs/2407.21783