The Llama 3 Herd of Models

Meta, 2024, The Llama 3 Herd of Models (arXiv preprint arXiv:2407.21783)

Subscriber deck — free during early access

1 cards in “3.3.4 Reliability and Operational Challenges” · whole deck

3.3.4 Reliability and Operational Challenges

How reliable was training on 16K GPUs?
In a 54-day period there were 466 interruptions (419 unexpected, about 78% hardware-related, GPUs alone 58.7%), yet automation kept effective training time above 90%, with significant manual intervention needed only three times.
- Meta, 2024, The Llama 3 Herd of Models (arXiv preprint arXiv:2407.21783), §3.3.4

Source

Llama Team, AI @ Meta. 2024. The Llama 3 Herd of Models. In arXiv preprint arXiv:2407.21783. https://arxiv.org/abs/2407.21783