LLaMA: Open and Efficient Foundation Language Models
Touvron et al., 2023, LLaMA (arXiv preprint arXiv:2302.13971)
Subscriber deck — free during early access
1 cards in “2.4 Efficient implementation” · whole deck
2.4 Efficient implementation
- How did LLaMA’s authors make training efficient?A memory-efficient causal attention (xformers) that doesn’t store attention weights or compute masked scores; checkpointing that saves expensive activations such as linear-layer outputs via a hand-written backward pass; model and sequence parallelism; and overlapping computation with GPU communication. The 65B model processed about 380 tokens/s/GPU, about 21 days on 2,048 A100s for 1.4T tokens.
- Touvron et al., 2023, LLaMA (arXiv preprint arXiv:2302.13971), §2.4
Source
- Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. In arXiv preprint arXiv:2302.13971. https://arxiv.org/abs/2302.13971