learn.ifs.name

LLaMA: Open and Efficient Foundation Language Models

Touvron et al., 2023, LLaMA (arXiv preprint arXiv:2302.13971)

Subscriber deck — free during early access

1 cards in “6 Carbon footprint” · whole deck

6 Carbon footprint

  1. What did developing LLaMA cost in energy, and what argument do Touvron et al. make about release?
    About 2,048 A100s for roughly five months: an estimated 2,638 MWh and 1,015 tCO₂eq (at the US average carbon intensity). Releasing the trained models, some small enough for a single GPU, may reduce future emissions since others needn’t retrain them.
    • Touvron et al., 2023, LLaMA (arXiv preprint arXiv:2302.13971), §6

Source

  • Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. In arXiv preprint arXiv:2302.13971. https://arxiv.org/abs/2302.13971