learn.ifs.name

LLaMA: Open and Efficient Foundation Language Models

Touvron et al., 2023, LLaMA (arXiv preprint arXiv:2302.13971)

Subscriber deck — free during early access

1 cards in “2.2 Architecture” · whole deck

2.2 Architecture

  1. Which three changes did LLaMA make to the original Transformer, and where did each come from?
    Pre-normalization with RMSNorm on each sub-layer’s input for training stability (from GPT-3); the SwiGLU activation instead of ReLU (from PaLM); and rotary positional embeddings (RoPE) at every layer instead of absolute position embeddings (from GPT-Neo).
    • Touvron et al., 2023, LLaMA (arXiv preprint arXiv:2302.13971), §2.2

Source

  • Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. In arXiv preprint arXiv:2302.13971. https://arxiv.org/abs/2302.13971