learn.ifs.name

LLaMA: Open and Efficient Foundation Language Models

Touvron et al., 2023, LLaMA (arXiv preprint arXiv:2302.13971)

Subscriber deck — free during early access

3 cards in “2.1 Pre-training Data” · whole deck

2.1 Pre-training Data

  1. What was LLaMA’s pre-training mixture?
    About 1.4T tokens: CommonCrawl via CCNet 67%, C4 15%, GitHub 4.5%, Wikipedia 4.5%, books (Gutenberg, Books3) 4.5%, arXiv 2.5%, Stack Exchange 2%. Most tokens were seen once; Wikipedia and books about twice.
    CommonCrawl 67%C418%C4: 15%other 18%:GitHub 4.5, Wikipedia 4.5,books 4.5, arXiv 2.5,Stack Exchange 2
    LLaMA’s pre-training mixture (≈ 1.4T tokens, all from public sources)
    • Touvron et al., 2023, LLaMA (arXiv preprint arXiv:2302.13971), §2.1
  2. How did LLaMA filter CommonCrawl for quality?
    With the CCNet pipeline (line-level deduplication, fastText language identification, n-gram language model filtering), plus a linear classifier trained to tell pages cited as references in Wikipedia from random pages, discarding pages not classified as references.
    • Touvron et al., 2023, LLaMA (arXiv preprint arXiv:2302.13971), §2.1
  3. What is distinctive about LLaMA’s tokenizer?
    BPE via SentencePiece, with all numbers split into individual digits and a fallback to bytes for unknown UTF-8 characters.
    • Touvron et al., 2023, LLaMA (arXiv preprint arXiv:2302.13971), §2.1

Source

  • Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. In arXiv preprint arXiv:2302.13971. https://arxiv.org/abs/2302.13971