The Llama 3 Herd of Models

3.1.1 Web Data Curation

1. What steps did Llama 3’s web data pipeline include?
   PII and safety filtering of domains; a custom HTML parser that preserves math and code (and keeps image alt text); URL-, document- (MinHash) and line-level deduplication; heuristic filters (duplicated n-grams, dirty words, token-distribution outliers); model-based quality classifiers trained on Llama 2 judgments; and dedicated code, math and multilingual pipelines.
   • Meta, 2024, The Llama 3 Herd of Models (arXiv preprint arXiv:2407.21783), §3.1.1
2. What surprising choice did Llama 3 make about markdown and repeated lines in web text?
   Markdown markers were removed, since markdown hurt a model trained mostly on web text compared with plain text; and aggressive line-level deduplication (dropping lines seen more than 6 times per 30M documents) helped despite also removing some frequent high-quality text.
   • Meta, 2024, The Llama 3 Herd of Models (arXiv preprint arXiv:2407.21783), §3.1.1