learn.ifs.name

The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track)

Subscriber deck — free during early access

14 cards

1 Introduction

  1. What gap does FineWeb address?
    Pretraining data strongly shapes model quality, but the datasets of leading “open” models (Llama 3, Mixtral) are unreleased and barely documented. FineWeb is a 15-trillion-token dataset from 96 Common Crawl snapshots, released with every design choice ablated and documented.
    • Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §1
  2. What are FineWeb and FineWeb-Edu?
    FineWeb: 15T tokens of filtered, deduplicated English web text that trains better models than other open web datasets. FineWeb-Edu: a 1.3T-token subset rated highly educational by a classifier, giving much better MMLU and ARC scores. Both are released under ODC-By with the datatrove code and ablation models.
    • Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §1

2 Background

  1. Why is curating web text for pretraining a trade-off?
    Web text contains “unnatural” language (boilerplate, gibberish) and much duplication that harm models, but filtering too aggressively leaves too little data for single-pass pretraining; and deduplication involves many choices (granularity, exact or fuzzy).
    • Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §2

3.1 Experimental setup

  1. How were FineWeb’s design choices evaluated?
    By “data ablations”: identical 1.71B Llama-architecture models trained on equal-sized random samples (usually 28B tokens, some 350B) of each data version, two seeds each, evaluated on benchmarks chosen for low run-to-run variance, monotonic improvement during training and above-chance scores at this scale.
    • Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §3.1

3.2 Text extraction

  1. Why did FineWeb extract text from WARC files instead of using Common Crawl’s WET text?
    WET files keep too much boilerplate and menu text; extracting from the raw HTML (WARC) with trafilatura is costlier but gave clearly better models.
    • Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §3.2

3.4 Deduplication

  1. How does FineWeb’s MinHash deduplication work?
    Each document’s word 5-grams are hashed with 112 hash functions in 14 buckets of 8; documents agreeing on all 8 hashes of any bucket (targeting ≥ 75% similarity) are duplicates, clusters are formed transitively, and one document per cluster is kept.
    • Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §3.4
  2. Why did global deduplication across all snapshots disappoint?
    It shrank the data to 4T tokens but barely improved models. In an old snapshot, the 10% of data that survived deduplication against all others was worse (ads, keyword lists, bad formatting) than the 90% removed: global deduplication upsamples low-quality unique text.
    • Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §3.4
  3. What deduplication strategy did FineWeb adopt instead?
    Deduplicating each snapshot independently, leaving 20T tokens and matching RefinedWeb’s performance. The main benefit seems to come from removing huge duplicate clusters within crawls; removing small clusters across crawls can hurt.
    WARC → text (trafilatura)base filters → ~36T tokensMinHash per snapshot → ~20TC4 filters (no terminal punct.)3 custom filters + PII removalFineWeb15T tokensFineWeb-Edu1.3T, edu ≥ 3
    Each step was kept only if it improved small ablation models
    • Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §3.4

3.5 Adding C4’s filters

  1. Which of C4’s heuristic filters did FineWeb keep, and why not all?
    All except the terminal-punctuation filter: that one gave the biggest single boost on HellaSwag but removed about 30% of tokens; the others combined did better while removing only about 7%.
    • Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §3.5

3.6 Developing additional heuristic filters

  1. How did Penedo et al. design new heuristic filters systematically?
    They computed over 50 document statistics on a high- and a low-quality version of the same crawl (individually vs globally deduplicated), set thresholds where the low-quality distribution dominated, and kept the 3 of 16 candidates that helped most: lines ending in punctuation ≤ 0.12, characters in duplicated lines ≥ 0.1, lines under 30 characters ≥ 0.67 (about 22% of tokens removed).
    • Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §3.6

4 FineWeb-Edu

  1. How was FineWeb-Edu’s educational classifier built?
    Llama-3-70B-Instruct rated 460K web pages for educational value on an additive 0–5 scale (focused on grade- and middle-school knowledge, to avoid favouring arXiv-style text); a linear regressor on frozen Snowflake-arctic-embed embeddings learned these scores; pages scoring ≥ 3 were kept (F1 82% at that threshold).
    • Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §4
  2. How much did FineWeb-Edu help, and what topics did it shift?
    With 350B training tokens MMLU rose from 33% to 37% and ARC from 46% to 57%; it matched the next best dataset’s MMLU with about 10× fewer tokens. It favours education, history and culture, and down-samples business, entertainment and travel.
    • Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §4

5 Bias analyses

  1. What biases did the FineWeb bias analysis find?
    Overrepresentation of words reflecting hegemonic norms (e.g. “man”, “christian”), with the most skewed associations between religion words and dating (e.g. “christian dating”); FineWeb-Edu’s associations looked more like educational history and health content.
    • Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §5

6 Conclusion

  1. What limitations of FineWeb do Penedo et al. acknowledge?
    Both datasets are web-only (books or transcripts might help); most ablations were at small scale due to compute; and evaluation used academic benchmarks on base models rather than settings reflecting how LLMs are actually used.
    • Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §6

Source