The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track)
Subscriber deck — free during early access
2 cards in “1 Introduction” · whole deck
1 Introduction
- What gap does FineWeb address?Pretraining data strongly shapes model quality, but the datasets of leading “open” models (Llama 3, Mixtral) are unreleased and barely documented. FineWeb is a 15-trillion-token dataset from 96 Common Crawl snapshots, released with every design choice ablated and documented.
- Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §1
- What are FineWeb and FineWeb-Edu?FineWeb: 15T tokens of filtered, deduplicated English web text that trains better models than other open web datasets. FineWeb-Edu: a 1.3T-token subset rated highly educational by a classifier, giving much better MMLU and ARC scores. Both are released under ODC-By with the datatrove code and ablation models.
- Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §1
Source
- Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra and Thomas Wolf. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track. https://papers.nips.cc/paper_files/paper/2024/hash/370df50ccfdf8bde18f8f9c2d9151bda-Abstract-Datasets_and_Benchmarks_Track.html