The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track)
Subscriber deck — free during early access
1 cards in “3.6 Developing additional heuristic filters” · whole deck
3.6 Developing additional heuristic filters
- How did Penedo et al. design new heuristic filters systematically?They computed over 50 document statistics on a high- and a low-quality version of the same crawl (individually vs globally deduplicated), set thresholds where the low-quality distribution dominated, and kept the 3 of 16 candidates that helped most: lines ending in punctuation ≤ 0.12, characters in duplicated lines ≥ 0.1, lines under 30 characters ≥ 0.67 (about 22% of tokens removed).
- Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §3.6
Source
- Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra and Thomas Wolf. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track. https://papers.nips.cc/paper_files/paper/2024/hash/370df50ccfdf8bde18f8f9c2d9151bda-Abstract-Datasets_and_Benchmarks_Track.html