The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track)
Subscriber deck — free during early access
1 cards in “3.5 Adding C4’s filters” · whole deck
3.5 Adding C4’s filters
- Which of C4’s heuristic filters did FineWeb keep, and why not all?All except the terminal-punctuation filter: that one gave the biggest single boost on HellaSwag but removed about 30% of tokens; the others combined did better while removing only about 7%.
- Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §3.5
Source
- Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra and Thomas Wolf. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track. https://papers.nips.cc/paper_files/paper/2024/hash/370df50ccfdf8bde18f8f9c2d9151bda-Abstract-Datasets_and_Benchmarks_Track.html