The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track)
Subscriber deck — free during early access
1 cards in “3.2 Text extraction” · whole deck
3.2 Text extraction
- Why did FineWeb extract text from WARC files instead of using Common Crawl’s WET text?WET files keep too much boilerplate and menu text; extracting from the raw HTML (WARC) with trafilatura is costlier but gave clearly better models.
- Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §3.2
Source
- Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra and Thomas Wolf. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track. https://papers.nips.cc/paper_files/paper/2024/hash/370df50ccfdf8bde18f8f9c2d9151bda-Abstract-Datasets_and_Benchmarks_Track.html