The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale
Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track)
Subscriber deck — free during early access
1 cards in “6 Conclusion” · whole deck
6 Conclusion
- What limitations of FineWeb do Penedo et al. acknowledge?Both datasets are web-only (books or transcripts might help); most ablations were at small scale due to compute; and evaluation used academic benchmarks on base models rather than settings reflecting how LLMs are actually used.
- Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §6
Source
- Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra and Thomas Wolf. 2024. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track. https://papers.nips.cc/paper_files/paper/2024/hash/370df50ccfdf8bde18f8f9c2d9151bda-Abstract-Datasets_and_Benchmarks_Track.html