The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

3.4 Deduplication

1. How does FineWeb’s MinHash deduplication work?
   Each document’s word 5-grams are hashed with 112 hash functions in 14 buckets of 8; documents agreeing on all 8 hashes of any bucket (targeting ≥ 75% similarity) are duplicates, clusters are formed transitively, and one document per cluster is kept.
   • Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §3.4
2. Why did global deduplication across all snapshots disappoint?
   It shrank the data to 4T tokens but barely improved models. In an old snapshot, the 10% of data that survived deduplication against all others was worse (ads, keyword lists, bad formatting) than the 90% removed: global deduplication upsamples low-quality unique text.
   • Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §3.4
3. What deduplication strategy did FineWeb adopt instead?
   Deduplicating each snapshot independently, leaving 20T tokens and matching RefinedWeb’s performance. The main benefit seems to come from removing huge duplicate clusters within crawls; removing small clusters across crawls can hurt.

   • Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §3.4