learn.ifs.name

The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track)

Subscriber deck — free during early access

2 cards in “4 FineWeb-Edu” · whole deck

4 FineWeb-Edu

  1. How was FineWeb-Edu’s educational classifier built?
    Llama-3-70B-Instruct rated 460K web pages for educational value on an additive 0–5 scale (focused on grade- and middle-school knowledge, to avoid favouring arXiv-style text); a linear regressor on frozen Snowflake-arctic-embed embeddings learned these scores; pages scoring ≥ 3 were kept (F1 82% at that threshold).
    • Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §4
  2. How much did FineWeb-Edu help, and what topics did it shift?
    With 350B training tokens MMLU rose from 33% to 37% and ARC from 46% to 57%; it matched the next best dataset’s MMLU with about 10× fewer tokens. It favours education, history and culture, and down-samples business, entertainment and travel.
    • Penedo et al., 2024, The FineWeb Datasets (Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track), §4

Source