learn.ifs.name

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Shao et al., 2024, DeepSeekMath (arXiv preprint arXiv:2402.03300)

Subscriber deck — free during early access

2 cards in “2.1 Data Collection and Decontamination” · whole deck

2.1 Data Collection and Decontamination

  1. How was the 120B-token DeepSeekMath Corpus collected?
    Iteratively: train a fastText classifier with OpenWebMath as positives and random Common Crawl pages as negatives, recall and rank math pages from 40B deduplicated pages; find web domains where over 10% of pages were collected, hand-annotate their math URLs, add uncollected pages to the seeds and retrain. After four rounds (the fourth adding almost nothing): 35.5M pages.
    • Shao et al., 2024, DeepSeekMath (arXiv preprint arXiv:2402.03300), §2.1
  2. How was benchmark contamination avoided in the DeepSeekMath Corpus?
    Any text segment sharing an exact 10-gram with GSM8K, MATH, CMATH, AGIEval and similar benchmarks was removed (exact matching for shorter benchmark texts of at least 3 grams).
    • Shao et al., 2024, DeepSeekMath (arXiv preprint arXiv:2402.03300), §2.1

Source

  • Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu and Daya Guo. 2024. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. In arXiv preprint arXiv:2402.03300. https://arxiv.org/abs/2402.03300