Scaling Data-Constrained Language Models

7 Results: Complementary Strategies for Obtaining Additional Data

1. Can code substitute for missing natural-language data?
   Yes, up to a point: filling up to 50% of the data budget with Python code didn’t hurt natural-language task performance (even helping some tasks such as bAbI and WebNLG), while more code degraded it. Code thus doubles the usable data.
   • Muennighoff et al., 2023, Scaling Data-Constrained Language Models (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §7
2. What did Muennighoff et al. find about filtering when data is scarce?
   Perplexity filtering helped, but deduplication didn’t improve downstream tasks (though it may reduce memorization); filtering mainly pays off on noisy datasets, so it should be reserved for those.
   • Muennighoff et al., 2023, Scaling Data-Constrained Language Models (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §7
3. What practical recipe for scarce data do Muennighoff et al. give?
   Double the data with code, then repeat the new dataset for about 4 epochs: 8× more training tokens expected to be about as good as 8× more unique data.
   • Muennighoff et al., 2023, Scaling Data-Constrained Language Models (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §7