Scaling Data-Constrained Language Models

1 Introduction

1. What problem motivates data-constrained scaling laws?
   Chinchilla-style compute-optimal training needs ever more tokens (11 trillion for a 530B model), but text is finite: most languages are already data-constrained, and high-quality English text was projected to run out. So what should we do when we run out of unique data?
   • Muennighoff et al., 2023, Scaling Data-Constrained Language Models (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §1
2. Why was training on repeated data an open question for large language models?
   Most large models were trained for a single epoch and some work advised against reusing data; Galactica trained for 4.25 epochs and kept improving but wasn’t compared with a model trained on the same amount of unique data, so the value of repetition was unknown.
   • Muennighoff et al., 2023, Scaling Data-Constrained Language Models (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §1
3. What were Muennighoff et al.’s headline findings?
   With a fixed compute budget, up to about 4 epochs of repeated data are almost as good as unique data; beyond that, returns diminish until extra compute is worthless. Data-constrained compute should go to both more parameters and more epochs, with epochs scaled slightly faster. Mixing in code can double the effective data.
   • Muennighoff et al., 2023, Scaling Data-Constrained Language Models (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §1