learn.ifs.name

Scaling Data-Constrained Language Models

Muennighoff et al., 2023, Scaling Data-Constrained Language Models (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))

Subscriber deck — free during early access

14 cards

1 Introduction

  1. What problem motivates data-constrained scaling laws?
    Chinchilla-style compute-optimal training needs ever more tokens (11 trillion for a 530B model), but text is finite: most languages are already data-constrained, and high-quality English text was projected to run out. So what should we do when we run out of unique data?
  2. Why was training on repeated data an open question for large language models?
    Most large models were trained for a single epoch and some work advised against reusing data; Galactica trained for 4.25 epochs and kept improving but wasn’t compared with a model trained on the same amount of unique data, so the value of repetition was unknown.
  3. What were Muennighoff et al.’s headline findings?
    With a fixed compute budget, up to about 4 epochs of repeated data are almost as good as unique data; beyond that, returns diminish until extra compute is worthless. Data-constrained compute should go to both more parameters and more epochs, with epochs scaled slightly faster. Mixing in code can double the effective data.

2 Background

  1. Which two questions about scaling do Muennighoff et al. separate, and what had Chinchilla answered?
    Allocation: how to split compute between parameters \(N\) and training tokens \(D\) (Chinchilla: scale both equally). Return: how much loss improves with more compute (a power law). Both assumed unlimited unique data.

3 Method

  1. How do Muennighoff et al. extend the Chinchilla loss formula to repeated data?
    They replace data \(D\) and parameters \(N\) with effective amounts: \(D' = U_D + U_D R_D^*(1 - e^{-R_D/R_D^*})\), where \(U_D\) is unique tokens and \(R_D\) the number of repetitions (epochs − 1). A symmetric formula discounts parameters beyond what the unique data can use. \(R_D^*\) and its parameter counterpart are fitted.
    161440epochs over the same datarepeated data(effective ÷ unique)fresh data
    Repeated tokens lose value: nearly as good as fresh up to about 4 epochs, then levelling off at about 16× (fitted R*D ≈ 15)
  2. What does the effective-data formula imply about repetition?
    For few repetitions (\(R_D \ll R_D^*\)), repeated tokens are worth almost as much as fresh ones; as repetitions grow, their value decays to zero, so no amount of repetition can beat the loss reachable with \(U_D + U_D R_D^*\) unique tokens.

4 Experimental Setup

  1. How were the data-constrained scaling experiments run?
    Over 400 GPT-2-style models from 10M to 9B parameters, trained for up to 1,500 epochs and 900B tokens on subsets of C4, always repeating the whole subset with shuffling, with cosine schedules and no early stopping; loss measured on held-out data.

5 Results: Resource Allocation for Data-Constrained Scaling

  1. With only 100M unique tokens, how should compute be spent?
    Far beyond the one-epoch optimum: training 20–60× more parameters and epochs (about 7,000× more compute) cut loss by more than half, so one-epoch models under-use their data.
  2. How does the data-constrained efficient frontier differ from Chinchilla’s?
    Excess parameters lose value faster than repeated data, so additional compute should go mostly to more epochs rather than equally to parameters and data. A model built this way with 27% fewer parameters beat the Chinchilla-recommended one for the same compute and data, and would also be cheaper at inference.

6 Results: Resource Return for Data-Constrained Scaling

  1. How much worse is training for 4 epochs than on unique data?
    Barely: an 8.7B model trained for four epochs on 44B unique tokens ended with only 0.5% higher validation loss than one trained for a single epoch on 178B unique tokens.
  2. Up to how many epochs does repetition pay off?
    The fitted \(R_D^* \approx 15\): meaningful gains up to about 16 epochs, after which returns diminish extremely fast and loss flattens; very long training (e.g. 44 epochs) can even diverge.

7 Results: Complementary Strategies for Obtaining Additional Data

  1. Can code substitute for missing natural-language data?
    Yes, up to a point: filling up to 50% of the data budget with Python code didn’t hurt natural-language task performance (even helping some tasks such as bAbI and WebNLG), while more code degraded it. Code thus doubles the usable data.
  2. What did Muennighoff et al. find about filtering when data is scarce?
    Perplexity filtering helped, but deduplication didn’t improve downstream tasks (though it may reduce memorization); filtering mainly pays off on noisy datasets, so it should be reserved for those.
  3. What practical recipe for scarce data do Muennighoff et al. give?
    Double the data with code, then repeat the new dataset for about 4 epochs: 8× more training tokens expected to be about as good as 8× more unique data.

Source