Scaling Data-Constrained Language Models
Muennighoff et al., 2023, Scaling Data-Constrained Language Models (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))
Subscriber deck — free during early access
2 cards in “6 Results: Resource Return for Data-Constrained Scaling” · whole deck
6 Results: Resource Return for Data-Constrained Scaling
- How much worse is training for 4 epochs than on unique data?Barely: an 8.7B model trained for four epochs on 44B unique tokens ended with only 0.5% higher validation loss than one trained for a single epoch on 178B unique tokens.
- Muennighoff et al., 2023, Scaling Data-Constrained Language Models (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §6
- Up to how many epochs does repetition pay off?The fitted \(R_D^* \approx 15\): meaningful gains up to about 16 epochs, after which returns diminish extremely fast and loss flattens; very long training (e.g. 44 epochs) can even diverge.
- Muennighoff et al., 2023, Scaling Data-Constrained Language Models (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §6
Source
- Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf and Colin Raffel. 2023. Scaling Data-Constrained Language Models. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://papers.nips.cc/paper_files/paper/2023/hash/9d89448b63ce1e2e8dc7af72c984c196-Abstract-Conference.html