Scaling Data-Constrained Language Models
Muennighoff et al., 2023, Scaling Data-Constrained Language Models (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))
Subscriber deck — free during early access
2 cards in “5 Results: Resource Allocation for Data-Constrained Scaling” · whole deck
5 Results: Resource Allocation for Data-Constrained Scaling
- With only 100M unique tokens, how should compute be spent?Far beyond the one-epoch optimum: training 20–60× more parameters and epochs (about 7,000× more compute) cut loss by more than half, so one-epoch models under-use their data.
- Muennighoff et al., 2023, Scaling Data-Constrained Language Models (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §5
- How does the data-constrained efficient frontier differ from Chinchilla’s?Excess parameters lose value faster than repeated data, so additional compute should go mostly to more epochs rather than equally to parameters and data. A model built this way with 27% fewer parameters beat the Chinchilla-recommended one for the same compute and data, and would also be cheaper at inference.
- Muennighoff et al., 2023, Scaling Data-Constrained Language Models (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §5
Source
- Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf and Colin Raffel. 2023. Scaling Data-Constrained Language Models. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://papers.nips.cc/paper_files/paper/2023/hash/9d89448b63ce1e2e8dc7af72c984c196-Abstract-Conference.html