Scaling Data-Constrained Language Models
Muennighoff et al., 2023, Scaling Data-Constrained Language Models (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))
Subscriber deck — free during early access
1 cards in “2 Background” · whole deck
2 Background
- Which two questions about scaling do Muennighoff et al. separate, and what had Chinchilla answered?Allocation: how to split compute between parameters \(N\) and training tokens \(D\) (Chinchilla: scale both equally). Return: how much loss improves with more compute (a power law). Both assumed unlimited unique data.
- Muennighoff et al., 2023, Scaling Data-Constrained Language Models (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §2
Source
- Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf and Colin Raffel. 2023. Scaling Data-Constrained Language Models. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://papers.nips.cc/paper_files/paper/2023/hash/9d89448b63ce1e2e8dc7af72c984c196-Abstract-Conference.html