Scaling Data-Constrained Language Models
Muennighoff et al., 2023, Scaling Data-Constrained Language Models (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))
Subscriber deck — free during early access
1 cards in “4 Experimental Setup” · whole deck
4 Experimental Setup
- How were the data-constrained scaling experiments run?Over 400 GPT-2-style models from 10M to 9B parameters, trained for up to 1,500 epochs and 900B tokens on subsets of C4, always repeating the whole subset with shuffling, with cosine schedules and no early stopping; loss measured on held-out data.
- Muennighoff et al., 2023, Scaling Data-Constrained Language Models (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §4
Source
- Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf and Colin Raffel. 2023. Scaling Data-Constrained Language Models. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://papers.nips.cc/paper_files/paper/2023/hash/9d89448b63ce1e2e8dc7af72c984c196-Abstract-Conference.html