Scaling Data-Constrained Language Models
Muennighoff et al., 2023, Scaling Data-Constrained Language Models (Advances in Neural Information Processing Systems 36 (NeurIPS 2023))
Subscriber deck — free during early access
2 cards in “3 Method” · whole deck
3 Method
- How do Muennighoff et al. extend the Chinchilla loss formula to repeated data?They replace data \(D\) and parameters \(N\) with effective amounts: \(D' = U_D + U_D R_D^*(1 - e^{-R_D/R_D^*})\), where \(U_D\) is unique tokens and \(R_D\) the number of repetitions (epochs − 1). A symmetric formula discounts parameters beyond what the unique data can use. \(R_D^*\) and its parameter counterpart are fitted.
Repeated tokens lose value: nearly as good as fresh up to about 4 epochs, then levelling off at about 16× (fitted R*D ≈ 15) - Muennighoff et al., 2023, Scaling Data-Constrained Language Models (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §3
- What does the effective-data formula imply about repetition?For few repetitions (\(R_D \ll R_D^*\)), repeated tokens are worth almost as much as fresh ones; as repetitions grow, their value decays to zero, so no amount of repetition can beat the loss reachable with \(U_D + U_D R_D^*\) unique tokens.
- Muennighoff et al., 2023, Scaling Data-Constrained Language Models (Advances in Neural Information Processing Systems 36 (NeurIPS 2023)), §3
Source
- Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf and Colin Raffel. 2023. Scaling Data-Constrained Language Models. In Advances in Neural Information Processing Systems 36 (NeurIPS 2023). https://papers.nips.cc/paper_files/paper/2023/hash/9d89448b63ce1e2e8dc7af72c984c196-Abstract-Conference.html