Scaling Laws for Neural Language Models
Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361)
Subscriber deck — free during early access
1 cards in “3.2.1 Comparing to LSTMs and Universal Transformers” · whole deck
3.2.1 Comparing to LSTMs and Universal Transformers
- How did Transformers compare with LSTMs in the scaling study?LSTMs matched Transformers on tokens early in the context but plateaued after under 100 tokens, while Transformers kept improving across the whole context, so Transformers asymptotically outperform LSTMs by using long contexts better.
- Kaplan et al., 2020, Scaling Laws for Neural Language Models (arXiv preprint arXiv:2001.08361), §3.2.1
Source
- Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu and Dario Amodei. 2020. Scaling Laws for Neural Language Models. In arXiv preprint arXiv:2001.08361. https://arxiv.org/abs/2001.08361