Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research)
Subscriber deck — free during early access
1 cards in “3.6. Scaling” · whole deck
3.6. Scaling
- Given 4× more compute, how should you spend it, according to T5?Training longer, using bigger batches, bigger models and ensembles all helped. A larger model usually added more than just longer training, 2× size for 2× steps was about as good as 4× size, and ensembling was an orthogonal gain (best on some tasks). The trade-offs differ: bigger models cost more at fine-tuning and inference time.
- Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 3.6. Scaling, pp. 34–35
Source
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. In Journal of Machine Learning Research, pp. 1–67. License: CC BY 4.0. Flashcards adapted from this work. https://jmlr.org/papers/v21/20-074.html