Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research)
Subscriber deck — free during early access
1 cards in “4.2. Outlook” · whole deck
4.2. Outlook
- Which open problems did the T5 authors highlight?The inconvenience of large models (need cheaper models via distillation, parameter sharing, conditional computation), more efficient ways to extract knowledge than denoising on a trillion tokens, formalizing similarity between pre-training and downstream tasks, and language-agnostic models.
- Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 4.2. Outlook, pp. 42–43
Source
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. In Journal of Machine Learning Research, pp. 1–67. License: CC BY 4.0. Flashcards adapted from this work. https://jmlr.org/papers/v21/20-074.html