Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research)
Subscriber deck — free during early access
1 cards in “3. Experiments” · whole deck
3. Experiments
- How did the T5 study explore the design space?“Coordinate ascent”: start from a reasonable baseline and change one factor at a time (architecture, objective, data, training strategy, scale), which may miss interactions but avoids an infeasible combinatorial search.
- Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 3. Experiments, p. 10
Source
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. In Journal of Machine Learning Research, pp. 1–67. License: CC BY 4.0. Flashcards adapted from this work. https://jmlr.org/papers/v21/20-074.html