Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research)
Subscriber deck — free during early access
1 cards in “2.1. Model” · whole deck
2.1. Model
- How does T5’s Transformer differ from the original Transformer?Only in details: layer normalization without bias, applied outside the residual path (to each subcomponent’s input), and simplified relative position embeddings: a learned scalar per bucket of key–query offsets added to the attention logits, shared across layers, 32 buckets growing logarithmically up to offset 128.
- Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 2.1. Model, pp. 4–5
Source
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. In Journal of Machine Learning Research, pp. 1–67. License: CC BY 4.0. Flashcards adapted from this work. https://jmlr.org/papers/v21/20-074.html