Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

3.7. Putting It All Together

1. What went into the final T5 models?
   Span corruption (mean span 3, 15%), about 1 trillion pre-training tokens of C4 (1M steps with batches of 2,048 sequences), multi-task pre-training followed by per-task fine-tuning, beam search for translation and summarization, and five sizes: Small 60M, Base 220M, Large 770M, 3B and 11B parameters.
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 3.7. Putting It All Together, pp. 36–40
2. What did T5-11B achieve?
   State of the art on 18 of 24 tasks: GLUE 90.3, SuperGLUE 88.9 (human 89.8), SQuAD EM over a point above the previous best, CNN/Daily Mail ROUGE; but not on WMT translation, probably because pre-training was English-only and top MT systems used back-translation.
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 3.7. Putting It All Together, pp. 36–40
3. Was T5’s success just scale?
   No: T5-Base clearly beat the same baseline trained on the same 1 trillion tokens on every task, so the non-scaling choices from the systematic study also contributed.
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 3.7. Putting It All Together, pp. 36–40