Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

3.1. Baseline

1. Describe T5’s baseline model and training.
   An encoder–decoder with 12 blocks each, each about BERT-base sized (\(d_{\text{model}} = 768\), \(d_{ff} = 3072\), 12 heads), 220M parameters. Pre-trained \(2^{19}\) steps on C4 (about 34B tokens, batches of 128 length-512 sequences) with AdaFactor and an inverse-square-root schedule; fine-tuned \(2^{18}\) steps per task; greedy decoding; a 32,000-wordpiece SentencePiece vocabulary covering English, German, French and Romanian.
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 3.1. Baseline, pp. 11–14
2. What is T5’s baseline denoising objective? Apply it to “The quick brown fox jumps over the lazy dog .” with brown fox and lazy corrupted.
   Drop 15% of tokens; replace each consecutive dropped span by a unique sentinel; the target lists the dropped spans after their sentinels, ending with a final sentinel. Input: “The quick <X> jumps over the <Y> dog .” Target: “<X> brown fox <Y> lazy <Z>”. Predicting only the corrupted spans keeps targets short and training cheap.
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 3.1. Baseline, pp. 11–14
3. How much did pre-training help T5’s baseline?
   A lot on almost everything (e.g. GLUE 83.3 vs 66.2, SuperGLUE 71.4 vs 53.0 without pre-training), except WMT English–French, whose data is large enough that pre-training gains are marginal.
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 3.1. Baseline, pp. 11–14