Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

3.3. Unsupervised Objectives

1. How did prefix LM, BERT-style and deshuffling objectives compare in T5?
   The BERT-style denoising objective was best (prefix LM matched it on translation), and deshuffling was considerably worse.
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 3.3. Unsupervised Objectives, pp. 20–25
2. Which variants of the BERT-style objective did T5 compare, and what was the lesson?
   Reconstructing the whole text with or without random token swaps (MASS-style), versus predicting only corrupted spans (replace spans with sentinels, or drop tokens). All performed similarly, so choose by computational cost: objectives with short targets train faster.
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 3.3. Unsupervised Objectives, pp. 20–25
3. How did corruption rate and span length affect T5’s denoising?
   Little: 10–25% corruption performed similarly, 50% hurt GLUE and SQuAD, so 15% was kept. Corrupting spans with mean length 3 slightly but significantly beat i.i.d. corruption on most non-translation tasks and gave shorter sequences.
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 3.3. Unsupervised Objectives, pp. 20–25