Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

3.5. Training Strategy

1. How did fine-tuning only some parameters compare with full fine-tuning in T5?
   Updating all parameters was best. Adapter layers worked if their size matched the task (small for low-resource tasks like SQuAD, large for big tasks); gradual unfreezing slightly hurt everywhere but sped fine-tuning.
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 3.5. Training Strategy, pp. 28–34
2. How did T5 set mixing proportions for multi-task training?
   Examples-proportional mixing with a cap \(K\): \(r_m = \min(e_m, K) / \sum_n \min(e_n, K)\), so huge data sets (including unlabeled text) don’t crowd out others; temperature scaling (rates raised to \(1/T\) and renormalized; \(T = 2\) worked best); or equal mixing, which was much worse.
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 3.5. Training Strategy, pp. 28–34
3. How did multi-task training compare with pre-training then fine-tuning in T5?
   Plain multi-task training underperformed unsupervised pre-training plus fine-tuning on most tasks. Multi-task pre-training followed by fine-tuning matched the baseline, and leave-one-out was only slightly worse; purely supervised multi-task pre-training was worse except for translation.
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 3.5. Training Strategy, pp. 28–34