Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

3.4. Pre-training Data set

1. What did T5 find about filtering and domain of pre-training data?
   Removing C4’s heuristic filtering hurt every task. Smaller in-domain data sets sometimes helped matching tasks (Wikipedia + books boosted MultiRC, news boosted ReCoRD), but single-domain sets are much smaller.
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 3.4. Pre-training Data set, pp. 25–28
2. What happens when pre-training data is repeated many times?
   Performance degraded as the data shrank and repeated (from 64 to 4,096 repeats over training), with training loss dropping, a sign of memorization; 64 repeats did little harm. So large pre-training data sets are preferable when available.
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 3.4. Pre-training Data set, pp. 25–28