Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

2.4. Input and Output Format

1. How does T5 cast tasks into text-to-text format?
   A task-specific text prefix tells the model what to do, and the target is text:

   input	target
   translate English to German: That is good.	Das ist gut.
   mnli premise: … hypothesis: …	entailment
   summarize: article	summary
   stsb sentence1: … sentence2: …	3.8

   Training always uses maximum likelihood with teacher forcing; an output that isn’t a valid label counts as wrong.
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 2.4. Input and Output Format, pp. 8–9
2. How did T5 handle the regression task STS-B and Winograd-style tasks as text-to-text?
   STS-B scores (1–5) were rounded to the nearest 0.2 and output as strings like “2.6”, making it a 21-class problem. For WNLI, WSC and DPR, the ambiguous pronoun is marked in the passage (“… because *they* feared violence”) and the model generates the referent noun phrase.
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 2.4. Input and Output Format, pp. 8–9
3. How does T5’s text-to-text framing differ from decaNLP and GPT-2’s zero-shot prompting?
   decaNLP casts everything as question answering and requires one multi-task model; T5 allows fine-tuning per task and uses short task prefixes. GPT-2 does zero-shot generation with a decoder-only LM; T5 mainly uses an encoder–decoder and studies transfer learning, not zero-shot.
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 2.4. Input and Output Format, pp. 8–9