Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Treat every text problem as text-to-text (text in, text out), so one model, objective, training procedure and decoding process apply to every task; use this unified setup for a systematic empirical survey of transfer learning choices, then scale up the best ones (up to 11B parameters) for state-of-the-art results.

No: its authors stress that the goal is not to propose new methods but to give a comprehensive perspective on where transfer learning for NLP stands, through a survey and controlled comparison of existing techniques, plus new resources (C4, code, models).

Only in details: layer normalization without bias, applied outside the residual path (to each subcomponent’s input), and simplified relative position embeddings: a learned scalar per bucket of key–query offsets added to the attention logits, shared across layers, 32 buckets growing logarithmically up to offset 128.

Keep lines ending in terminal punctuation; drop pages with fewer than 5 sentences and lines under 3 words; drop pages with words from a bad-words list, “lorem ipsum”, or curly brackets (code); drop lines mentioning Javascript; deduplicate repeated three-sentence spans; keep pages detected as English with probability ≥ 0.99. The result is about 750 GB.

A task-specific text prefix tells the model what to do, and the target is text:

input	target
translate English to German: That is good.	Das ist gut.
mnli premise: … hypothesis: …	entailment
summarize: article	summary
stsb sentence1: … sentence2: …	3.8

Training always uses maximum likelihood with teacher forcing; an output that isn’t a valid label counts as wrong.

STS-B scores (1–5) were rounded to the nearest 0.2 and output as strings like “2.6”, making it a 21-class problem. For WNLI, WSC and DPR, the ambiguous pronoun is marked in the passage (“… because *they* feared violence”) and the model generates the referent noun phrase.

decaNLP casts everything as question answering and requires one multi-task model; T5 allows fine-tuning per task and uses short task prefixes. GPT-2 does zero-shot generation with a decoder-only LM; T5 mainly uses an encoder–decoder and studies transfer learning, not zero-shot.

“Coordinate ascent”: start from a reasonable baseline and change one factor at a time (architecture, objective, data, training strategy, scale), which may miss interactions but avoids an infeasible combinatorial search.

An encoder–decoder with 12 blocks each, each about BERT-base sized (\(d_{\text{model}} = 768\), \(d_{ff} = 3072\), 12 heads), 220M parameters. Pre-trained \(2^{19}\) steps on C4 (about 34B tokens, batches of 128 length-512 sequences) with AdaFactor and an inverse-square-root schedule; fine-tuned \(2^{18}\) steps per task; greedy decoding; a 32,000-wordpiece SentencePiece vocabulary covering English, German, French and Romanian.

Drop 15% of tokens; replace each consecutive dropped span by a unique sentinel; the target lists the dropped spans after their sentinels, ending with a final sentinel. Input: “The quick <X> jumps over the <Y> dog .” Target: “<X> brown fox <Y> lazy <Z>”. Predicting only the corrupted spans keeps targets short and training cheap.

A lot on almost everything (e.g. GLUE 83.3 vs 66.2, SuperGLUE 71.4 vs 53.0 without pre-training), except WMT English–French, whose data is large enough that pre-training gains are marginal.

Encoder–decoder (fully visible mask in the encoder and encoder–decoder attention, causal mask in the decoder); decoder-only language model (causal mask over the concatenated input and target); and prefix LM (fully visible mask over the input prefix, causal over the target).

With causal masking, each input token’s representation depends only on earlier input tokens, an unnecessarily limited view of the context. A prefix LM uses a fully visible mask over the input, so its representations use the whole input; for classification it behaves much like BERT.

Not on both counts at once: an L + L-layer encoder–decoder has about as many parameters as a 2L-layer LM but costs about the same computation as an L-layer LM, since the encoder only processes the input and the decoder only the output.

The encoder–decoder with a denoising objective was best on every task. Sharing parameters between encoder and decoder performed nearly as well with half the parameters; halving the layers hurt; the shared encoder–decoder beat the prefix LM; and denoising beat language modelling for every architecture.

The BERT-style denoising objective was best (prefix LM matched it on translation), and deshuffling was considerably worse.

Reconstructing the whole text with or without random token swaps (MASS-style), versus predicting only corrupted spans (replace spans with sentinels, or drop tokens). All performed similarly, so choose by computational cost: objectives with short targets train faster.

Little: 10–25% corruption performed similarly, 50% hurt GLUE and SQuAD, so 15% was kept. Corrupting spans with mean length 3 slightly but significantly beat i.i.d. corruption on most non-translation tasks and gave shorter sequences.

Removing C4’s heuristic filtering hurt every task. Smaller in-domain data sets sometimes helped matching tasks (Wikipedia + books boosted MultiRC, news boosted ReCoRD), but single-domain sets are much smaller.

Performance degraded as the data shrank and repeated (from 64 to 4,096 repeats over training), with training loss dropping, a sign of memorization; 64 repeats did little harm. So large pre-training data sets are preferable when available.

Updating all parameters was best. Adapter layers worked if their size matched the task (small for low-resource tasks like SQuAD, large for big tasks); gradual unfreezing slightly hurt everywhere but sped fine-tuning.

Examples-proportional mixing with a cap \(K\): \(r_m = \min(e_m, K) / \sum_n \min(e_n, K)\), so huge data sets (including unlabeled text) don’t crowd out others; temperature scaling (rates raised to \(1/T\) and renormalized; \(T = 2\) worked best); or equal mixing, which was much worse.

Plain multi-task training underperformed unsupervised pre-training plus fine-tuning on most tasks. Multi-task pre-training followed by fine-tuning matched the baseline, and leave-one-out was only slightly worse; purely supervised multi-task pre-training was worse except for translation.

Training longer, using bigger batches, bigger models and ensembles all helped. A larger model usually added more than just longer training, 2× size for 2× steps was about as good as 4× size, and ensembling was an orthogonal gain (best on some tasks). The trade-offs differ: bigger models cost more at fine-tuning and inference time.

Span corruption (mean span 3, 15%), about 1 trillion pre-training tokens of C4 (1M steps with batches of 2,048 sequences), multi-task pre-training followed by per-task fine-tuning, beam search for translation and summarization, and five sizes: Small 60M, Base 220M, Large 770M, 3B and 11B parameters.

State of the art on 18 of 24 tasks: GLUE 90.3, SuperGLUE 88.9 (human 89.8), SQuAD EM over a point above the previous best, CNN/Daily Mail ROUGE; but not on WMT translation, probably because pre-training was English-only and top MT systems used back-translation.

No: T5-Base clearly beat the same baseline trained on the same 1 trillion tokens on every task, so the non-scaling choices from the systematic study also contributed.

The inconvenience of large models (need cheaper models via distillation, parameter sharing, conditional computation), more efficient ways to extract knowledge than denoising on a trillion tokens, formalizing similarity between pre-training and downstream tasks, and language-agnostic models.