Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

3.2. Architectures

1. Which three Transformer architectures did T5 compare, in terms of attention masks?
   Encoder–decoder (fully visible mask in the encoder and encoder–decoder attention, causal mask in the decoder); decoder-only language model (causal mask over the concatenated input and target); and prefix LM (fully visible mask over the input prefix, causal over the target).
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 3.2. Architectures, pp. 15–20
2. Why does a causal language model represent inputs poorly in text-to-text tasks, and how does a prefix LM fix it?
   With causal masking, each input token’s representation depends only on earlier input tokens, an unnecessarily limited view of the context. A prefix LM uses a fully visible mask over the input, so its representations use the whole input; for classification it behaves much like BERT.
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 3.2. Architectures, pp. 15–20
3. How can an encoder–decoder be compared fairly with a decoder-only model?
   Not on both counts at once: an L + L-layer encoder–decoder has about as many parameters as a 2L-layer LM but costs about the same computation as an L-layer LM, since the encoder only processes the input and the decoder only the output.
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 3.2. Architectures, pp. 15–20
4. What did T5’s architecture comparison find?
   The encoder–decoder with a denoising objective was best on every task. Sharing parameters between encoder and decoder performed nearly as well with half the parameters; halving the layers hurt; the shared encoder–decoder beat the prefix LM; and denoising beat language modelling for every architecture.
   • Raffel et al., 2020, Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (Journal of Machine Learning Research), 3.2. Architectures, pp. 15–20