Attention Is All You Need

1 Introduction

1. What is the Transformer, in one sentence?
   A sequence transduction model with an encoder–decoder structure that relies entirely on attention (self-attention and encoder–decoder attention plus position-wise feed-forward layers), dispensing with recurrence and convolutions.
   • Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §1
2. What limitation of recurrent models motivated the Transformer?
   RNNs compute hidden states sequentially, \(h_t\) from \(h_{t-1}\), which precludes parallelization within a training example, critical for long sequences when memory limits batching. Factorization tricks and conditional computation helped, but the sequential constraint remained.
   • Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §1
3. What were the Transformer’s headline translation results?
   On WMT 2014 English–German, 28.4 BLEU, over 2 BLEU better than the best previous results including ensembles; on English–French, a single-model state of the art of 41.0 BLEU after 3.5 days on eight P100 GPUs, a small fraction of earlier training costs.
   • Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §1