Attention Is All You Need

2 Background

1. How do convolutional sequence models (ByteNet, ConvS2S) compare with the Transformer in relating distant positions?
   They compute representations in parallel, but the number of operations to relate two positions grows with their distance (linearly for ConvS2S, logarithmically for ByteNet). In the Transformer it is constant, at the cost of reduced resolution from averaging attention-weighted positions, which multi-head attention counteracts.
   • Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §2
2. What is self-attention?
   An attention mechanism relating different positions of a single sequence to compute a representation of that sequence (also called intra-attention). The Transformer was the first transduction model relying entirely on it, without sequence-aligned RNNs or convolutions.
   • Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §2