Attention Is All You Need

3.5 Positional Encoding

1. Why does the Transformer need positional encodings, and what are they?
   Without recurrence or convolution the model has no notion of order, so encodings are added to the input embeddings: \(PE_{(pos, 2i)} = \sin(pos / 10000^{2i/d_{\text{model}}})\), \(PE_{(pos, 2i+1)} = \cos(pos / 10000^{2i/d_{\text{model}}})\), sinusoids with wavelengths from \(2\pi\) to \(10000 \cdot 2\pi\).
   • Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §3.5
2. Why sinusoidal rather than learned positional encodings?
   For any fixed offset \(k\), \(PE_{pos+k}\) is a linear function of \(PE_{pos}\), which may help attend by relative position, and sinusoids may extrapolate to sequences longer than those seen in training. Learned embeddings gave nearly identical results.
   • Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §3.5