Attention Is All You Need
Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017))
Subscriber deck — free during early access
2 cards in “3.2.3 Applications of Attention in our Model” · whole deck
3.2.3 Applications of Attention in our Model
- In which three ways does the Transformer use multi-head attention?Encoder–decoder attention (queries from the previous decoder layer, keys and values from the encoder output, so every decoder position attends over the whole input); encoder self-attention (all from the previous encoder layer); and masked decoder self-attention (each position attends to positions up to and including itself).
- Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §3.2.3
- How is decoder self-attention masked?Inside scaled dot-product attention, inputs to the softmax for illegal connections (from a position to later ones) are set to \(-\infty\), so their weights become 0 and the model stays auto-regressive.
Rows are queries, columns keys: each position attends only to itself and earlier ones - Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §3.2.3
Source
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems 30 (NIPS 2017). https://papers.nips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html