Attention Is All You Need
Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017))
Subscriber deck — free during early access
1 cards in “3.4 Embeddings and Softmax” · whole deck
3.4 Embeddings and Softmax
- How are embeddings and the output softmax tied in the Transformer?The same weight matrix is shared between the input and output embedding layers and the pre-softmax linear transformation; in the embedding layers the weights are multiplied by \(\sqrt{d_{\text{model}}}\).
- Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §3.4
Source
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems 30 (NIPS 2017). https://papers.nips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html