Attention Is All You Need
Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017))
Subscriber deck — free during early access
1 cards in “3.3 Position-wise Feed-Forward Networks” · whole deck
3.3 Position-wise Feed-Forward Networks
- What is the position-wise feed-forward network in a Transformer layer?Two linear maps with a ReLU, applied to each position separately and identically: \(\text{FFN}(x) = \max(0, x W_1 + b_1) W_2 + b_2\), with inner dimension \(d_{ff} = 2048\) and parameters differing between layers (equivalently, two convolutions with kernel size 1).
- Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §3.3
Source
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems 30 (NIPS 2017). https://papers.nips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html