Attention Is All You Need
Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017))
Subscriber deck — free during early access
1 cards in “6.2 Model Variations” · whole deck
6.2 Model Variations
- What did the Transformer ablations (model variations) show?A single head was 0.9 BLEU worse than the best setting, but too many heads also hurt; smaller key size \(d_k\) hurt, suggesting compatibility is hard to compute; bigger models were better; dropout was very helpful against overfitting; learned positional embeddings matched sinusoids.
- Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §6.2
Source
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems 30 (NIPS 2017). https://papers.nips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html