Attention Is All You Need
Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017))
Subscriber deck — free during early access
1 cards in “6.1 Machine Translation” · whole deck
6.1 Machine Translation
- How did the Transformer’s cost compare with earlier systems?The base model, trained in 12 hours, already beat all previous models and ensembles on English–German; the big model reached 41.0 BLEU on English–French at less than 1/4 of the previous state of the art’s training cost. Checkpoint averaging and beam search (beam 4, length penalty 0.6) were used.
- Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §6.1
Source
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems 30 (NIPS 2017). https://papers.nips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html