Attention Is All You Need
Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017))
Subscriber deck — free during early access
1 cards in “5.1 Training Data and Batching” · whole deck
5.1 Training Data and Batching
- What data and tokenization did the Transformer use?WMT 2014 English–German (about 4.5M sentence pairs, shared BPE vocabulary of about 37,000 tokens) and English–French (36M sentences, 32,000 word-pieces). Batches held about 25,000 source and 25,000 target tokens, grouped by length.
- Vaswani et al., 2017, Attention Is All You Need (Advances in Neural Information Processing Systems 30 (NIPS 2017)), §5.1
Source
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems 30 (NIPS 2017). https://papers.nips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html