Sequence to Sequence Learning with Neural Networks

Sutskever et al., 2014, Sequence to Sequence Learning with Neural Networks (Advances in Neural Information Processing Systems 27 (NIPS 2014))

Subscriber deck — free during early access

1 cards in “3.5 Parallelization” · whole deck

3.5 Parallelization

How was seq2seq training parallelized?
On 8 GPUs: each of the 4 LSTM layers ran on its own GPU, passing activations onward as soon as they were computed, and the other 4 GPUs split the 80,000-word softmax. This raised speed from about 1,700 to 6,300 words per second; training took about ten days.
- Sutskever et al., 2014, Sequence to Sequence Learning with Neural Networks (Advances in Neural Information Processing Systems 27 (NIPS 2014)), §3.5

Source

Ilya Sutskever, Oriol Vinyals and Quoc V. Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems 27 (NIPS 2014). https://papers.nips.cc/paper/2014/hash/5a18e133cbf9f257297f410bb7eca942-Abstract.html