Sequence to Sequence Learning with Neural Networks

3.4 Training details

1. Describe the seq2seq LSTM architecture and its size.
   4 layers of 1000 cells each in both encoder and decoder, 1000-dimensional word embeddings, input vocabulary 160,000, output vocabulary 80,000 with a naive softmax: 380M parameters, 64M of them recurrent. Each added layer reduced perplexity by nearly 10%.
   • Sutskever et al., 2014, Sequence to Sequence Learning with Neural Networks (Advances in Neural Information Processing Systems 27 (NIPS 2014)), §3.4
2. What training recipe did Sutskever et al. use?
   Uniform initialization in \([-0.08, 0.08]\); plain SGD with learning rate 0.7, halved every half epoch after 5 epochs, 7.5 epochs in total; batches of 128; gradient norm clipping (rescale if \(\|g\|_2 > 5\)); and batches of similar-length sentences for a 2× speed-up.
   • Sutskever et al., 2014, Sequence to Sequence Learning with Neural Networks (Advances in Neural Information Processing Systems 27 (NIPS 2014)), §3.4
3. Why did seq2seq training need gradient clipping?
   LSTMs largely avoid vanishing gradients but can still have exploding ones, so the gradient was rescaled to norm 5 whenever its norm exceeded 5.
   • Sutskever et al., 2014, Sequence to Sequence Learning with Neural Networks (Advances in Neural Information Processing Systems 27 (NIPS 2014)), §3.4