Sequence to Sequence Learning with Neural Networks

2 The model

1. How does the seq2seq LSTM define \(p(y_1, \ldots, y_{T'} \mid x_1, \ldots, x_T)\)?
   The encoder’s last hidden state gives a fixed-dimensional representation \(v\) of the input; the decoder, an LSTM language model whose initial state is \(v\), factorizes \(p(y_1, \ldots, y_{T'} \mid x_1, \ldots, x_T) = \prod_{t=1}^{T'} p(y_t \mid v, y_1, \ldots, y_{t-1})\), each a softmax over the vocabulary.
   • Sutskever et al., 2014, Sequence to Sequence Learning with Neural Networks (Advances in Neural Information Processing Systems 27 (NIPS 2014)), §2
2. Why does every sentence end with an end-of-sentence symbol in seq2seq?
   It lets the model define a distribution over sequences of all possible lengths: generation stops once it outputs ⟨EOS⟩.

   • Sutskever et al., 2014, Sequence to Sequence Learning with Neural Networks (Advances in Neural Information Processing Systems 27 (NIPS 2014)), §2
3. In what three ways did the actual seq2seq model differ from the basic description?
   Two separate LSTMs for input and output (more parameters at negligible cost, and natural for multiple language pairs); deep LSTMs with four layers, which significantly beat shallow ones; and reversing the word order of the source sentence.
   • Sutskever et al., 2014, Sequence to Sequence Learning with Neural Networks (Advances in Neural Information Processing Systems 27 (NIPS 2014)), §2
4. Why does reversing the source sentence help seq2seq?
   Mapping \(c, b, a\) to \(\alpha, \beta, \gamma\) puts \(a\) right next to its translation \(\alpha\), \(b\) fairly close to \(\beta\), and so on. The average distance between corresponding words is unchanged, but the minimal time lag is greatly reduced, making it easier for SGD to “establish communication” between input and output.
   • Sutskever et al., 2014, Sequence to Sequence Learning with Neural Networks (Advances in Neural Information Processing Systems 27 (NIPS 2014)), §2