Sequence to Sequence Learning with Neural Networks

1 Introduction

1. What limitation of deep neural networks did sequence-to-sequence learning address?
   DNNs need inputs and targets encoded as vectors of fixed dimensionality, but many problems (speech recognition, machine translation, question answering) map sequences to sequences whose lengths aren’t known in advance.
   • Sutskever et al., 2014, Sequence to Sequence Learning with Neural Networks (Advances in Neural Information Processing Systems 27 (NIPS 2014)), §1
2. What is the core idea of Sutskever, Vinyals and Le’s sequence-to-sequence model?
   One LSTM reads the input sequence, one step at a time, into a large fixed-dimensional vector; another LSTM, essentially a language model conditioned on that vector, generates the output sequence from it.
   • Sutskever et al., 2014, Sequence to Sequence Learning with Neural Networks (Advances in Neural Information Processing Systems 27 (NIPS 2014)), §1
3. What were the headline results of Sutskever et al. (2014)?
   On WMT’14 English–French, an ensemble of 5 deep LSTMs translated directly with 34.81 BLEU, beating a phrase-based SMT baseline (33.30) despite an 80k-word output vocabulary that penalized out-of-vocabulary words. Rescoring the baseline’s 1000-best lists gave 36.5, close to the previous state of the art (37.0).
   • Sutskever et al., 2014, Sequence to Sequence Learning with Neural Networks (Advances in Neural Information Processing Systems 27 (NIPS 2014)), §1
4. How does the seq2seq model relate to earlier neural sequence-to-sequence work?
   Kalchbrenner and Blunsom first mapped whole sentences to vectors (with convolutional networks, which lose word order); Cho et al. used a similar RNN encoder–decoder mainly inside an SMT system; Bahdanau et al. added attention for long sentences. CTC also maps sequences to sequences but assumes monotonic alignment.
   • Sutskever et al., 2014, Sequence to Sequence Learning with Neural Networks (Advances in Neural Information Processing Systems 27 (NIPS 2014)), §1