Sequence to Sequence Learning with Neural Networks

DNNs need inputs and targets encoded as vectors of fixed dimensionality, but many problems (speech recognition, machine translation, question answering) map sequences to sequences whose lengths aren’t known in advance.

One LSTM reads the input sequence, one step at a time, into a large fixed-dimensional vector; another LSTM, essentially a language model conditioned on that vector, generates the output sequence from it.

On WMT’14 English–French, an ensemble of 5 deep LSTMs translated directly with 34.81 BLEU, beating a phrase-based SMT baseline (33.30) despite an 80k-word output vocabulary that penalized out-of-vocabulary words. Rescoring the baseline’s 1000-best lists gave 36.5, close to the previous state of the art (37.0).

Kalchbrenner and Blunsom first mapped whole sentences to vectors (with convolutional networks, which lose word order); Cho et al. used a similar RNN encoder–decoder mainly inside an SMT system; Bahdanau et al. added attention for long sentences. CTC also maps sequences to sequences but assumes monotonic alignment.

The encoder’s last hidden state gives a fixed-dimensional representation \(v\) of the input; the decoder, an LSTM language model whose initial state is \(v\), factorizes \(p(y_1, \ldots, y_{T'} \mid x_1, \ldots, x_T) = \prod_{t=1}^{T'} p(y_t \mid v, y_1, \ldots, y_{t-1})\), each a softmax over the vocabulary.

It lets the model define a distribution over sequences of all possible lengths: generation stops once it outputs ⟨EOS⟩.

Two separate LSTMs for input and output (more parameters at negligible cost, and natural for multiple language pairs); deep LSTMs with four layers, which significantly beat shallow ones; and reversing the word order of the source sentence.

Mapping \(c, b, a\) to \(\alpha, \beta, \gamma\) puts \(a\) right next to its translation \(\alpha\), \(b\) fairly close to \(\beta\), and so on. The average distance between corresponding words is unchanged, but the minimal time lag is greatly reduced, making it easier for SGD to “establish communication” between input and output.

A clean 12M-sentence subset of WMT’14 English–French (348M French and 304M English words); the 160,000 most frequent source words and 80,000 most frequent target words, with all other words replaced by UNK.

Training maximizes \(\frac{1}{|\mathcal{S}|} \sum_{(T, S) \in \mathcal{S}} \log p(T \mid S)\) over the training set. Decoding searches for \(\arg\max_T p(T \mid S)\) with a left-to-right beam search keeping the \(B\) best prefixes; a hypothesis ending in ⟨EOS⟩ is complete.

Little: a beam of 1 (greedy) already worked well, and a beam of 2 gave most of the benefit. An ensemble of 5 LSTMs got 33.00 BLEU with beam 1, 34.50 with beam 2 and 34.81 with beam 12.

Test perplexity dropped from 5.8 to 4.7 and BLEU of decoded translations rose from 25.9 to 30.6. Unexpectedly, it also made long sentences translate much better, suggesting better memory utilization, not just more confident early predictions.

4 layers of 1000 cells each in both encoder and decoder, 1000-dimensional word embeddings, input vocabulary 160,000, output vocabulary 80,000 with a naive softmax: 380M parameters, 64M of them recurrent. Each added layer reduced perplexity by nearly 10%.

Uniform initialization in \([-0.08, 0.08]\); plain SGD with learning rate 0.7, halved every half epoch after 5 epochs, 7.5 epochs in total; batches of 128; gradient norm clipping (rescale if \(\|g\|_2 > 5\)); and batches of similar-length sentences for a 2× speed-up.

LSTMs largely avoid vanishing gradients but can still have exploding ones, so the gradient was rescaled to norm 5 whenever its norm exceeded 5.

On 8 GPUs: each of the 4 LSTM layers ran on its own GPU, passing activations onward as soon as they were computed, and the other 4 GPUs split the 80,000-word softmax. This raised speed from about 1,700 to 6,300 words per second; training took about ten days.

It was the first time a pure neural translation system beat a phrase-based SMT baseline on a large MT task by a sizeable margin, despite being relatively unoptimized and unable to handle out-of-vocabulary words.

Surprisingly well: there was no degradation for sentences under 35 words and only a minor drop for the longest, unlike earlier reports with similar models. Performance did drop on sentences with many rare words.

Phrases cluster by meaning, which is sensitive to word order (“John admires Mary” vs “Mary admires John” land apart) but fairly invariant to active vs passive voice (“I gave her a card in the garden” near “In the garden, I gave her a card” and “She was given a card by me”), something a bag-of-words model couldn’t capture.

Find a problem encoding with the greatest number of short-term dependencies, since they make learning much simpler. They suspected even a standard RNN could be trained on reversed translation data, though they didn’t verify it.