Neural Machine Translation by Jointly Learning to Align and Translate

The encoder must compress all the information of a source sentence into one fixed-length vector, which makes long sentences hard; Cho et al. had shown that basic encoder–decoder performance deteriorates rapidly as input length grows.

Encode the source into a sequence of vectors instead of one, and each time a target word is generated, (soft-)search for the source positions where the most relevant information is, building a context vector from them. The decoder no longer depends on a single fixed-length vector.

The encoder RNN reads \(x_1, \ldots, x_{T_x}\) into hidden states \(h_t = f(x_t, h_{t-1})\) and a vector \(c = q(\{h_1, \ldots, h_{T_x}\})\) (e.g. \(c = h_{T_x}\)); the decoder factorizes \(p(\mathbf{y}) = \prod_t p(y_t \mid \{y_1, \ldots, y_{t-1}\}, c)\), with \(p(y_t \mid \ldots) = g(y_{t-1}, s_t, c)\).

Each target word gets its own context vector: \(p(y_i \mid y_1, \ldots, y_{i-1}, \mathbf{x}) = g(y_{i-1}, s_i, c_i)\) with decoder state \(s_i = f(s_{i-1}, y_{i-1}, c_i)\), instead of a single \(c\) shared by all steps.

As a weighted sum of the encoder annotations: \(c_i = \sum_{j=1}^{T_x} \alpha_{ij} h_j\), with \(\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T_x} \exp(e_{ik})}\) and \(e_{ij} = a(s_{i-1}, h_j)\), an alignment model scoring how well input around position \(j\) matches output position \(i\).

A one-hidden-layer perceptron: \(a(s_{i-1}, h_j) = v_a^\top \tanh(W_a s_{i-1} + U_a h_j)\) (additive attention). It must be evaluated \(T_x \times T_y\) times per sentence pair, and \(U_a h_j\) doesn’t depend on \(i\), so it can be precomputed.

Alignment isn’t a latent variable to be summed or searched over: the model computes a soft alignment directly, so gradients flow through it and the alignment model is trained jointly with the whole translation model. \(c_i\) is the expected annotation under alignment probabilities \(\alpha_{ij}\).

The decoder decides which parts of the source sentence to pay attention to when generating each word. This relieves the encoder of encoding everything into a fixed-length vector: information can be spread across the annotations and selectively retrieved.

With a bidirectional RNN: a forward RNN reads \(x_1 \to x_{T_x}\) and a backward RNN reads \(x_{T_x} \to x_1\); \(h_j = [\overrightarrow{h}_j^\top; \overleftarrow{h}_j^\top]^\top\) summarizes both preceding and following words, focused on the words around \(x_j\), since RNNs better represent recent inputs.

WMT’14 English–French parallel corpora reduced to 348M words by data selection, no monolingual data; news-test-2014 (3003 sentences) as the test set; a 30,000-word shortlist per language with other words mapped to [UNK]; no lowercasing or stemming.

RNNencdec (Cho et al.’s encoder–decoder, 1000 gated hidden units each side) and RNNsearch (bidirectional encoder with 1000 units per direction, 1000-unit decoder, attention), each trained on sentences of up to 30 and up to 50 words; minibatch SGD with Adadelta, about 5 days each, decoding with beam search.

RNNsearch beat RNNencdec in all settings: on all test sentences 26.75 vs 17.82 (trained on ≤50 words), and RNNsearch-30 even beat RNNencdec-50. On sentences without unknown words, RNNsearch-50 trained longer scored 36.15, above Moses (35.63), which also used a 418M-word monolingual corpus.

model	all	no UNK
RNNencdec-30	13.93	24.19
RNNsearch-30	21.50	31.44
RNNencdec-50	17.82	26.71
RNNsearch-50	26.75	34.16
RNNsearch-50, trained longer	28.45	36.15
Moses	33.30	35.63

RNNencdec’s BLEU dropped dramatically as sentences got longer, while RNNsearch was much more robust; RNNsearch-50 showed no deterioration even for sentences of 50 or more words.

English–French alignments are largely monotonic (strong diagonal), with sensible non-monotonic exceptions: for “European Economic Area” → “zone économique européenne”, the model aligned zone with Area, jumping over two words, then looked back one word at a time.

A hard alignment maps the → l’, but choosing between le, la, les and l’ depends on the following word. Soft alignment lets the model look at both the and man. It also handles phrases of different lengths without mapping words to or from NULL.

It needn’t encode a whole long sentence perfectly into a fixed vector, only the parts around each word accurately. In examples, RNNencdec-50 drifted from the meaning after about 30 words (e.g. “based on his status as a health care worker” became “based on his state of health”), while RNNsearch-50 kept all details.

Graves’s Gaussian-mixture weights move monotonically in one direction, too restrictive for translation, which needs (long-distance) reordering. Bahdanau attention computes a weight for every source word for every target word: \(O(T_x T_y)\), fine for 15–40-word sentences but possibly limiting for other tasks.

Better handling of unknown and rare words, needed to match state-of-the-art systems in all contexts; their models used a 30,000-word shortlist with everything else mapped to [UNK].