Neural Machine Translation by Jointly Learning to Align and Translate

3.1 Decoder: General Description

1. How does the attention-based decoder differ from the basic encoder–decoder?
   Each target word gets its own context vector: \(p(y_i \mid y_1, \ldots, y_{i-1}, \mathbf{x}) = g(y_{i-1}, s_i, c_i)\) with decoder state \(s_i = f(s_{i-1}, y_{i-1}, c_i)\), instead of a single \(c\) shared by all steps.
   • Bahdanau et al., 2015, Neural Machine Translation by Jointly Learning to Align and Translate (3rd International Conference on Learning Representations (ICLR 2015)), §3.1
2. How is the context vector \(c_i\) computed in Bahdanau attention?
   As a weighted sum of the encoder annotations: \(c_i = \sum_{j=1}^{T_x} \alpha_{ij} h_j\), with \(\alpha_{ij} = \frac{\exp(e_{ij})}{\sum_{k=1}^{T_x} \exp(e_{ik})}\) and \(e_{ij} = a(s_{i-1}, h_j)\), an alignment model scoring how well input around position \(j\) matches output position \(i\).

   • Bahdanau et al., 2015, Neural Machine Translation by Jointly Learning to Align and Translate (3rd International Conference on Learning Representations (ICLR 2015)), §3.1
3. What alignment model \(a(s_{i-1}, h_j)\) did Bahdanau et al. use, and why that form?
   A one-hidden-layer perceptron: \(a(s_{i-1}, h_j) = v_a^\top \tanh(W_a s_{i-1} + U_a h_j)\) (additive attention). It must be evaluated \(T_x \times T_y\) times per sentence pair, and \(U_a h_j\) doesn’t depend on \(i\), so it can be precomputed.
   • Bahdanau et al., 2015, Neural Machine Translation by Jointly Learning to Align and Translate (3rd International Conference on Learning Representations (ICLR 2015)), §3.1
4. How does soft alignment differ from alignment in traditional statistical MT?
   Alignment isn’t a latent variable to be summed or searched over: the model computes a soft alignment directly, so gradients flow through it and the alignment model is trained jointly with the whole translation model. \(c_i\) is the expected annotation under alignment probabilities \(\alpha_{ij}\).
   • Bahdanau et al., 2015, Neural Machine Translation by Jointly Learning to Align and Translate (3rd International Conference on Learning Representations (ICLR 2015)), §3.1
5. Why is this mechanism called attention?
   The decoder decides which parts of the source sentence to pay attention to when generating each word. This relieves the encoder of encoding everything into a fixed-length vector: information can be spread across the annotations and selectively retrieved.
   • Bahdanau et al., 2015, Neural Machine Translation by Jointly Learning to Align and Translate (3rd International Conference on Learning Representations (ICLR 2015)), §3.1