RNNs and LSTMs

14.8 Attention

1. What is the bottleneck problem of the basic encoder-decoder?
   The final encoder hidden state must represent everything about the source sentence, since it is all the decoder sees; information from the beginning of long sentences may be poorly represented.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.8, p. 323
2. How does the attention mechanism solve the bottleneck?
   It lets the decoder use all encoder hidden states: at each decoding step \(i\), a new context vector \(c_i\) is computed as a weighted average of the encoder states, with weights depending on the decoder’s previous state; then \(h^d_i = g(\hat{y}_{i-1}, h^d_{i-1}, c_i)\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.8, p. 323
3. How is the attention context vector computed with dot-product attention?
   Score each encoder state by similarity to the previous decoder state, \(\text{score} = h^d_{i-1} \cdot h^e_j\); softmax the scores into weights \(\alpha_{ij}\); take \(c_i = \sum_j \alpha_{ij}h^e_j\).

   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.8, p. 324
4. What is a more powerful alternative to dot-product attention scoring?
   A bilinear score with learned weights, \(\text{score} = h^d_{i-1}W_sh^e_j\), which learns which aspects of similarity matter and allows encoder and decoder states of different dimensionality.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.8, p. 325