Machine Translation

13.3 The encoder-decoder model

1. How do the encoder and decoder transformers differ in an MT model?
   The encoder uses standard (unmasked) transformer blocks over the whole source. The decoder is a conditional language model whose blocks add a cross-attention layer, with causal self-attention over the target generated so far.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 13, §13.3, pp. 284–285
2. What is cross-attention?
   Attention in the decoder (also called encoder-decoder or source attention) where the queries come from the previous decoder layer and the keys and values from the encoder’s final output \(H^{enc}\): \(Q = H^{dec}W^Q\), \(K = H^{enc}W^K\), \(V = H^{enc}W^V\), \(\text{softmax}(QK^\top/\sqrt{d_k})V\).

   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 13, §13.3, p. 285
3. How is an encoder-decoder MT model trained?
   Give the encoder the source; train the decoder autoregressively from the separator/start token to predict each next target token with cross-entropy loss, using teacher forcing (feeding the gold target token, not the model’s prediction, at each step).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 13, §13.3, p. 285