RNNs and LSTMs

14.7 Encoder-decoder with RNNs

1. When is an encoder-decoder (sequence-to-sequence) model needed instead of sequence labeling?
   When input and output sequences can differ in length and the mapping between their tokens is indirect, as in machine translation; it is also used for summarization, question answering and dialogue.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.7, p. 319
2. What are the three conceptual components of an encoder-decoder network?
   An encoder producing contextualized representations \(h_{1:n}\) of the input; a context vector \(c\), a function of \(h_{1:n}\) conveying the essence of the input; and a decoder generating an arbitrary-length sequence of hidden states and outputs from \(c\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.7, pp. 319–320
3. How can an RNN language model be turned into a basic translation model?
   Concatenate the source sentence, a separator token <s>, and the target sentence, and model \(p(y \mid x) = \prod_t p(y_t \mid y_{1:t-1}, x)\): after reading the source, generate the target autoregressively.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.7, p. 320
4. In the basic RNN encoder-decoder, how is the context passed to the decoder?
   The encoder’s final hidden state \(c = h^e_n\) initializes the decoder (\(h^d_0 = c\)) and is also given to every decoder step, so its influence doesn’t wane: \(h^d_t = g(\hat{y}_{t-1}, h^d_{t-1}, c)\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.7, pp. 321–322
5. Why is teacher forcing used when training an encoder-decoder?
   At inference the decoder feeds its own outputs back and drifts from the gold target; in training, feeding the gold token as the next input instead speeds up learning.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14, §14.7, p. 322