Sequence to Sequence Learning with Neural Networks

3.2 Decoding and Rescoring

1. How is seq2seq trained and decoded?
   Training maximizes \(\frac{1}{|\mathcal{S}|} \sum_{(T, S) \in \mathcal{S}} \log p(T \mid S)\) over the training set. Decoding searches for \(\arg\max_T p(T \mid S)\) with a left-to-right beam search keeping the \(B\) best prefixes; a hypothesis ending in ⟨EOS⟩ is complete.
   • Sutskever et al., 2014, Sequence to Sequence Learning with Neural Networks (Advances in Neural Information Processing Systems 27 (NIPS 2014)), §3.2
2. How much did beam size matter for seq2seq translation?
   Little: a beam of 1 (greedy) already worked well, and a beam of 2 gave most of the benefit. An ensemble of 5 LSTMs got 33.00 BLEU with beam 1, 34.50 with beam 2 and 34.81 with beam 12.
   • Sutskever et al., 2014, Sequence to Sequence Learning with Neural Networks (Advances in Neural Information Processing Systems 27 (NIPS 2014)), §3.2