Neural Machine Translation of Rare Words with Subword Units

3.1 Related Work

1. How does Sennrich et al.’s segmentation differ from segmentation used in phrase-based SMT?
   SMT segmenters (compound splitters, Morfessor) are conservative; subword NMT wants aggressive segmentation that gives open-vocabulary translation with a compact network vocabulary and no back-off dictionary, while keeping text sequences reasonably short.
   • Sennrich et al., 2016, Neural Machine Translation of Rare Words with Subword Units (Proceedings of the 54th Annual Meeting of ACL (Volume 1: Long Papers)), §3.1, p. 1717
2. Why might variable-length subword sequences beat fixed-length character-based word vectors in NMT?
   With subwords the attention mechanism can focus on different units of a word at each step; a fixed-length vector per word (as in parallel work) is an information bottleneck, e.g. for a long compound like Abwasser|behandlungs|anlage.
   • Sennrich et al., 2016, Neural Machine Translation of Rare Words with Subword Units (Proceedings of the 54th Annual Meeting of ACL (Volume 1: Long Papers)), §3.1, p. 1717