Neural Machine Translation of Rare Words with Subword Units

1 Introduction

1. What problem does subword NMT address?
   NMT models have a fixed vocabulary (typically 30,000–50,000 words), but translation is open-vocabulary, especially for languages with productive word formation like compounding and agglutination. Rare and unseen words were translated poorly.
   • Sennrich et al., 2016, Neural Machine Translation of Rare Words with Subword Units (Proceedings of the 54th Annual Meeting of ACL (Volume 1: Long Papers)), §1, pp. 1715–1716
2. Why are back-off dictionaries a poor fix for rare words in NMT, according to Sennrich et al.?
   They assume a 1-to-1 correspondence between source and target words, which fails when languages differ in morphological synthesis (e.g. compounds), and word-level models can’t generate unseen words. Copying unknown words works for names only when no transliteration or morphological change is needed.
   • Sennrich et al., 2016, Neural Machine Translation of Rare Words with Subword Units (Proceedings of the 54th Annual Meeting of ACL (Volume 1: Long Papers)), §1, pp. 1715–1716
3. What are the two main contributions of Sennrich, Haddow and Birch (2016)?
   Showing that open-vocabulary NMT is possible by encoding (rare) words as sequences of subword units, simpler and more effective than large vocabularies plus back-off dictionaries; and adapting byte pair encoding (BPE) to word segmentation.
   • Sennrich et al., 2016, Neural Machine Translation of Rare Words with Subword Units (Proceedings of the 54th Annual Meeting of ACL (Volume 1: Long Papers)), §1, pp. 1715–1716