Neural Machine Translation of Rare Words with Subword Units

4.2 Translation experiments

1. What did subword NMT achieve compared with the back-off dictionary baseline?
   All subword systems (no back-off dictionary) beat the word-level baseline with back-off dictionary, especially on rare words (unigram F1 36.8 → 41.8% EN→DE, 26.5 → 29.7% EN→RU). Ensembles improved by 0.3–1.3 BLEU and 0.6–2 chrF3; joint BPE gave +1.3 BLEU for EN→RU.
   • Sennrich et al., 2016, Neural Machine Translation of Rare Words with Subword Units (Proceedings of the 54th Annual Meeting of ACL (Volume 1: Long Papers)), §4.2, pp. 1720–1721
2. Why did subword models help much more on OOVs for English→Russian than for English→German?
   Copying unknown source words works well for German, since most OOVs are names in the same alphabet. Russian uses a different alphabet, so names must be transliterated, which subword models can learn and a copying dictionary can’t.
   • Sennrich et al., 2016, Neural Machine Translation of Rare Words with Subword Units (Proceedings of the 54th Annual Meeting of ACL (Volume 1: Long Papers)), §4.2, pp. 1720–1721
3. Why might BLEU and chrF3 underestimate the gains of subword NMT?
   Rare and unseen words make up only 9–11% of test tokens, but they tend to carry central information in a sentence. BLEU also has a precision bias and chrF3 a recall bias, which explains some disagreement between them.
   • Sennrich et al., 2016, Neural Machine Translation of Rare Words with Subword Units (Proceedings of the 54th Annual Meeting of ACL (Volume 1: Long Papers)), §4.2, pp. 1720–1721