Neural Machine Translation of Rare Words with Subword Units

NMT models have a fixed vocabulary (typically 30,000–50,000 words), but translation is open-vocabulary, especially for languages with productive word formation like compounding and agglutination. Rare and unseen words were translated poorly.

They assume a 1-to-1 correspondence between source and target words, which fails when languages differ in morphological synthesis (e.g. compounds), and word-level models can’t generate unseen words. Copying unknown words works for names only when no transliteration or morphological change is needed.

Showing that open-vocabulary NMT is possible by encoding (rare) words as sequences of subword units, simpler and more effective than large vocabularies plus back-off dictionaries; and adapting byte pair encoding (BPE) to word segmentation.

Named entities (copied or transliterated, e.g. Barack Obama in Russian or Japanese script), cognates and loanwords (claustrophobia → German Klaustrophobie), and morphologically complex words such as compounds (solar system → Sonnen|system).

Most were potentially translatable from English through smaller units: 56 compounds, 21 names, 6 loanwords, 5 cases of transparent affixation, 1 number and 1 computer language identifier.

SMT segmenters (compound splitters, Morfessor) are conservative; subword NMT wants aggressive segmentation that gives open-vocabulary translation with a compact network vocabulary and no back-off dictionary, while keeping text sequences reasonably short.

With subwords the attention mechanism can focus on different units of a word at each step; a fixed-length vector per word (as in parallel work) is an information bottleneck, e.g. for a long compound like Abwasser|behandlungs|anlage.

Start with the character vocabulary, each word a character sequence plus an end-of-word symbol. Repeatedly count all adjacent symbol pairs (weighted by word frequency, not crossing word boundaries) and merge the most frequent pair (A, B) into a new symbol AB. The number of merges is the only hyperparameter.

Pairs are counted with word frequencies; ties go to the first pair found. The first five merges:

#	merge	count	effect
1	e + s	9	n e w es t·, w i d es t·
2	es + t	9	…est·
3	est + ·	9	…est·
4	l + o	7	lo w·, lo w e r·
5	lo + w	7	low·, low e r·

(· marks the end of a word.)

Split the word into characters (plus end-of-word marker), then apply the learned merges in order. Any word can be segmented into known symbols, so the vocabulary is open but fixed in size: with the merges above, the unseen lowest becomes low est·.

Initial characters plus the number of merge operations. Frequent words end up as single symbols (no shortlist of unsegmented words is needed), while rare words are split into frequent subword units.

BPE symbols remain interpretable subword units (character n-grams), so the network can learn to translate and generate new, unseen words from them.

Learning one set of merges on the union of source and target vocabularies, instead of separate encodings. It segments names and cognates consistently in both languages, making mappings easier to learn; for English–Russian the Russian side was transliterated to Latin script to learn the merges, then transliterated back.

On WMT 2015 English→German (4.2M sentence pairs) and English→Russian (2.6M), with BLEU, chrF3 (character n-gram F3), and unigram F1 measured separately for all words, rare words (outside the 50,000 most frequent) and OOVs (unseen in training), using an attention-based encoder–decoder.

Characters give a tiny vocabulary but 5.5× longer text; character n-grams and shortlists sit in between. Compound splitting, Morfessor and hyphenation reduce the vocabulary only moderately and leave unknown words. BPE (63k types) barely lengthens the text (112M vs 100M tokens) and leaves no unknown symbols.

All subword systems (no back-off dictionary) beat the word-level baseline with back-off dictionary, especially on rare words (unigram F1 36.8 → 41.8% EN→DE, 26.5 → 29.7% EN→RU). Ensembles improved by 0.3–1.3 BLEU and 0.6–2 chrF3; joint BPE gave +1.3 BLEU for EN→RU.

Copying unknown source words works well for German, since most OOVs are names in the same alphabet. Russian uses a different alphabet, so names must be transliterated, which subword models can learn and a copying dictionary can’t.

Rare and unseen words make up only 9–11% of test tokens, but they tend to carry central information in a sentence. BLEU also has a precision bias and chrF3 a recall bias, which explains some disagreement between them.

Words with frequency ranks between 50,000 and 500,000 occur only 2–60 times in training, too sparse to learn as single units; subword units are less sparse. A model representing them as subwords was more stable than one keeping them as whole words.

Subword systems learned compounding and transliteration, translating “health research institutes” correctly even with odd segmentations like Forsch|ungsinstitu|ten. Errors came from data sparseness (asinine) and inconsistent source/target segmentations, which joint BPE reduced.