Machine Translation

Information access: translating web instructions, recipes, news articles, Wikipedia or government pages written in another language.

Much more information is available in English and other languages of wealthy countries (more search results, bigger Wikipedias). Good MT can bring that information to speakers of lower-resourced languages.

CAT: MT produces a draft that a human translator fixes in a post-editing phase. Localization: adapting content or a product to a particular language community, commonly using CAT.

Incremental translation of speech on the fly before the sentence is complete (as in simultaneous interpretation), and image-centric translation, e.g. OCR of menus or street signs from a phone camera.

The output is a complex function of the entire input: target words don’t correspond to source words in number or order (Japanese puts the verb last and drops the pronoun “he”).

Universals hold for every language (e.g. words for people, eating and drinking; nouns and verbs; ways to ask questions or give commands); statistical universals hold for most languages.

Differences between languages that make translation hard: idiosyncratic lexical differences handled one by one (the word for “dog”), and systematic differences that can be modeled generally (verb before or after the object).

The study of systematic cross-linguistic similarities and differences; WALS, the World Atlas of Language Structures.

The basic order of subject, verb and object: SVO (English, German, French, Mandarin), SOV (Hindi, Japanese), VSO (Irish, Arabic).

VO languages generally have prepositions; OV languages generally have postpositions.

English puts adjectives before nouns (green witch); Spanish and Modern Hebrew put them after (bruja verde).

English bass → Spanish lubina (fish) or bajo (instrument); wall → German Wand (inside) or Mauer (outside); brother → Mandarin gege (older) or didi (younger). This links MT to word sense disambiguation.

One language may require information another leaves unmarked: English marks plural on nouns but Mandarin doesn’t; French and Spanish mark gender on adjectives, so translating into them requires choosing a gender.

English leg, foot, paw vs French: an animal’s leg is patte, a journey’s leg étape, a chair’s leg pied, a human leg jambe; foot is pied for humans but patte for animals.

English	context	French
leg	human	jambe
leg	animal	patte
leg	journey	étape
leg	chair	pied
foot	human	pied
foot, paw	animal	patte

When no word or phrase short of an explanatory footnote expresses a word’s exact meaning in another language: English has no neat equivalent of Mandarin xiào / Japanese oyakōkō (“filial piety”).

Verb-framed languages mark the direction of motion on the verb and manner on satellites (Spanish La botella salió flotando “exited floating”). Satellite-framed languages mark direction on particles or phrases and manner on the verb (English The bottle floated out).

Verb-framed: Japanese, Tamil, Romance, Semitic and Mayan languages. Satellite-framed: Chinese and non-Romance Indo-European languages like English, Swedish, Russian, Hindi, Farsi.

Languages that can omit pronouns when the referent is given in the discourse, e.g. Spanish “[El jefe] dio con un libro. Ø Mostró su hallazgo…”.

How much a language uses explicit pronouns rather than zeros. Referentially sparse languages like Chinese or Japanese, which make the hearer infer more, are cold; more explicit languages are hot (after McLuhan’s hot vs cold media).

The model must identify each omitted argument (zero) and recover its referent in order to insert the right English pronoun.

Generate a target-language sentence corresponding to a source sentence, typically translating each sentence independently.

A large set of parallel sentences; it is trained to maximize \(P(y_1, \ldots, y_m \mid x_1, \ldots, x_n)\) of the target subword tokens given the source tokens.

The encoder maps the source tokens to a context \(h = \text{encoder}(x)\); the decoder generates the output token by token, \(y_{t+1} = \text{decoder}(h, y_1, \ldots, y_t)\).

It makes it easy to copy tokens like names from source to target, and handles languages with and without spaces (English vs Chinese, Thai) uniformly. It is learned on combined source and target data.

Instead of merging the most frequent pair, it adds the merge of two existing wordpieces that most increases the language-model probability of the training corpus, until a desired vocabulary size is reached (BPE instead fixes the number of merges).

Top-down: start from a huge vocabulary (all characters plus frequent character sequences), estimate token probabilities (with EM), and iteratively remove tokens that don’t appear in high-probability tokenizations until the desired size is reached.

BPE creates many small meaningless pieces and merges very common tokens like ed onto neighbors (cor rupted), while unigram tends to find meaningful units (corrupt ed).

Text appearing in two or more languages: Europarl (European Parliament proceedings), the UN Parallel Corpus, OpenSubtitles (movie and TV subtitles), ParaCrawl (web text).

A cost function scoring how likely a span of source sentences and a span of target sentences are translations of each other, and an alignment algorithm (usually dynamic programming, like minimum edit distance) that finds a good overall alignment.

With a multilingual embedding space where sentences from different languages share one space: cosine similarity, normalized against random sentences and biased toward aligning single sentences (Thompson & Koehn).

1–1 mappings, 2–1 (or 1–2) alignments where one sentence is split or merged, and null alignments where a sentence has no counterpart.

Remove noisy pairs with rules (too long, too short, different URLs, suspiciously identical) or by discarding pairs with low multilingual-embedding cosine.

The encoder uses standard (unmasked) transformer blocks over the whole source. The decoder is a conditional language model whose blocks add a cross-attention layer, with causal self-attention over the target generated so far.

Attention in the decoder (also called encoder-decoder or source attention) where the queries come from the previous decoder layer and the keys and values from the encoder’s final output \(H^{enc}\): \(Q = H^{dec}W^Q\), \(K = H^{enc}W^K\), \(V = H^{enc}W^V\), \(\text{softmax}(QK^\top/\sqrt{d_k})V\).

Give the encoder the source; train the decoder autoregressively from the separator/start token to predict each next target token with cross-entropy loss, using teacher forcing (feeding the gold target token, not the model’s prediction, at each step).

A locally most probable token may lead to a worse sequence: if “yes” (0.5) beats “ok” (0.4) at step 1, greedy picks yes, even if “ok ok” has the highest total probability.

Generation has long-distance dependencies between output decisions, so dynamic programming doesn’t apply, and exhaustive search over all \(V^T\) sentences is far too slow.

A heuristic search keeping the \(k\) best hypotheses (partial translations with their probabilities) at each step: extend each with every vocabulary token, score the \(k \times V\) candidates, and prune back to the best \(k\). \(k\) is the beam width.

When a hypothesis generates EOS it is complete and removed from the frontier, and the beam shrinks by one; search continues until the beam is empty, returning \(k\) complete hypotheses.

By the log probability of the sequence, built incrementally with the chain rule: \(\text{score}(y) = \sum_{i=1}^{t}\log P(y_i \mid y_1, \ldots, y_{i-1}, x)\).

Language models give longer strings lower probabilities, so completed hypotheses of different lengths would favor short outputs. Divide the log probability by the number of words: \(\frac{1}{t}\log P(y \mid x)\).

Between 5 and 10.

Instead of the most probable translation, choose the one likely to have the least error: from a set of candidate translations \(Y\) (e.g. 32–64 samples), pick the candidate most similar to all the others, \(\hat{y} = \arg\max_{y \in Y}\sum_{c \in Y}\text{util}(y, c)\), with a utility like chrF, BLEU or BERTScore.

Widely in speech recognition before MT, and in other generation tasks such as summarization, dialogue and image captioning.

Adding synthetic training data generated from the current natural data to address insufficient training data.

To improve source→target MT with a small bitext and plenty of target-language monolingual text: train a target→source system on the bitext, translate the monolingual target text into the source language, and add these synthetic pairs (MT source, natural target) to the training data.

How the synthetic source is generated (greedy, beam search, sampling) and the ratio of backtranslated to natural data (natural bitext can be upsampled). It works surprisingly well: roughly two-thirds of the gain of the same amount of natural bitext.

Special tokens: a source-language token \(l_s\) for the encoder and a target-language token \(l_t\) for the decoder: \(h = \text{encoder}(x, l_s)\), \(y_{i+1} = \text{decoder}(h, l_t, y_{1:i})\).

It can draw on a similar higher-resource language in the training data, e.g. understanding a Galician word through Spanish.

Native speakers are often not involved as curators, technologists or evaluators; audits found many multilingual corpora mostly of unacceptable quality (boilerplate, wrong translations); and resources focus on pairs involving English.

Having labelers minimally edit MT output and measuring the distance is simpler to train evaluators for, and captures true errors rather than legitimate linguistic variation.

Adequacy (faithfulness, fidelity): how well the translation captures the meaning of the source; and fluency: how grammatical, clear, readable and natural it is in the target language.

Rate fluency and adequacy on scales (e.g. 1–5 or 1–100); bilingual raters compare with the source, monolingual raters with a reference translation; or rank pairs of candidates; or post-edit outputs and measure the edits.

Untrained raters struggle to separate fluency from adequacy, and raters disagree (ambiguous sources, different knowledge, different use of scales); outliers are removed and scores normalized per rater.

A good machine translation tends to contain the characters and words that occur in a human translation of the same sentence (going back to Miller & Beebe-Center, 1956).

Character F-score: average the character n-gram precisions (chrP, \(n = 1..k\), fraction of hypothesis n-grams in the reference) and recalls (chrR, fraction of reference n-grams in the hypothesis), then combine: \(\text{chrF}_\beta = (1 + \beta^2)\frac{\text{chrP}\cdot\text{chrR}}{\beta^2\,\text{chrP} + \text{chrR}}\), usually with \(\beta = 2\) (recall weighted twice as much). Spaces are ignored.

Without spaces: REF has 18 unigrams and 17 bigrams; HYP 17 and 16. Matches: 17 unigrams, 13 bigrams. chrP = (17/17 + 13/16)/2 ≈ .906; chrR = (17/18 + 13/17)/2 ≈ .855; chrF = 5·.906·.855/(4·.906 + .855) ≈ .86.

BLEU is word-based and precision-only: the geometric mean of clipped word n-gram precisions (n = 1..4) over the corpus, times a brevity penalty for too-short outputs. Being word-based, it is sensitive to tokenization (use SacreBLEU) and weaker for morphologically complex languages.

With the paired bootstrap (or randomization) test: resample pseudo-test sets; the middle 95% of their scores gives a confidence interval, and the share of pseudo-test sets where A beats B compares two systems.

They are very local (a moved phrase barely changes the score), can’t judge cross-sentence properties like discourse coherence, and compare very different systems poorly; they are best for evaluating changes to a single system.

They train a predictor (a BERT-like model plus a linear layer) on datasets of human quality ratings of (reference, candidate) pairs; their outputs correlate highly with human judgments.

It embeds both with BERT, scores token pairs by cosine, and greedily matches each reference token to its most similar candidate token (recall) and each candidate token to its most similar reference token (precision), combining them into F1. Tokens can be weighted by idf.

When translating from gender-neutral languages (Hungarian ő, often-dropped Spanish pronouns) into English, they default to male or follow stereotypes: “she is a nurse”, “he is a CEO”, amplifying biases beyond actual labor statistics.

MT systems translate worse when sentences describe people in non-stereotypical gender roles (“The doctor asked the nurse to help her”).

They are used in urgent situations without human translators (medicine, law); to “do no harm” they should be able to abstain from giving incorrect translations.

A critical 1966 US report (Automatic Language Processing Advisory Committee) that led to dramatic funding cuts for MT and a call for basic research in computational linguistics; it also included an influential early evaluation study.

Direct translation (word by word with a large bilingual dictionary), transfer (parse the source, transform the parse with rules, generate), and interlingua (analyze into an abstract meaning representation, generate from it). Deeper analysis needs less transfer knowledge.

As an interlingual approach, with attention integrating direct and transfer approaches by letting the decoder access source words and their representations directly.

Large bilingual corpora like the Canadian Hansards and then the web. The IBM group, drawing on the noisy channel model of speech recognition, proposed generative IBM Models 1–5 (the Candide system), published in full detail.

Phrase-based translation: a generative statistical approach that induces translations of phrase pairs. MERT (Minimum Error Rate Training): directly optimizing a discriminative log-linear model for metrics like BLEU. Toolkits: GIZA, Moses.

Grammars assigning parallel syntactic trees to a sentence pair, translating via reordering operations on the trees (e.g. inversion transduction grammar, synchronous CFGs).

Kalchbrenner & Blunsom (2013, CNN encoder + RNN decoder); first applied to MT by Bahdanau et al. (2015); the transformer encoder-decoder by Vaswani et al. (2017).