The Mathematics of Statistical Machine Translation: Parameter Estimation

4 Translation models

1. How is \(\Pr(\mathbf{f} \mid \mathbf{e})\) related to alignments?
   By summing over alignments: \(\Pr(\mathbf{f} \mid \mathbf{e}) = \sum_{\mathbf{a}} \Pr(\mathbf{f}, \mathbf{a} \mid \mathbf{e})\). Representing an alignment as \(a_1 \ldots a_m\), \(a_j = i\) if French word \(j\) connects to English word \(i\) (0 for none), the joint can be factored exactly (no approximation) as length × per-position alignment and word choices.
   • Brown et al., 1993, The Mathematics of Statistical Machine Translation (Computational Linguistics), §4
2. How are the parameters of the IBM models estimated?
   With the EM algorithm: guess parameters, compute expected counts (e.g. how often \(e\) connects to \(f\)) under the current model by weighting alignments by \(\Pr(\mathbf{a} \mid \mathbf{e}, \mathbf{f})\), re-normalize the counts into new parameters, and repeat, approaching a local maximum of the training-data likelihood.
   • Brown et al., 1993, The Mathematics of Statistical Machine Translation (Computational Linguistics), §4
3. Summarize how IBM Models 1–5 differ.

   model	adds
   1	word translation \(t(f \mid e)\); all alignments equally likely
   2	alignment probabilities \(a(i \mid j, m, l)\) by position
   3	fertility \(n(\phi \mid e)\) and distortion \(d(j \mid i, m, l)\)
   4	relative distortion of cepts, word classes
   5	removes deficiency (places words only in vacancies)

   Each serves as a stepping stone to train the next.
   • Brown et al., 1993, The Mathematics of Statistical Machine Translation (Computational Linguistics), §4