learn.ifs.name

Machine Translation

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 13

11 cards in “13.6 MT evaluation” · whole deck

13.6 MT evaluation

  1. On what two dimensions are translations evaluated?
    Adequacy (faithfulness, fidelity): how well the translation captures the meaning of the source; and fluency: how grammatical, clear, readable and natural it is in the target language.
  2. How do human raters evaluate MT?
    Rate fluency and adequacy on scales (e.g. 1–5 or 1–100); bilingual raters compare with the source, monolingual raters with a reference translation; or rank pairs of candidates; or post-edit outputs and measure the edits.
  3. Why do human MT evaluations need rater training and normalization?
    Untrained raters struggle to separate fluency from adequacy, and raters disagree (ambiguous sources, different knowledge, different use of scales); outliers are removed and scores normalized per rater.
  4. What is the intuition behind overlap-based MT metrics?
    A good machine translation tends to contain the characters and words that occur in a human translation of the same sentence (going back to Miller & Beebe-Center, 1956).
  5. Define chrF.
    Character F-score: average the character n-gram precisions (chrP, \(n = 1..k\), fraction of hypothesis n-grams in the reference) and recalls (chrR, fraction of reference n-grams in the hypothesis), then combine: \(\text{chrF}_\beta = (1 + \beta^2)\frac{\text{chrP}\cdot\text{chrR}}{\beta^2\,\text{chrP} + \text{chrR}}\), usually with \(\beta = 2\) (recall weighted twice as much). Spaces are ignored.
  6. Compute chrF (\(k = \beta = 2\)) for HYP “witness of the past,” against REF “witness for the past,”.
    Without spaces: REF has 18 unigrams and 17 bigrams; HYP 17 and 16. Matches: 17 unigrams, 13 bigrams. chrP = (17/17 + 13/16)/2 ≈ .906; chrR = (17/18 + 13/17)/2 ≈ .855; chrF = 5·.906·.855/(4·.906 + .855) ≈ .86.
  7. How does BLEU differ from chrF?
    BLEU is word-based and precision-only: the geometric mean of clipped word n-gram precisions (n = 1..4) over the corpus, times a brevity penalty for too-short outputs. Being word-based, it is sensitive to tokenization (use SacreBLEU) and weaker for morphologically complex languages.
  8. How is statistical significance assessed for chrF comparisons?
    With the paired bootstrap (or randomization) test: resample pseudo-test sets; the middle 95% of their scores gives a confidence interval, and the share of pseudo-test sets where A beats B compares two systems.
  9. What are the limitations of overlap metrics like chrF and BLEU?
    They are very local (a moved phrase barely changes the score), can’t judge cross-sentence properties like discourse coherence, and compare very different systems poorly; they are best for evaluating changes to a single system.
  10. How do learned metrics like COMET and BLEURT evaluate translations?
    They train a predictor (a BERT-like model plus a linear layer) on datasets of human quality ratings of (reference, candidate) pairs; their outputs correlate highly with human judgments.
  11. How does BERTScore compare a candidate to a reference?
    It embeds both with BERT, scores token pairs by cosine, and greedily matches each reference token to its most similar candidate token (recall) and each candidate token to its most similar reference token (precision), combining them into F1. Tokens can be weighted by idf.

Source