learn.ifs.name

BLEU: a Method for Automatic Evaluation of Machine Translation

Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL)

Subscriber deck — free during early access

27 cards

1 Introduction

  1. Why did MT need an automatic evaluation metric, according to Papineni et al.?
    Human evaluation of adequacy, fidelity and fluency is expensive and can take weeks or months, while developers need to monitor daily changes to weed out bad ideas: “MT progress stems from evaluation”, and there was a logjam of ideas waiting behind the evaluation bottleneck.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §1
  2. What does BLEU stand for, and what is the central idea?
    Bilingual Evaluation Understudy, an automated understudy to skilled human judges. The closer a machine translation is to professional human translations, the better it is; closeness is measured numerically against one or more reference translations.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §1
  3. What two ingredients does BLEU-style evaluation require?
    A numerical “translation closeness” metric, and a corpus of good-quality human reference translations.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §1
  4. What metric was BLEU modelled after?
    Speech recognition’s word error rate, modified to allow multiple references and legitimate differences in word choice and word order: a weighted average of variable-length phrase (n-gram) matches against the references.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §1

2.1 Modified n-gram precision

  1. Why does plain unigram precision fail as a translation metric?
    MT systems can overgenerate “reasonable” words: “the the the the the the the” against references “The cat is on the mat” and “There is a cat on the mat” has unigram precision 7/7.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.1
  2. Define modified (clipped) n-gram precision, and compute it for “the the the the the the the”.
    Clip each candidate n-gram’s count by the maximum number of times it occurs in any single reference, \(\text{Count}_{\text{clip}} = \min(\text{Count}, \text{Max\_Ref\_Count})\), sum the clipped counts and divide by the total number of candidate n-grams. Here the occurs at most twice in a reference, so the modified unigram precision is 2/7 (and bigram precision 0).
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.1
  3. Why are BLEU’s n-gram matches position-independent, and what aspects of quality do different n-gram orders capture?
    Matches are counted anywhere in the sentence, allowing legitimate variation in order. Unigram matches with the references tend to reflect adequacy (using the right words); longer n-gram matches reflect fluency.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.1
  4. How is modified n-gram precision computed over a test corpus?
    Compute clipped n-gram matches sentence by sentence, sum them over all candidate sentences, and divide by the total number of candidate n-grams in the corpus: \(p_n = \frac{\sum_{C} \sum_{\text{n-gram} \in C} \text{Count}_{\text{clip}}(\text{n-gram})}{\sum_{C'} \sum_{\text{n-gram}' \in C'} \text{Count}(\text{n-gram}')}\).
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.1
  5. Why does BLEU only need to agree with human judgment over a corpus, not per sentence?
    Individual sentences can be penalized unfairly, e.g. “East Asian economy” when all references say “economy of East Asia”. Because all systems are treated alike and multiple translators with different styles provide references, such effects cancel out in comparisons between systems.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.1
  6. How did modified n-gram precision behave on human vs machine translations?
    It strongly separated a good human translation from a poor MT system, more so for longer n-grams, and also ranked two human translations (native vs non-native English) and three MT systems in the same order as human judges.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.1

2.2 Combining the modified n-gram precisions

  1. Why does BLEU combine n-gram precisions with a geometric mean rather than an arithmetic one?
    Modified precision decays roughly exponentially with \(n\), so a linear average would be dominated by low orders; averaging logarithms (equivalently a geometric mean) accounts for the decay, and correlated slightly better with human judgments.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.2
  2. What is the downside of the geometric mean in BLEU?
    It is harsh: if any modified precision is zero, the whole score is zero, which the authors expected to be very rare on test corpora of reasonable size with \(N \le 4\) (but is common for single sentences).
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.2

2.2.1 The trouble with recall

  1. Why doesn’t BLEU use recall to penalize short translations?
    With multiple references, a good translation uses only one of the possible word choices; recalling all of them makes it worse: “I always invariably perpetually do.” recalls more reference words than “I always do.” but is a poorer translation.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.2.1
  2. Why is modified precision alone not enough to enforce proper length?
    A very short candidate like “of the” gets inflated precision (unigram 2/2, bigram 1/1) against long references; precision already penalizes overly long candidates, but nothing penalizes overly short ones.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.2.1

2.2.2 Sentence brevity penalty

  1. Define BLEU’s brevity penalty.
    With \(c\) the total candidate length and \(r\) the effective reference length (the sum over sentences of the best match length, the reference length closest to each candidate’s): \(\text{BP} = 1\) if \(c > r\), else \(e^{1 - r/c}\).
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.2.2
  2. Why is the brevity penalty computed over the whole corpus rather than per sentence?
    Averaging per-sentence penalties would punish length deviations on short sentences harshly; a corpus-level penalty allows some freedom at the sentence level.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.2.2
  3. Give the full BLEU formula.
    \(\text{BLEU} = \text{BP} \cdot \exp\big(\sum_{n=1}^{N} w_n \log p_n\big)\), or \(\log \text{BLEU} = \min(1 - \tfrac{r}{c}, 0) + \sum_{n} w_n \log p_n\). The baseline uses \(N = 4\) and uniform weights \(w_n = 1/4\).
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.2.2
  4. Compute BLEU for a candidate corpus of length \(c = 90\) with effective reference length \(r = 100\) and modified precisions \(p_1 = 0.6\), \(p_2 = 0.4\), \(p_3 = 0.3\), \(p_4 = 0.2\).
    \(\text{BP} = e^{1 - 100/90} \approx e^{-0.111} \approx 0.895\). Geometric mean \(= (0.6 \cdot 0.4 \cdot 0.3 \cdot 0.2)^{1/4} = 0.0144^{1/4} \approx 0.346\). BLEU \(\approx 0.895 \times 0.346 \approx 0.31\).
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.2.2
  5. What does a high BLEU score require of a candidate translation?
    Matching the references in length (brevity penalty), word choice (unigram precision) and word order (higher-order n-gram precision). Neither the penalty nor precision considers the source length, only the range of reference lengths.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.2.2

3 The BLEU evaluation

  1. Why can’t BLEU scores be compared across test sets with different numbers of references?
    More references give higher scores: on the same test corpus a human translator scored about 0.35 against four references but 0.26 against two. Even a human translation seldom scores 1.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §3
  2. How did Papineni et al. check whether BLEU differences between systems are reliable?
    They split the 500-sentence test corpus into 20 blocks of 25 sentences, computed BLEU per block, and used paired t-statistics between systems; the differences were statistically very significant.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §3
  3. Do you need multiple reference translations for BLEU?
    Not necessarily: using a single randomly chosen reference per story (for stylistic variety) kept the same system ranking, suggesting a big test corpus with one reference each can work if references aren’t all from the same translator.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §3

4 The human evaluation

  1. How were the human judgments used to validate BLEU collected?
    Two groups of 10 judges (monolingual native English speakers, and bilingual native Chinese speakers) rated 250 Chinese–English sentence pairs (5 systems) from 1 to 5; monolinguals judged readability and fluency only. Ratings were compared per judge and sentence across systems to remove judge and sentence effects.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §4
  2. How did the bilingual judges differ from the monolingual ones?
    Bilinguals judged the two human translations nearly indistinguishable, suggesting they focused more on adequacy than fluency, while monolinguals found the non-native translator clearly less fluent.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §4

5 BLEU vs the human evaluation

  1. How well did BLEU correlate with human judgments?
    Linear regression of human scores on BLEU (two references, five systems) gave correlation 0.99 with monolingual judges and 0.96 with bilingual judges; BLEU even captured the small difference between two close MT systems and the large gap between MT and human translations.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §5

6 Conclusion

  1. How does BLEU achieve its high correlation with human judgments, in its authors’ words?
    By averaging out individual sentence judgment errors over a test corpus rather than trying to divine the exact human judgment for every sentence: “quantity leads to quality”.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §6
  2. Beyond MT, where did the authors think BLEU could apply?
    Summarization and similar natural language generation tasks, since MT and summarization can both be viewed as generation from a textual context.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §6

Source

  • Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318. https://aclanthology.org/P02-1040/