BLEU: a Method for Automatic Evaluation of Machine Translation

2.1 Modified n-gram precision

1. Why does plain unigram precision fail as a translation metric?
   MT systems can overgenerate “reasonable” words: “the the the the the the the” against references “The cat is on the mat” and “There is a cat on the mat” has unigram precision 7/7.
   • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.1
2. Define modified (clipped) n-gram precision, and compute it for “the the the the the the the”.
   Clip each candidate n-gram’s count by the maximum number of times it occurs in any single reference, \(\text{Count}_{\text{clip}} = \min(\text{Count}, \text{Max\_Ref\_Count})\), sum the clipped counts and divide by the total number of candidate n-grams. Here the occurs at most twice in a reference, so the modified unigram precision is 2/7 (and bigram precision 0).
   • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.1
3. Why are BLEU’s n-gram matches position-independent, and what aspects of quality do different n-gram orders capture?
   Matches are counted anywhere in the sentence, allowing legitimate variation in order. Unigram matches with the references tend to reflect adequacy (using the right words); longer n-gram matches reflect fluency.
   • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.1
4. How is modified n-gram precision computed over a test corpus?
   Compute clipped n-gram matches sentence by sentence, sum them over all candidate sentences, and divide by the total number of candidate n-grams in the corpus: \(p_n = \frac{\sum_{C} \sum_{\text{n-gram} \in C} \text{Count}_{\text{clip}}(\text{n-gram})}{\sum_{C'} \sum_{\text{n-gram}' \in C'} \text{Count}(\text{n-gram}')}\).
   • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.1
5. Why does BLEU only need to agree with human judgment over a corpus, not per sentence?
   Individual sentences can be penalized unfairly, e.g. “East Asian economy” when all references say “economy of East Asia”. Because all systems are treated alike and multiple translators with different styles provide references, such effects cancel out in comparisons between systems.
   • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.1
6. How did modified n-gram precision behave on human vs machine translations?
   It strongly separated a good human translation from a poor MT system, more so for longer n-grams, and also ranked two human translations (native vs non-native English) and three MT systems in the same order as human judges.
   • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.1