BLEU: a Method for Automatic Evaluation of Machine Translation

1 Introduction

1. Why did MT need an automatic evaluation metric, according to Papineni et al.?
   Human evaluation of adequacy, fidelity and fluency is expensive and can take weeks or months, while developers need to monitor daily changes to weed out bad ideas: “MT progress stems from evaluation”, and there was a logjam of ideas waiting behind the evaluation bottleneck.
   • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §1
2. What does BLEU stand for, and what is the central idea?
   Bilingual Evaluation Understudy, an automated understudy to skilled human judges. The closer a machine translation is to professional human translations, the better it is; closeness is measured numerically against one or more reference translations.
   • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §1
3. What two ingredients does BLEU-style evaluation require?
   A numerical “translation closeness” metric, and a corpus of good-quality human reference translations.
   • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §1
4. What metric was BLEU modelled after?
   Speech recognition’s word error rate, modified to allow multiple references and legitimate differences in word choice and word order: a weighted average of variable-length phrase (n-gram) matches against the references.
   • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §1