BLEU: a Method for Automatic Evaluation of Machine Translation

2.2.1 The trouble with recall

1. Why doesn’t BLEU use recall to penalize short translations?
   With multiple references, a good translation uses only one of the possible word choices; recalling all of them makes it worse: “I always invariably perpetually do.” recalls more reference words than “I always do.” but is a poorer translation.
   • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.2.1
2. Why is modified precision alone not enough to enforce proper length?
   A very short candidate like “of the” gets inflated precision (unigram 2/2, bigram 1/1) against long references; precision already penalizes overly long candidates, but nothing penalizes overly short ones.
   • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.2.1