BLEU: a Method for Automatic Evaluation of Machine Translation

3 The BLEU evaluation

1. Why can’t BLEU scores be compared across test sets with different numbers of references?
   More references give higher scores: on the same test corpus a human translator scored about 0.35 against four references but 0.26 against two. Even a human translation seldom scores 1.
   • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §3
2. How did Papineni et al. check whether BLEU differences between systems are reliable?
   They split the 500-sentence test corpus into 20 blocks of 25 sentences, computed BLEU per block, and used paired t-statistics between systems; the differences were statistically very significant.
   • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §3
3. Do you need multiple reference translations for BLEU?
   Not necessarily: using a single randomly chosen reference per story (for stylistic variety) kept the same system ranking, suggesting a big test corpus with one reference each can work if references aren’t all from the same translator.
   • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §3