learn.ifs.name

BLEU: a Method for Automatic Evaluation of Machine Translation

Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL)

Subscriber deck — free during early access

2 cards in “4 The human evaluation” · whole deck

4 The human evaluation

  1. How were the human judgments used to validate BLEU collected?
    Two groups of 10 judges (monolingual native English speakers, and bilingual native Chinese speakers) rated 250 Chinese–English sentence pairs (5 systems) from 1 to 5; monolinguals judged readability and fluency only. Ratings were compared per judge and sentence across systems to remove judge and sentence effects.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §4
  2. How did the bilingual judges differ from the monolingual ones?
    Bilinguals judged the two human translations nearly indistinguishable, suggesting they focused more on adequacy than fluency, while monolinguals found the non-native translator clearly less fluent.
    • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §4

Source

  • Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318. https://aclanthology.org/P02-1040/