BLEU: a Method for Automatic Evaluation of Machine Translation

2.2.2 Sentence brevity penalty

1. Define BLEU’s brevity penalty.
   With \(c\) the total candidate length and \(r\) the effective reference length (the sum over sentences of the best match length, the reference length closest to each candidate’s): \(\text{BP} = 1\) if \(c > r\), else \(e^{1 - r/c}\).
   • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.2.2
2. Why is the brevity penalty computed over the whole corpus rather than per sentence?
   Averaging per-sentence penalties would punish length deviations on short sentences harshly; a corpus-level penalty allows some freedom at the sentence level.
   • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.2.2
3. Give the full BLEU formula.
   \(\text{BLEU} = \text{BP} \cdot \exp\big(\sum_{n=1}^{N} w_n \log p_n\big)\), or \(\log \text{BLEU} = \min(1 - \tfrac{r}{c}, 0) + \sum_{n} w_n \log p_n\). The baseline uses \(N = 4\) and uniform weights \(w_n = 1/4\).
   • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.2.2
4. Compute BLEU for a candidate corpus of length \(c = 90\) with effective reference length \(r = 100\) and modified precisions \(p_1 = 0.6\), \(p_2 = 0.4\), \(p_3 = 0.3\), \(p_4 = 0.2\).
   \(\text{BP} = e^{1 - 100/90} \approx e^{-0.111} \approx 0.895\). Geometric mean \(= (0.6 \cdot 0.4 \cdot 0.3 \cdot 0.2)^{1/4} = 0.0144^{1/4} \approx 0.346\). BLEU \(\approx 0.895 \times 0.346 \approx 0.31\).
   • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.2.2
5. What does a high BLEU score require of a candidate translation?
   Matching the references in length (brevity penalty), word choice (unigram precision) and word order (higher-order n-gram precision). Neither the penalty nor precision considers the source length, only the range of reference lengths.
   • Papineni et al., 2002, BLEU (Proceedings of the 40th Annual Meeting of ACL), §2.2.2