Coreference Resolution and Entity Linking

24.8 Evaluation of coreference resolution

1. Name the common coreference evaluation metrics.
   Link-based MUC and BLANC, mention-based B³, entity-based CEAF, and link-based entity-aware LEA. CoNLL F1 averages MUC, B³ and CEAF-e; use the reference implementations.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 24, §24.8, p. 562
2. How is the MUC F-measure computed, and what are its biases?
   From coreference links common to hypothesis and reference: precision = common links / hypothesis links, recall = common links / reference links. It favors systems producing large chains and ignores singletons.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 24, §24.8, p. 562
3. How is B³ computed? Work it for reference {a, b, c}, {d, e} and hypothesis {a, b}, {c, d, e}.
   Per mention: precision \(|H \cap R| / |H|\), recall \(|H \cap R| / |R|\), then a weighted average (here uniform).

   mention	P	R
   a, b	2/2	2/3
   c	1/3	1/3
   d, e	2/3	2/2
   average	11/15 ≈ .73	11/15 ≈ .73

   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 24, §24.8, pp. 562–563
4. Why might named-entity tasks need alternative coreference metrics?
   A chain with all the right pronouns but no name, or the wrong name, is useless for IE; metrics can weight mentions by informativeness or require at least one name variant (NEC F1).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 24, §24.8, p. 563