Context-Free Grammars and Constituency Parsing

19.8 Evaluating parsers

1. What are the PARSEVAL metrics?
   Measures of how well a hypothesis parse’s constituents match a hand-labeled reference (gold) parse: a hypothesis constituent is correct if a reference constituent has the same start, end and label. Labeled recall = correct / reference constituents; labeled precision = correct / hypothesis constituents; and F1.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 19, §19.8, p. 443
2. What is the crossing brackets metric?
   The number of constituents bracketed one way in the reference, e.g. ((A B) C), but crossing it in the hypothesis, e.g. (A (B C)).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 19, §19.8, p. 444
3. What is evalb?
   The canonical implementation of PARSEVAL, which also canonicalizes grammar-specific details (auxiliaries, pre-infinitival to…) for comparing parsers with different grammars.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 19, §19.8, p. 444