Dependency Parsing

20.4 Evaluation

1. Why isn’t exact match a good metric for dependency parsers?
   It is too pessimistic: most sentences are marked wrong, so it isn’t fine-grained enough to guide development.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 20, §20.4, p. 469
2. What are LAS, UAS and LS?
   Labeled attachment score: % of words with the correct head and relation. Unlabeled attachment score: % with the correct head regardless of label. Label accuracy score: % with the correct label regardless of head.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 20, §20.4, p. 469
3. If a parse of a 6-relation sentence gets 4 relations fully right and one more with the correct head but wrong label, what are LAS and UAS?
   LAS = 4/6 = 2/3; UAS = 5/6.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 20, §20.4, p. 469
4. How can performance on a particular relation like nsubj be evaluated?
   With precision (share of system nsubj labels that are correct) and recall (share of gold nsubj relations found), and a confusion matrix of dependency types.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 20, §20.4, pp. 469–470