Building a Large Annotated Corpus of English: The Penn Treebank
Marcus et al., 1993, Building a Large Annotated Corpus of English (Computational Linguistics)
Subscriber deck — free during early access
2 cards in “3 Two modes of annotation: an experiment” · whole deck
3 Two modes of annotation: an experiment
- What did the experiment comparing manual tagging with correcting automatic output find?Correcting was better on all counts: about twice as fast as tagging from scratch, with roughly half the inter-annotator disagreement, and higher accuracy against a benchmark.
- Marcus et al., 1993, Building a Large Annotated Corpus of English (Computational Linguistics), §3
- How did the experiment measure consistency versus accuracy?Consistency: inter-annotator disagreement rate (pairwise disagreements over words). Accuracy: disagreement with a carefully hand-corrected benchmark version of the texts, since consistency says nothing about validity.
- Marcus et al., 1993, Building a Large Annotated Corpus of English (Computational Linguistics), §3
Source
- Mitchell P. Marcus, Beatrice Santorini and Mary Ann Marcinkiewicz. 1993. Building a Large Annotated Corpus of English: The Penn Treebank. In Computational Linguistics, pp. 313–330. https://aclanthology.org/J93-2004/