Building a Large Annotated Corpus of English: The Penn Treebank

2.1 A simplified POS tagset for English

1. How did the Penn Treebank tagset differ in size from earlier tagsets, and why?
   It is much smaller: 36 POS tags plus 12 for punctuation and symbols, versus 87 simple tags in the Brown Corpus (and about 135–197 in LOB, UCREL, London-Lund). The project’s stochastic orientation made sparse data a concern.
   • Marcus et al., 1993, Building a Large Annotated Corpus of English (Computational Linguistics), §2.1
2. What is lexical recoverability, and how did it shrink the tagset?
   Distinctions recoverable from the word itself needn’t be tags. Brown tagged forms of have, be and do with their own tags (HV, BEM, BER, BEDZ, DOD…); the Penn Treebank tags them like other verbs (VB, VBD, VBZ…), and collapses pre-qualifiers, pre-quantifiers and both into PDT.
   • Marcus et al., 1993, Building a Large Annotated Corpus of English (Computational Linguistics), §2.1
3. What is syntactic recoverability? Give two examples.
   Distinctions recoverable from the parse tree aren’t tagged. Subject and object pronouns are both PRP; subordinating conjunctions and prepositions are both IN, since the former precede clauses and the latter noun phrases.
   • Marcus et al., 1993, Building a Large Annotated Corpus of English (Computational Linguistics), §2.1
4. Why does a smaller tagset improve consistency?
   Fewer distinctions mean fewer chances of inconsistent tagging. Brown always tagged there and now as RB but tagged here and then inconsistently as RB or RN even in identical contexts.
   • Marcus et al., 1993, Building a Large Annotated Corpus of English (Computational Linguistics), §2.1
5. How does the Penn Treebank tag by syntactic function where Brown didn’t?
   It encodes a word’s function in its tag when possible: one heading a noun phrase is NN (not CD); both is PDT in “both the boys”, RB in “the boys both”, NNS in “both of the boys”, CC in “both boys and girls”.
   • Marcus et al., 1993, Building a Large Annotated Corpus of English (Computational Linguistics), §2.1
6. In which case did the Penn Treebank split a Brown category instead of merging?
   The bare form of verbs: VB (infinitive or imperative) is distinguished from VBP (non-third-person singular present), since tensed and untensed clauses differ.
   • Marcus et al., 1993, Building a Large Annotated Corpus of English (Computational Linguistics), §2.1
7. How does the Penn Treebank handle words whose tag can’t be decided?
   To avoid arbitrary decisions, a word may get multiple tags, e.g. JJ|NN (adjective or noun as prenominal modifier), JJ|VBG, JJ|VBN, NN|VBG, RB|RP, signalling true indeterminacy or annotator uncertainty.
   • Marcus et al., 1993, Building a Large Annotated Corpus of English (Computational Linguistics), §2.1