Building a Large Annotated Corpus of English: The Penn Treebank

A corpus of over 4.5 million words of American English annotated for part of speech, with over half also annotated for skeletal syntactic structure, built 1989–1992 and distributed through the Linguistic Data Consortium.

Automatically building statistical models of grammar for written and spoken language, formal theories of the grammars of writing and speech, studying prosody, and evaluating and comparing parsing models.

Hand correction of the output of an errorful automatic process: a stochastic tagger, then a parser, produce a first pass that annotators correct.

It is much smaller: 36 POS tags plus 12 for punctuation and symbols, versus 87 simple tags in the Brown Corpus (and about 135–197 in LOB, UCREL, London-Lund). The project’s stochastic orientation made sparse data a concern.

Distinctions recoverable from the word itself needn’t be tags. Brown tagged forms of have, be and do with their own tags (HV, BEM, BER, BEDZ, DOD…); the Penn Treebank tags them like other verbs (VB, VBD, VBZ…), and collapses pre-qualifiers, pre-quantifiers and both into PDT.

Distinctions recoverable from the parse tree aren’t tagged. Subject and object pronouns are both PRP; subordinating conjunctions and prepositions are both IN, since the former precede clauses and the latter noun phrases.

Fewer distinctions mean fewer chances of inconsistent tagging. Brown always tagged there and now as RB but tagged here and then inconsistently as RB or RN even in identical contexts.

It encodes a word’s function in its tag when possible: one heading a noun phrase is NN (not CD); both is PDT in “both the boys”, RB in “the boys both”, NNS in “both of the boys”, CC in “both boys and girls”.

The bare form of verbs: VB (infinitive or imperative) is distinguished from VBP (non-third-person singular present), since tensed and untensed clauses differ.

To avoid arbitrary decisions, a word may get multiple tags, e.g. JJ|NN (adjective or noun as prenominal modifier), JJ|VBG, JJ|VBN, NN|VBG, RB|RP, signalling true indeterminacy or annotator uncertainty.

NN	singular or mass noun
NNS	plural noun
NNP	singular proper noun
NNPS	plural proper noun
PRP	personal pronoun
PRP$	possessive pronoun
WP	wh-pronoun

NNP, NNPS and PRP replaced earlier NP, NPS and PP to avoid confusion with the phrase labels NP and PP.

Split into morphemes, each tagged separately (no compound tags as in Brown’s PPSS+BEM): children’s → children/NNS ’s/POS; won’t → wo/MD n’t/RB.

Automatic tagging (initially Church’s stochastic PARTS tagger, mapped to the Penn tagset; later a cascade of stochastic and rule-based taggers), then manual correction by annotators in a GNU Emacs interface.

To make recurring errors of the automatic stage easy to identify, giving confusion matrices useful for designing better taggers.

PARTS didn’t make some Penn distinctions: infinitives vs non-third-person present verbs (VB vs VBP), and prepositions vs particles (run up a hill vs run up a bill).

Correcting was better on all counts: about twice as fast as tagging from scratch, with roughly half the inter-annotator disagreement, and higher accuracy against a benchmark.

Consistency: inter-annotator disagreement rate (pairwise disagreements over words). Accuracy: disagreement with a carefully hand-corrected benchmark version of the texts, since consistency says nothing about validity.

It always gives exactly one analysis; it never attaches a constituent whose role it can’t determine with certainty, leaving a string of tree fragments; and its chunks are usually accurate. Annotators mainly “glue” fragments together.

Prepositional phrases, relative clauses and adverbial modifiers (they need extra-syntactic information), arguments vs adjuncts like infinitives (no large verb lexicon), and conjunction scope; every comma breaks the input into chunks since decisions are local. It does attach of-PPs right after a noun.

E.g. * (“understood” subject of infinitives or imperatives), 0 (zero that), T (trace of a moved wh-constituent). Fidditch supplies them cheaply, and they help recover predicate–argument structure (e.g. verb transitivity) from a flat context-free notation.

By searching combinations of structures: an SBAR with to right before the VP is infinitival; one with a tensed verb or auxiliary is tensed; that-clauses are SBARs starting with that or 0.

Annotators were much faster correcting a simplified, skeletal structure; argument vs adjunct distinctions were hard and couldn’t be made consistently. So a skeletal representation without a forced argument/adjunct distinction was adopted.

POS tags, non-branching lexical nodes and some phrasal nodes (notably NBAR) are removed, flattening the structure; NP-internal structure up through the head is left unannotated. Illustration (not from the paper):

( (S (NP The annotators)
     (VP glued
         (NP the fragments)
         (PP into (NP one tree)))
     .))

Letting annotators mark only structure they are sure of: X brackets a constituent of uncertain category; pseudo-attachment marks a constituent’s alternative attachment site, for “permanent predictable ambiguities” (where both attachments mean the same thing) and for the underlying position of extraposed elements.

Coordinated adjective phrases inside NPs must be bracketed, and deleting brackets with the mouse is much faster than creating them, so it was more efficient to leave ADJPs in and remove the unneeded ones.

Training stochastic parsers and POS taggers, inside–outside re-estimation from partially bracketed text, using prosody to disambiguate parses, bootstrapping lexicons, and finding examples for linguistic theory with tgrep, a fast tree-pattern search tool.

Its essentially context-free representation leaves many argument/adjunct relations unmarked (e.g. complements after a sentential adverb get “trapped”), and some category schemes are inconsistent. Users wanted richer, less skeletal annotation and explicit predicate–argument structure (logical subject and object), planned for the next phase.