learn.ifs.name

Sequence Labeling for Parts of Speech and Named Entities

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 18

6 cards in “18.7 Further details” · whole deck

18.7 Further details

  1. What datasets exist for POS tagging and NER?
    Universal Dependencies (POS-tagged corpora in over a hundred languages), Penn Treebanks (English, Chinese, Arabic), OntoNotes (named entities in English, Chinese, Arabic), plus domain-specific NER corpora for biomedical and literary text.
  2. How do commercial NER systems often work?
    Pragmatic combinations of lists, rules and some supervised learning, e.g. IBM System T compiling declarative rules (regexes, dictionaries, constraints) into efficient extractors.
  3. Describe the multi-pass rule-plus-learning approach to NER.
    (1) Tag unambiguous mentions with high-precision rules; (2) find substring matches of names found so far; (3) use application-specific name lists; (4) apply a supervised sequence labeler using earlier tags as features.
  4. How did rule-based POS taggers like English Constraint Grammar work?
    Two stages: a morphological analyzer returns all possible parts of speech for each word, then thousands of hand-written constraints rule out those inconsistent with the context.
  5. Why is POS tagging harder for morphologically rich languages like Czech, Hungarian or Turkish?
    Productive morphology gives huge vocabularies, hence many unknown words; and words carry information like case and gender that taggers must label.
  6. What tags do taggers for morphologically rich languages use?
    Sequences of morphological tags rather than one primitive tag (Turkish izin: iz+Noun+A3sg+Pnon+Gen, iz+Noun+A3sg+P2sg+Nom, izin+Noun+A3sg+Pnon+Nom), giving much larger tagsets. A morphological analyzer proposes candidate analyses and the tagger disambiguates; this also handles unknown stems.

Source