learn.ifs.name

Information Extraction: Relations, Events, and Time

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 21

24 cards in “21.2 Relation extraction algorithms” · whole deck

21.2 Relation extraction algorithms

  1. What are the main classes of relation extraction algorithms?
    Hand-written patterns, supervised machine learning, semi-supervised learning (bootstrapping or distant supervision), and unsupervised (Open IE).
  2. What are Hearst patterns?
    Lexico-syntactic patterns for relation extraction (Hearst, 1992), e.g. “NP₀ such as NP₁ {, NP₂ …, (and|or) NPᵢ}” implies hyponym(NPᵢ, NP₀): from “red algae, such as Gelidium” infer Gelidium is a kind of red algae.
  3. Give Hearst patterns for hyponymy with examples.
    patternexample
    NP {, NP}* (and|or) other NP_Htemples, treasuries, and other civic buildings
    NP_H such as NP …red algae such as Gelidium
    such NP_H as NP …such authors as Herrick and Shakespeare
    NP_H, including NP …common-law countries, including Canada
    NP_H, especially NP …European countries, especially France
  4. How can pattern-based extraction use named entity constraints?
    Patterns over NE types, e.g. PER, POSITION of ORG (“George Marshall, Secretary of State of the United States”) or PER (named|appointed) PER Prep? POSITION, to find who holds which office in which organization.
  5. What are the pros and cons of hand-built patterns?
    High precision and tailorable to domains, but low recall and laborious to write for all cases.
  6. Describe the basic supervised relation extraction pipeline.
    Annotate a corpus with a fixed set of entities and relations; then (1) find pairs of named entities (usually within a sentence) and (2) classify the relation for each pair with any supervised classifier. An optional filter first decides whether a pair is related at all.
  7. What features does a feature-based relation classifier use?
    Headwords of the two mentions and their concatenation, bags of words and bigrams in and between the mentions, NE types and their concatenation (ORG-PER), entity level (name, nominal, pronoun), the number of entities between them, and syntactic paths between the mentions.
  8. How does a typical neural relation extractor for TACRED work?
    Feed the sentence to a pretrained encoder (e.g. BERT or SpanBERT) with the subject and object replaced by their NER tags (“[SUBJ-PERSON] was born in [OBJ-LOC]”), and put a linear classifier over the relation labels on the [CLS] output.
  9. Why are the subject and object replaced by NER tags in neural relation extraction?
    Partial de-lexification keeps the system from overfitting to the particular entity words.
  10. What are the drawbacks of supervised relation extraction?
    Labeling large training sets is very expensive, and models are brittle: they don’t generalize well to new genres.
  11. How does bootstrapping relation extraction work?
    Start from a few seed tuples (e.g. Ryanair has a hub at Charleroi) or seed patterns; find sentences containing the seed entities, generalize their contexts into new patterns (“[ORG], which uses [LOC] as a hub”), use the patterns to find new tuples, keep high-confidence ones, and iterate.
  12. What is semantic drift in bootstrapping?
    An erroneous pattern introduces wrong tuples, which create more bad patterns, so the meaning of the extracted relation drifts (e.g. “Sydney has a ferry hub at Circular Quay” yielding ⟨Sydney, Circular Quay⟩).
  13. How is the confidence of a bootstrapping pattern computed?
    By balancing its performance on the current tuple set (hits(p): known tuples it matches) against its productivity (finds(p): all tuples it finds), e.g. \(\text{Conf}(p) = \frac{|hits(p)|}{|finds(p)|}\log|finds(p)|\).
  14. How is confidence in a new tuple combined over the patterns supporting it?
    With noisy-or: assuming a tuple is false only if all its supporting patterns failed, independently, \(\text{Conf}(t) = 1 - \prod_{p}(1 - \text{Conf}(p))\).
  15. What is distant supervision for relation extraction?
    Use a large database (DBpedia, Freebase) of relation tuples as supervision: find sentences in a big corpus containing both entities of a known tuple, extract (noisy) features from all of them, and train a supervised classifier; no-relation examples come from random unrelated entity pairs.
  16. Why can distant supervision use very rich features?
    Its training sets are huge, so it can use conjunctions of features, e.g. M1 = ORG & M2 = PER & nextword = “said” & a specific syntactic path.
  17. What are the advantages and the main problem of distant supervision?
    Like supervised learning it uses many features and detailed human knowledge; like patterns it uses high-precision evidence; like unsupervised methods it needs no labeled text corpus and avoids genre issues, and without iterative expansion there is no semantic drift. But precision tends to be low, and it only works for relations already in a large database.
  18. What did Snow et al. (2005) find when using WordNet hypernym pairs as distant supervision?
    Their system rediscovered Hearst’s original patterns plus tens of thousands of new ones, like “NP_H like NP” and “NP_H called NP”.
  19. What is open information extraction (Open IE)?
    Unsupervised relation extraction from the web with no labeled data and no predefined relations: relations are simply strings of words, usually starting with a verb.
  20. How does the ReVerb Open IE system extract relations?
    POS-tag and chunk the sentence; for each verb, find the longest verb-initial word sequence meeting syntactic and lexical constraints; take the nearest noun phrases to its left and right as arguments; score the triple with a confidence classifier trained on hand-labeled extractions.
  21. What do ReVerb’s syntactic and lexical constraints enforce?
    Syntactic: a verb-initial sequence, possibly with nouns, adjectives, prepositions or particles (light verbs like have express the relation with a noun: has a hub in). Lexical: the relation must occur with many distinct arguments in a large corpus, pruning rare long strings.
  22. What does ReVerb extract from “United has a hub in Chicago, which is the headquarters of United Continental Holdings”?
    ⟨United, has a hub in, Chicago⟩ and ⟨Chicago, is the headquarters of, United Continental Holdings⟩.
  23. What are the advantage and disadvantages of Open IE?
    It handles huge numbers of relations without specifying them in advance; but the strings must be mapped to canonical forms for databases, and verb-centered methods miss relations expressed nominally.
  24. How are supervised and unsupervised relation extractors evaluated?
    Supervised: precision, recall, F1 against gold test relations (labeled or unlabeled). Semi- and unsupervised: no gold set, so estimate precision by having humans check a random sample of extracted tuples, possibly at different levels of recall (top 1,000, 10,000…); recall can’t be measured directly.

Source