Coreference Resolution and Entity Linking

Mentions are linguistic expressions like her or Victoria Chen; the referent is the discourse entity referred to. Two or more mentions that refer to the same discourse entity corefer.

A dialogue system resolving “I’ll take the second one” to a flight it listed; QA knowing who she is in “She was born in Warsaw”; MT from a pro-drop language like Spanish choosing “he says” vs “she says”.

A mental model built incrementally while interpreting a text, holding the entities referred to, their properties and relations. A first mention evokes a representation into the model; later mentions access it.

Anaphora: reference to an entity previously introduced; the referring expression is an anaphor; the prior coreferring mention is its antecedent. A singleton is an entity mentioned only once.

Determining whether mentions corefer, producing coreference chains (clusters). Subtasks, often done jointly: (1) identify the mentions, (2) cluster them into chains/discourse entities.

1	{Victoria Chen, her, the 38-year-old, she}
2	{Megabucks Banking, the company, Megabucks}
3	{her pay}
4	{Lotsabucks}

Mentions can nest: her is inside her pay, a different entity.

Mapping a discourse entity to a real-world individual in an ontology (e.g. Washington: the state, the city, or George Washington). Wikification uses Wikipedia pages as the entity ids.

Deciding whether two event mentions refer to the same event, e.g. buy and acquisition in two sentences about AMD and ATI. Event mentions are harder to detect since they can be verbal or nominal.

An anaphor referring back to a discourse segment: after “According to Soleil, Beau just opened a restaurant”, that can be a speech act (“that turned out to be a lie”), a proposition (“that was false”) or a manner of description.

Indefinite NPs, definite NPs, pronouns (including demonstratives and zero anaphors), and names.

Introduce entities new to the hearer: a beautiful goose, some walnuts, colloquial this beautiful cauliflower.

Prior mention in the text (the white stallion), the hearer’s world knowledge (the New York Times), or uniqueness implied by the description (the car keys). Over half of newswire definite NPs are non-anaphoric.

Cataphora: the pronoun comes before its referent (“Even before she saw it, Dorothy had been thinking about the Emerald City”). Bound: the pronoun acts like a variable bound by a quantifier (“Every dancer brought her left arm forward”).

An anaphor with no lexical realization, possible in languages like Chinese, Japanese and Italian: “Giovanni andò a far visita a degli amici. Per via φ comprò del vino.” Zero anaphors complicate mention detection.

Inferring that a new entity belongs to the discourse model through world knowledge about other entities: “I went to a superb restaurant. The chef had just opened it.”; “Mix flour, butter and water. Knead the dough.”

Very accessible referents can be referred to with less material (pronouns need high salience). First mentions tend to be long (full name, title, appositive: Victoria Chen, CFO of Megabucks Banking); later ones shorter (Ms. Chen, the 38-year-old, she), though not monotonically.

Under negation the NP doesn’t create a discourse referent (Karttunen 1969), so “*It is a Toyota” is bad.

Appositives (United, a unit of UAL,), predicative/attributive NPs (became the company’s president, jump to $2.3 million), expletive/pleonastic pronouns (it is raining, clefts, extraposition), and generics (“I love mangos. They are very tasty”, generic you).

Semantically plural entities can be it or they (“IBM announced… They have been working on it”), and singular they refers to individuals.

In “‘I voted for Nader because he was most aligned with my values,’ she said”, I, my and she corefer.

Reflexives corefer with the subject of the most immediate clause (“Janet bought herself…”, herself = Janet); non-reflexives cannot (“Janet bought her…”, her ≠ Janet).

Recent entities are more salient; subjects are more salient than objects, which are more salient than obliques: “Billy Bones went to the bar with Jim Hawkins. He called for rum.” He = Billy; swap the subject and he = Jim.

With telephoned, he = John; with criticized, he = Bill. Implicit causality: the implicit cause of criticizing is its object, of telephoning its subject, and the implicit cause is more salient.

Verbs restrict their arguments: food, not a bowl, is what one eats and cooks for hours, so it = the soup.

Detecting mentions, pronominal anaphora (her → Victoria Chen), filtering non-referential pronouns (pleonastic it), definite NPs (the 38-year-old, the company) and name variants (Megabucks = Megabucks Banking).

Singletons are the majority of mentions in running text (60–70% of entities) and are hard to tell from non-referential NPs.

The system is given human-labeled mention boundaries and only has to cluster them.

English and Chinese (about a million words each, many genres) plus Arabic newswire; the CoNLL-2012 shared task is based on it. No singletons; generics and pleonastic pronouns unmarked; appositives included inside the mention; prenominal modifiers are mentions only if proper nouns (UN in UN policy, not wheat in wheat fields).

ISNotes (information status, bridging), LitBank (literary novels, with singletons), AnCora-CO (Spanish and Catalan, with discourse deixis), ARRAU (all NPs including singletons; dialogue and fiction; bridging, discourse deixis, generics, ambiguity).

Very liberally, favoring recall and filtering later: all NPs, possessive pronouns and named entities from parsers and NER, or (span-based systems) all n-gram spans up to about 10 words.

Patterns with dictionaries of modal adjectives and cognitive verbs (Lappin and Leass):

It is Modaladjective that S
It is Modaladjective (for NP) to VP
It is Cogv-ed that S
It seems/appears/means/follows (that) S

(1) Take all NPs, possessive pronouns and named entities. (2) Remove numeric quantities, mentions embedded in larger mentions, adjectival nation forms and stop words. (3) Remove pleonastic it by regex patterns.

Referentiality classifiers, anaphoricity classifiers (is this NP an anaphor?) and discourse-new classifiers (is it new and a potential antecedent?), using features like head word, context, definiteness, animacy, length and position.

A high threshold loses recall; a low one admits pleonastic mentions and hurts precision. Modern systems do mention detection, anaphoricity and coreference jointly in one end-to-end model.

Turn the context into a pattern (make * in advance, make * in Hollywood) and see what words fill the wildcard: non-anaphoric (idiomatic) contexts mostly allow only it, anaphoric ones allow many NPs (make them in advance).

Entity-based (representing discourse entities) vs mention-based (considering mentions independently), and whether ranking models directly compare candidate antecedents.

A binary classifier given a candidate anaphor and a candidate antecedent decides coreferring or not, e.g. \(p(coref \mid \text{Victoria Chen}, \text{she})\) should be high and \(p(coref \mid \text{Megabucks Banking}, \text{she})\) low.

For each anaphor: one positive pair with its closest antecedent, and a negative pair for each mention in between. For the company: positive (the company, Megabucks Banking); negatives (the company, her), (the company, her pay), (the company, the 38-year-old).

After pairwise classification: closest-first runs right to left from mention \(i-1\) and links the first antecedent with probability > .5; best-first links the most probable antecedent. Then take the transitive closure.

It doesn’t compare candidate antecedents with each other, and it ignores the discourse model: each decision is local to a pair, without other mentions of the same entity.

For mention \(i\), a softmax over all prior mentions plus a dummy \(\epsilon\) (no antecedent: discourse-new or non-anaphoric), jointly learning anaphoricity and coreference.

The best antecedent is latent: the gold clustering gives a whole set of legal antecedents. Heuristics pick one (e.g. the closest), or the loss sums the probability of all correct antecedents.

They link a mention to a previous discourse entity (cluster) rather than a mention, using cluster features (size, shape like P-Pr-D for proper–pronoun–definite, aggregated pair probabilities) or learned cluster representations. Gains have been small, so mention ranking is more common.

They are lightweight when compute or data are scarce, and their features help error analysis even for neural systems.

Features of the anaphor, of the candidate antecedent, and of the pair; entity-based models add features of the antecedent’s cluster and of the relation between the anaphor and the cluster’s mentions.

Exact string match, entity headword agreement, mention distance; for pronouns exact attribute match and i-within-i; for nominals and names word inclusion and cosine.

Moving from individual features to conjunctions gained about 4 F1 points; conjunctions can be hand-designed, all pairs, or learned by decision trees or random forests.

Not shallow ones like string match or mention type (contextual embeddings cover them), but mention length, distance between mentions, and genre.

Consider all spans up to about 10 words; score each as a mention; prune; for each remaining span \(i\), choose an antecedent \(y_i \in \{1, \ldots, i-1, \epsilon\}\) with \(P(y_i) = \frac{\exp s(i, y_i)}{\sum_{y'} \exp s(i, y')}\), where \(s(i, j) = m(i) + m(j) + c(i, j)\).

So if any real antecedent scores positive the model picks the highest-scoring one, and if all are negative it abstains (no antecedent).

\(g_i = [h_{START(i)}, h_{END(i)}, h_{ATT(i)}]\): the encoder outputs (e.g. BERT) of the first and last tokens plus an attention-weighted sum over the span’s tokens that learns to find the head word.

By feedforward networks: \(m(i) = w_m \cdot FFN_m(g_i)\), \(c(i, j) = w_c \cdot FFN_c([g_i, g_j, g_i \circ g_j])\), where \(\circ\) is elementwise multiplication (a similarity feature). \(m\) also prunes spans at inference.

Since the antecedent is latent, maximize the total probability of all gold antecedents before \(i\): \(L = \sum_i -\log \sum_{\hat{y} \in Y(i) \cap GOLD(i)} P(\hat{y})\) (with \(GOLD(i) = \epsilon\) if \(i\) is in no gold cluster).

Mention detection and mention disambiguation.

A catalog of entities (pages) indexed for search, in-link counts in(e), and an anchor dictionary: for each page, its title and the anchor texts of links pointing to it, with each anchor’s frequency, link count and \(linkprob(a) = link(a)/freq(a)\).

Look up every token sequence up to 6 words in the anchor dictionary, then prune, e.g. substrings with low linkprob (Ada Lovelace kept, Lovelace pruned, born not an anchor at all).

\(p(e \mid a) = \frac{count(a \to e)}{link(a)}\). In “What Chinese Dynasty came before the Yuan?” it picks the most common sense, the currency, missing Yuan dynasty.

By shared in-links: \(rel(A, B) = \frac{\log\max(|in(A)|, |in(B)|) - \log|in(A) \cap in(B)|}{\log|W| - \log\min(|in(A)|, |in(B)|)}\).

Each other anchor \(b\) votes for \(a \to X\) with \(\frac{1}{|E(b)|}\sum_{Y \in E(b)} rel(X, Y)p(Y \mid b)\); sum the votes, take the entities within \(\epsilon\) of the best relatedness, and choose the one with the highest prior. Finally prune pairs whose average of coherence and linkprob is below a threshold.

Encode the candidate mention span and the entity separately and score them by dot product, so entity embeddings for the whole knowledge base can be precomputed and cached.

Encode the question with BERT, then \(p([i, j]) = \sigma\big(s_{start}(i) + s_{end}(j) + \sum_{t=i}^{j} s_{mention}(t)\big)\), each score a learned vector dotted with token encodings.

Entity embedding: BERT[CLS] over the page title and first 128 tokens, [CLS] t(e) [ENT] d(e) [SEP]. Span embedding: the average of its token encodings. Score by dot product, softmax over entities.

Fully supervised on questions with linked entity spans (e.g. WebQSP-EL, GraphQ-EL), jointly optimizing a binary cross-entropy mention detection loss and \(-\log p(e_g \mid [i, j])\) for the gold entity.

Link-based MUC and BLANC, mention-based B³, entity-based CEAF, and link-based entity-aware LEA. CoNLL F1 averages MUC, B³ and CEAF-e; use the reference implementations.

From coreference links common to hypothesis and reference: precision = common links / hypothesis links, recall = common links / reference links. It favors systems producing large chains and ignores singletons.

Per mention: precision \(|H \cap R| / |H|\), recall \(|H \cap R| / |R|\), then a weighted average (here uniform).

mention	P	R
a, b	2/2	2/3
c	1/3	1/3
d, e	2/3	2/2
average	11/15 ≈ .73	11/15 ≈ .73

A chain with all the right pronouns but no name, or the wrong name, is useless for IE; metrics can weight mentions by informativeness or require at least one name variant (NEC F1).

“The city council denied the demonstrators a permit because they feared/advocated violence.” They is the council with feared and the demonstrators with advocated, requiring causal reasoning and world knowledge.

Two parties; a pronoun that prefers one but could grammatically refer to either; a question asking which; changing one word flips the answer (“The trophy didn’t fit into the suitcase because it was too large/small”).

A large share of the problems became solvable by fine-tuned pretrained LMs, which encode much world knowledge, so harder datasets were proposed (e.g. KnowRef).

Sentences with a stereotypically male and a stereotypically female occupation and a pronoun that grammar and gender can’t disambiguate: “The secretary called the physician and told him/her about a new patient.” Biased systems are more accurate on pro-stereotypical cases.

Gender-swap all entities in OntoNotes to create a second dataset, train on original plus swapped, and use debiased embeddings (later, retrain ELMo with augmentation); WinoBias bias disappeared without hurting OntoNotes accuracy.

A gender-balanced corpus of Wikipedia sentences with an ambiguous gendered pronoun and two same-gender names, for Gendered Pronoun Resolution; systems do worse on feminine pronouns.

Karttunen (1969) formulated the discourse model; Bonnie Webber’s 1978 dissertation explored how entities are represented in it and how they license later reference.

Through NPs: (i) the current sentence right to left from the pronoun, (ii) the previous sentence left to right, (iii) two sentences back left to right, (iv) the current sentence left to right after the pronoun (cataphora). Pick the first NP agreeing in number, gender and person.

They focused on identity coreference only (ignoring bridging, metonymy, part–whole) and pushed the field toward supervised learning and the MUC metric.

Coreference supplies more surface forms for linking; linking adds encyclopedic knowledge to coreference, e.g. that Donald Tsang is a president helps resolve the President.

Metonymy (Washington for the US) and refocused referents, e.g. “in France the president is elected for seven years, while in the United States he is elected for four”: different individuals.