Unsupervised Word Sense Disambiguation Rivaling Supervised Methods

An unsupervised word sense disambiguation algorithm, trained on unannotated text, that rivals supervised methods needing hand-tagged data (accuracy above 96%), by bootstrapping from a few seeds with two constraints: one sense per collocation and one sense per discourse.

One sense per collocation: nearby words give strong, consistent clues to a word’s sense, depending on distance, order and syntactic relation. One sense per discourse: a word’s sense is highly consistent within a given document.

Disambiguation assigns each instance of a word to established senses (e.g. a dictionary’s); induction partitions instances into clusters by distributional similarity, which may not match standard senses.

No: it affects classification probabilistically and can be overridden when local evidence is strong.

By accuracy (when a word occurs more than once in a discourse, how often it takes the discourse’s majority sense: very high) and applicability (how often it occurs more than once in a discourse at all: about half the time).

Mostly statistics: two distinct terms of moderate frequency rarely co-occur in a discourse anyway, especially “bursty” content words. But writers also seem to avoid mixing senses: homograph pairs co-occurred about 5 times less often than comparable arbitrary word pairs.

It is strongest for immediately adjacent words and weakens with distance; stronger for predicate–argument relations than arbitrary words at the same distance; and much stronger for content words than function words (over 97% for adjacent content words).

Collocational features (lemmas, word forms, POS, word classes at various positions: adjacent, in a ±k window, trigrams, predicate–argument) are ranked by \(\left|\log \frac{P(\text{Sense}_A \mid \text{Collocation}_i)}{P(\text{Sense}_B \mid \text{Collocation}_i)}\right|\); a new example is classified by the single highest-ranked feature present in its context.

It avoids modelling the complex dependencies between highly non-independent evidence sources, which suits the many overlapping collocations available in WSD.

Most ratios involve a zero count for some sense; the smoothing depends on the collocation type, distance, content vs function word and expected noise.

(1) Collect all contexts of the target word from a large untagged corpus. (2) Tag a few examples per sense via seed collocations (e.g. life vs manufacturing for plant), leaving a large untagged residual. (3) Train a decision list on the seed sets, apply it to the residual, and add examples classified above a threshold (optionally using one sense per discourse to filter and extend). (4) Repeat until the residual is stable. (5) Use the final classifier on new data.

Collocations ranked by log-likelihood ratio, each pointing to a sense, e.g. initially manufacturing plant, life (within ±10 words) and animal → living, equipment and employee → factory. In the final list the seeds are displaced by broader collocations such as car, union, flower, job, fruit.

Extending a sense label to other occurrences of the word in the same discourse can form a bridge to collocations (like cell for living plants) that never co-occur with the original seeds nearby; it can also correct mislabelled examples.

Seeds are only a starting point: if a seed collocate turns out more compatible with the other class, the final list reflects that, so a few misleading seeds aren’t fatal as long as most seeds form a coherent collocation space.

Words from dictionary definitions of each sense (weighted by how reliable their collocations are); a single defining collocate per sense (e.g. bird and machine for crane, from WordNet); or a human quickly labelling salient corpus collocates (flock, bulldozer), typically in a couple of minutes.

Two words as seeds already did surprisingly well; dictionary definitions did better by covering more of each sense’s concept space (though example sentences add noise); quick hand labelling of salient collocates did best, at minimal cost.

Examples stay in a seed set only while their classification probability exceeds the threshold; if new evidence discredits the crucial collocate they return to the residual and may be relabelled. Language’s redundancy makes the process largely self-correcting.

Incrementally widening the context window after intermediate convergence (adding new features to shake up the system), and randomly perturbing the inclusion threshold, similar to simulated annealing.

For error correction: low-confidence tags of a word that occurs several times in a discourse are overridden by the dominant tag, when the cumulative evidence (summed log-likelihood ratios) for the majority sense exceeds the minority’s by a threshold depending on the number of occurrences.

Where a discourse strongly disagrees about the dominant sense, all its instances are returned to the residual instead of keeping their tags, preventing mistagged collocates from gaining a foothold and keeping training data purer.

Coverage: about half the examples occur in discourses with no other instance of the word to corroborate the sense.

It nearly matched a supervised decision list trained on identical hand-tagged contexts (95.5% vs 96.1%), exceeded it with one sense per discourse (96.5%), and beat Schütze’s unsupervised algorithm on the same 4 words (96.7% vs 92.2%).

Applied after convergence, it cut the error rate by about 27% on average; applied at each iteration, it gave the best accuracy.

Aligned bilingual corpora are of limited availability, and translation differences often fail to model monolingual sense differences.

Earlier methods measured overlap with dictionary definitions; Yarowsky combines the limited seed information in definitions with the nearly unlimited co-occurrence information in corpora.

Schütze clustered contexts without supervision and aligned clusters to senses afterwards by hand, using bag-of-words documents. Yarowsky ties classes to the desired senses from the start with seeds, and models collocational distance, word order and predicate–argument relations.

Hearst started from hand-labelled sentences and added confidently tagged examples (described in two sentences, not developed further). Yarowsky needs no hand-labelled data and jointly uses collocation and discourse constraints.