Word Association Norms, Mutual Information, and Lexicography

An objective measure, the association ratio, based on mutual information, for estimating word association norms directly from corpora, making norms possible for tens of thousands of words rather than a few hundred.

The degree to which one word primes another (people respond faster to nurse after doctor). Measuring it by testing a few thousand subjects on a few hundred words is costly and unreliable.

“You shall know a word by the company it keeps” (Firth 1957): words are classified not only by meaning but by their co-occurrence with other words, e.g. bank with money, loan, robbery vs with river, swim, boat.

Not the idea, which goes back to Firth and to Hornby’s verb patterns (1948), but computational storage and analysis of large corpora, making it possible to test such informal assertions rigorously.

Constraining language models for speech recognition and OCR, disambiguating parses (noun compounds, conjunctions, PP attachment), text retrieval, and helping computational linguists and lexicographers find lexico-syntactic facts and conventional usage.

Farm is far more likely in “federal _ credit” and form in “some _ of”. Part-of-speech constraints wouldn’t help, since both are common nouns.

The most influential empirical word-association study, though small and dated: a few thousand subjects wrote a word after each of 200 stimulus words. Doctor was most often associated with nurse, then sick, health, medicine, hospital…

Following Fano (1961): \(I(x, y) = \log_2 \frac{P(x, y)}{P(x)P(y)}\), comparing the probability of seeing \(x\) and \(y\) together with the probability of seeing them independently.

\(I \gg 0\): a genuine association (joint probability much larger than chance). \(I \approx 0\): no interesting relationship. \(I \ll 0\): complementary distribution (they occur together much less than by chance).

\(P(x)\) and \(P(y)\) from word counts \(f(x)\), \(f(y)\) divided by corpus size \(N\); \(P(x, y)\) from \(f_w(x, y)\), the number of times \(x\) is followed by \(y\) within a window of \(w\) words, divided by \(N\).

Small windows find fixed expressions (bread and butter) and short-range relations; larger ones highlight semantic concepts and relations over larger scales. The paper uses \(w = 5\) as a compromise, large enough for some verb–argument constraints but not so large as to wash out adjacency.

The association ratio becomes unstable for very small counts. A better approach would use t-scores to discard insignificant pairs, but that requires estimating the variance of \(f(x, y)\).

(1) It isn’t symmetric: \(f(x, y)\) counts \(x\) before \(y\), and the order information was deliberately kept as interesting. (2) \(f(x, y)\) can exceed \(f(x)\) if words recur within the window; dividing by \(w - 1\) fixes this (subtracting \(\log_2 4 = 2\) for \(w = 5\)).

Biases “ranging from sexism to syntax”: doctors … nurses far more than nurses … doctors, man … woman more than the reverse, bread … butter, save … life, supposed … to.

Pairs with \(I(x, y) > 3\) tend to be interesting, and those with smaller values generally not (a rule of thumb that could be calibrated against subjective judgments).

If \(x\) and \(y\) each occur about 10 times per million words, chance is \(10^{-10}\), but with corpora of about \(10^7\) words we can’t observe probabilities below about \(10^{-7}\), so we can’t tell whether a pair occurs much less often than chance unless chance is large.

Dentists, nurses, treating, treat, hospitals — credible results like Palermo and Jenkins’s, while pairs such as a … doctors or is … doctors scored about 1, uninteresting.

Sinclair compared the observed probability of set off with the tiny chance probability \(P(set)P(off)\) if words were distributed at random. Mutual information measures exactly that: \(I(set; off) \approx 6.1\) in Sinclair’s corpus and 6.2 in AP news, more than \(2^6 = 64\) times chance.

Set up, set off and set out are clearly associated; set on, set in and set about are not so clear, so the approach suits finding phrasal verbs, at least in some cases.

To separate the preposition to from the infinitive marker. Verbs associated with to/IN include alluding, adhere, amounted, revert, relegated; with to/TO obligated, trying, compelled, enables, supposed, intends.

Parse the corpus into subject–verb–object triples and compute mutual information between verbs and objects across triples, e.g. \(I(drink/V, beer/O) = \log_2 \frac{29N}{660 \times 195} \approx 9.9\). This reveals what you can drink (martinis, champagne, beer, coffee…) or do to a telephone (sit by, disconnect, answer, hang up…).

Lexicographers marked up printouts with coloured pens to find senses. This works for a few dozen lines and two or three senses, but for words like take, save or from the patterns are buried in thousands of lines, beyond what the unaided mind can find and rank.

Association norms apply to function words too: from is significantly associated with save (about 21 times chance), yet save ranks only 319th among verbs associated with from, strong enough to matter but not to pop out of a concordance. Many dictionaries don’t mention it.

Of the 27 bare verbs among the top 100 associated with from, the Cobuild dictionary listed all but seven with from (missing e.g. vary, ferry, strip, divert, forbid, reap). The measure could improve coverage at lower cost.

The tags the authors invented (save PERSON from BAD, save INSTITUTION from ECONOMIC BAD, save ANIMAL from DESTRUCTION) could mostly have been suggested by the association table. It also pointed to patterns they had missed, such as time adverbials (annually, month) going with MONEY objects, and save the forests (ENVIRONMENT).

Although a concordance is indexed by one word, lexicographers use a second word (like from) or a common semantic concept to categorize lines, clustering them into senses by usage (distributional evidence) rather than intuitive meaning.

It uses only distributional evidence (it favours set … for over set … down, not knowing the former is compositional); it’s superficial, unable to cluster words into syntactic classes without a tagger or parser; and neither preprocess reveals similarities like picture/photograph.

As an important tool, rather like an index to the concordances: it helps decide what to look for and gives a quick summary of “what company our words do keep”.