Word Association Norms, Mutual Information, and Lexicography

4 An information theoretic measure

1. Define (pointwise) mutual information as used by Church and Hanks.
   Following Fano (1961): \(I(x, y) = \log_2 \frac{P(x, y)}{P(x)P(y)}\), comparing the probability of seeing \(x\) and \(y\) together with the probability of seeing them independently.
   • Church & Hanks, 1990, Word Association Norms, Mutual Information, and Lexicography (Computational Linguistics), §4, p. 23
2. What do large positive, near-zero and large negative values of \(I(x, y)\) mean?
   \(I \gg 0\): a genuine association (joint probability much larger than chance). \(I \approx 0\): no interesting relationship. \(I \ll 0\): complementary distribution (they occur together much less than by chance).
   • Church & Hanks, 1990, Word Association Norms, Mutual Information, and Lexicography (Computational Linguistics), §4, p. 23
3. How did Church and Hanks estimate the probabilities for the association ratio?
   \(P(x)\) and \(P(y)\) from word counts \(f(x)\), \(f(y)\) divided by corpus size \(N\); \(P(x, y)\) from \(f_w(x, y)\), the number of times \(x\) is followed by \(y\) within a window of \(w\) words, divided by \(N\).
   • Church & Hanks, 1990, Word Association Norms, Mutual Information, and Lexicography (Computational Linguistics), §4, p. 23
4. How does the window size change what the association ratio finds?
   Small windows find fixed expressions (bread and butter) and short-range relations; larger ones highlight semantic concepts and relations over larger scales. The paper uses \(w = 5\) as a compromise, large enough for some verb–argument constraints but not so large as to wash out adjacency.
   • Church & Hanks, 1990, Word Association Norms, Mutual Information, and Lexicography (Computational Linguistics), §4, p. 23
5. What do the mean and variance of the separation between two words reveal?

   relation	example	separation
   fixed expression	bread … butter	always 2, variance 0
   compound	computer scientist	about 1, little variance
   semantic	man … woman	large variance, either order
   lexical	refraining / coming / keeping … from	fixed to almost always separated by an argument

   • Church & Hanks, 1990, Word Association Norms, Mutual Information, and Lexicography (Computational Linguistics), §4, p. 23
6. Why did Church and Hanks ignore pairs with \(f(x, y) \le 5\)?
   The association ratio becomes unstable for very small counts. A better approach would use t-scores to discard insignificant pairs, but that requires estimating the variance of \(f(x, y)\).
   • Church & Hanks, 1990, Word Association Norms, Mutual Information, and Lexicography (Computational Linguistics), §4, p. 24
7. In what two ways does the association ratio differ from mutual information?
   (1) It isn’t symmetric: \(f(x, y)\) counts \(x\) before \(y\), and the order information was deliberately kept as interesting. (2) \(f(x, y)\) can exceed \(f(x)\) if words recur within the window; dividing by \(w - 1\) fixes this (subtracting \(\log_2 4 = 2\) for \(w = 5\)).
   • Church & Hanks, 1990, Word Association Norms, Mutual Information, and Lexicography (Computational Linguistics), §4, p. 24
8. What asymmetries did the association ratio show in news text?
   Biases “ranging from sexism to syntax”: doctors … nurses far more than nurses … doctors, man … woman more than the reverse, bread … butter, save … life, supposed … to.
   • Church & Hanks, 1990, Word Association Norms, Mutual Information, and Lexicography (Computational Linguistics), §4, p. 24
9. What rough threshold did Church and Hanks suggest for interesting associations?
   Pairs with \(I(x, y) > 3\) tend to be interesting, and those with smaller values generally not (a rule of thumb that could be calibrated against subjective judgments).
   • Church & Hanks, 1990, Word Association Norms, Mutual Information, and Lexicography (Computational Linguistics), §4, p. 24
10. Why are strongly negative association scores rarely observable?
    If \(x\) and \(y\) each occur about 10 times per million words, chance is \(10^{-10}\), but with corpora of about \(10^7\) words we can’t observe probabilities below about \(10^{-7}\), so we can’t tell whether a pair occurs much less often than chance unless chance is large.
    • Church & Hanks, 1990, Word Association Norms, Mutual Information, and Lexicography (Computational Linguistics), §4, p. 24
11. Which words were most associated with doctor by the association ratio?
    Dentists, nurses, treating, treat, hospitals — credible results like Palermo and Jenkins’s, while pairs such as a … doctors or is … doctors scored about 1, uninteresting.
    • Church & Hanks, 1990, Word Association Norms, Mutual Information, and Lexicography (Computational Linguistics), §4, p. 24