GloVe: Global Vectors for Word Representation

3 The GloVe Model

1. Define \(X_{ij}\), \(X_i\) and \(P_{ij}\) in GloVe.
   \(X_{ij}\) is the number of times word \(j\) occurs in the context of word \(i\); \(X_i = \sum_k X_{ik}\); \(P_{ij} = P(j \mid i) = X_{ij} / X_i\), the probability that \(j\) appears in the context of \(i\).
   • Pennington et al., 2014, GloVe (Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §3, pp. 1533–1534
2. Why does GloVe start from ratios of co-occurrence probabilities? Use ice and steam.
   The ratio \(P(k \mid \textit{ice}) / P(k \mid \textit{steam})\) is large for probe words related only to ice, small for those related only to steam, and near 1 for words related to both or neither. Ratios separate relevant from irrelevant words better than raw probabilities (data from a 6B-token corpus):

   probe \(k\)	\(P(k \mid \textit{ice})\)	\(P(k \mid \textit{steam})\)	ratio
   solid	1.9·10−4	2.2·10−5	8.9
   gas	6.6·10−5	7.8·10−4	0.085
   water	3.0·10−3	2.2·10−3	1.36
   fashion	1.7·10−5	1.8·10−5	0.96

   • Pennington et al., 2014, GloVe (Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §3, pp. 1533–1534
3. How is the GloVe model derived from probability ratios?
   Start with \(F(w_i, w_j, \tilde w_k) = P_{ik}/P_{jk}\). Use vector differences, \(F(w_i - w_j, \tilde w_k)\); take a dot product so \(F\) can’t mix dimensions, \(F((w_i - w_j)^\top \tilde w_k)\); require \(F\) to turn differences into ratios, so \(F = \exp\) and \(w_i^\top \tilde w_k = \log P_{ik} = \log X_{ik} - \log X_i\).
   • Pennington et al., 2014, GloVe (Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §3, pp. 1533–1534
4. How does GloVe restore the symmetry between words and context words?
   The term \(\log X_i\) is independent of \(k\), so it becomes a bias \(b_i\); adding a context bias \(\tilde b_k\) gives the symmetric \(w_i^\top \tilde w_k + b_i + \tilde b_k = \log X_{ik}\).
   • Pennington et al., 2014, GloVe (Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §3, pp. 1533–1534
5. Give the GloVe cost function.
   A weighted least squares regression over the vocabulary: \(J = \sum_{i,j=1}^{V} f(X_{ij}) (w_i^\top \tilde w_j + b_i + \tilde b_j - \log X_{ij})^2\). Since \(f(0) = 0\), only nonzero co-occurrences contribute.
   • Pennington et al., 2014, GloVe (Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §3, pp. 1533–1534
6. Why does GloVe need a weighting function, and what properties should it have?
   Unweighted log-count factorization treats all co-occurrences equally, though rare ones are noisy and zeros (75–95% of \(X\)) make \(\log\) diverge. \(f\) should satisfy \(f(0) = 0\) (vanishing fast enough), be non-decreasing so rare pairs aren’t overweighted, and stay relatively small for large \(x\) so frequent pairs aren’t overweighted.
   • Pennington et al., 2014, GloVe (Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §3, pp. 1533–1534
7. What weighting function does GloVe use?
   \(f(x) = (x / x_{\max})^\alpha\) if \(x < x_{\max}\), else 1, with \(x_{\max} = 100\) and \(\alpha = 3/4\) (slightly better than \(\alpha = 1\)), echoing the 3/4 power in word2vec’s noise distribution.

   • Pennington et al., 2014, GloVe (Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §3, pp. 1533–1534