GloVe: Global Vectors for Word Representation

Global matrix factorization (e.g. LSA) uses corpus statistics efficiently but does poorly on word analogies, a sign of sub-optimal vector space structure. Local context window methods (e.g. skip-gram) do better on analogies but use the statistics poorly, training on separate windows instead of global co-occurrence counts.

The efficiency of global count-based methods with the meaningful linear substructure of window-based prediction methods: a global log-bilinear regression model trained only on the nonzero entries of a word–word co-occurrence matrix.

The most frequent words dominate similarity: co-occurring with the or and affects similarity a lot while saying little about meaning. Fixes compress the counts, which span 8–9 orders of magnitude: entropy or correlation normalization (COALS), positive PMI, square roots (Hellinger PCA).

They don’t operate directly on co-occurrence statistics; scanning context windows across the whole corpus fails to take advantage of the vast repetition in the data.

\(X_{ij}\) is the number of times word \(j\) occurs in the context of word \(i\); \(X_i = \sum_k X_{ik}\); \(P_{ij} = P(j \mid i) = X_{ij} / X_i\), the probability that \(j\) appears in the context of \(i\).

The ratio \(P(k \mid \textit{ice}) / P(k \mid \textit{steam})\) is large for probe words related only to ice, small for those related only to steam, and near 1 for words related to both or neither. Ratios separate relevant from irrelevant words better than raw probabilities (data from a 6B-token corpus):

probe \(k\)	\(P(k \mid \textit{ice})\)	\(P(k \mid \textit{steam})\)	ratio
solid	1.9·10−4	2.2·10−5	8.9
gas	6.6·10−5	7.8·10−4	0.085
water	3.0·10−3	2.2·10−3	1.36
fashion	1.7·10−5	1.8·10−5	0.96

Start with \(F(w_i, w_j, \tilde w_k) = P_{ik}/P_{jk}\). Use vector differences, \(F(w_i - w_j, \tilde w_k)\); take a dot product so \(F\) can’t mix dimensions, \(F((w_i - w_j)^\top \tilde w_k)\); require \(F\) to turn differences into ratios, so \(F = \exp\) and \(w_i^\top \tilde w_k = \log P_{ik} = \log X_{ik} - \log X_i\).

The term \(\log X_i\) is independent of \(k\), so it becomes a bias \(b_i\); adding a context bias \(\tilde b_k\) gives the symmetric \(w_i^\top \tilde w_k + b_i + \tilde b_k = \log X_{ik}\).

A weighted least squares regression over the vocabulary: \(J = \sum_{i,j=1}^{V} f(X_{ij}) (w_i^\top \tilde w_j + b_i + \tilde b_j - \log X_{ij})^2\). Since \(f(0) = 0\), only nonzero co-occurrences contribute.

Unweighted log-count factorization treats all co-occurrences equally, though rare ones are noisy and zeros (75–95% of \(X\)) make \(\log\) diverge. \(f\) should satisfy \(f(0) = 0\) (vanishing fast enough), be non-decreasing so rare pairs aren’t overweighted, and stay relatively small for large \(x\) so frequent pairs aren’t overweighted.

\(f(x) = (x / x_{\max})^\alpha\) if \(x < x_{\max}\), else 1, with \(x_{\max} = 100\) and \(\alpha = 3/4\) (slightly better than \(\alpha = 1\)), echoing the 3/4 power in word2vec’s noise distribution.

Grouping identical \((i, j)\) terms, skip-gram’s objective is \(J = -\sum_i X_i H(P_i, Q_i)\), a weighted sum of cross-entropies between corpus distributions \(P_i\) and model softmax \(Q_i\). Replacing cross-entropy with least squares on unnormalized log values and generalizing the weight \(X_i\) to \(f(X_{ij})\) gives GloVe’s cost.

Cross-entropy gives too much weight to unlikely events in long-tailed distributions, and it needs a normalized model \(Q\), which requires an expensive sum over the vocabulary. Least squares on log counts drops normalization, and logs tame the very large values of \(X_{ij}\).

Proportional to the number of nonzero entries of \(X\), at most \(O(|V|^2)\). Assuming co-occurrence counts follow a power law in pair frequency rank with \(\alpha = 1.25\) (as observed), the number of nonzeros is \(O(|C|^{0.8})\), better than window-based methods, which scale as \(O(|C|)\) in corpus size.

On the word analogy task (19,544 semantic and syntactic questions answered by cosine nearest neighbour to \(w_b - w_a + w_c\)), on five word similarity data sets (Spearman correlation with human judgments), and as extra features in a CRF for CoNLL-2003 NER.

On corpora of 1B to 42B tokens with a 400,000-word vocabulary, a 10-word context window on each side with pairs \(d\) words apart counted as \(1/d\), AdaGrad on randomly sampled nonzero entries of \(X\), 50–100 iterations. The final vectors are \(W + \tilde W\), the sum of word and context vectors, which gave a small boost.

With a symmetric \(X\), the two sets are equivalent up to random initialization; summing them, like combining several trained networks, reduces noise and overfitting and gave a small improvement, mostly on semantic analogies.

75.0% overall with 300-dimensional vectors on 42B tokens; on 6B tokens 71.7% vs 69.1% for skip-gram and 65.7% for CBOW trained by the authors with word2vec. It often beat other models with smaller vectors and corpora; SVD baselines didn’t scale to the larger corpus.

Best on the similarity sets, beating the word2vec CBOW vectors trained on 100B words with less than half the data. As CRF features for NER, best on all metrics except the CoNLL test set, where HPCA was slightly better.

Returns diminish above about 200 dimensions. Small and asymmetric (left-only) windows help syntactic analogies, since syntax depends on immediate context and word order; larger windows capture more semantic, non-local information.

Syntactic accuracy rose monotonically with corpus size. Semantic accuracy was higher with the smaller Wikipedia corpora than with the larger Gigaword news corpus, probably because many semantic analogies are about cities and countries, which Wikipedia covers comprehensively.

With the same corpus, vocabulary, window size and training time (GloVe iterations vs word2vec negative samples), GloVe was consistently better, reaching better results faster. word2vec got worse with more than about 10 negative samples.

The two families aren’t fundamentally different, since both probe the same co-occurrence statistics; count-based methods capture global statistics more efficiently. GloVe uses count data while capturing the linear substructures of log-bilinear prediction models like word2vec.