GloVe: Global Vectors for Word Representation

3.1 Relationship to Other Models

1. How is skip-gram’s implicit global objective related to GloVe?
   Grouping identical \((i, j)\) terms, skip-gram’s objective is \(J = -\sum_i X_i H(P_i, Q_i)\), a weighted sum of cross-entropies between corpus distributions \(P_i\) and model softmax \(Q_i\). Replacing cross-entropy with least squares on unnormalized log values and generalizing the weight \(X_i\) to \(f(X_{ij})\) gives GloVe’s cost.
   • Pennington et al., 2014, GloVe (Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §3.1, pp. 1535–1536
2. Why did Pennington et al. prefer least squares to cross-entropy for word vectors?
   Cross-entropy gives too much weight to unlikely events in long-tailed distributions, and it needs a normalized model \(Q\), which requires an expensive sum over the vocabulary. Least squares on log counts drops normalization, and logs tame the very large values of \(X_{ij}\).
   • Pennington et al., 2014, GloVe (Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §3.1, pp. 1535–1536