GloVe: Global Vectors for Word Representation

2 Related Work

1. What problem with raw term–term co-occurrence counts (e.g. HAL) did later methods address?
   The most frequent words dominate similarity: co-occurring with the or and affects similarity a lot while saying little about meaning. Fixes compress the counts, which span 8–9 orders of magnitude: entropy or correlation normalization (COALS), positive PMI, square roots (Hellinger PCA).
   • Pennington et al., 2014, GloVe (Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §2, p. 1533
2. What disadvantage of window-based methods like skip-gram does GloVe target?
   They don’t operate directly on co-occurrence statistics; scanning context windows across the whole corpus fails to take advantage of the vast repetition in the data.
   • Pennington et al., 2014, GloVe (Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §2, p. 1533