GloVe: Global Vectors for Word Representation

1 Introduction

1. What are the two main families of word vector methods, and the drawback of each, according to Pennington et al.?
   Global matrix factorization (e.g. LSA) uses corpus statistics efficiently but does poorly on word analogies, a sign of sub-optimal vector space structure. Local context window methods (e.g. skip-gram) do better on analogies but use the statistics poorly, training on separate windows instead of global co-occurrence counts.
   • Pennington et al., 2014, GloVe (Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §1, p. 1532
2. What does GloVe aim to combine?
   The efficiency of global count-based methods with the meaningful linear substructure of window-based prediction methods: a global log-bilinear regression model trained only on the nonzero entries of a word–word co-occurrence matrix.
   • Pennington et al., 2014, GloVe (Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §1, p. 1532