GloVe: Global Vectors for Word Representation

4.2 Corpora and training details

1. How was GloVe trained?
   On corpora of 1B to 42B tokens with a 400,000-word vocabulary, a 10-word context window on each side with pairs \(d\) words apart counted as \(1/d\), AdaGrad on randomly sampled nonzero entries of \(X\), 50–100 iterations. The final vectors are \(W + \tilde W\), the sum of word and context vectors, which gave a small boost.
   • Pennington et al., 2014, GloVe (Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §4.2, pp. 1538–1539
2. Why does GloVe use \(W + \tilde W\) as the word vectors?
   With a symmetric \(X\), the two sets are equivalent up to random initialization; summing them, like combining several trained networks, reduces noise and overfitting and gave a small improvement, mostly on semantic analogies.
   • Pennington et al., 2014, GloVe (Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §4.2, pp. 1538–1539