Distributed Representations of Words and Phrases and their Compositionality
Mikolov et al., 2013, Distributed Representations of Words and Phrases and their Compositionality (Advances in Neural Information Processing Systems 26 (NIPS 2013))
Subscriber deck — free during early access
1 cards in “7 Conclusion” · whole deck
7 Conclusion
- What hyperparameter choices mattered most for Skip-gram quality, according to Mikolov et al.?The model architecture (Skip-gram vs CBOW), vector size, subsampling rate and training window size. The best choices are task-specific. Negative sampling learns especially good vectors for frequent words; hierarchical softmax with subsampling did best for rare phrases.
- Mikolov et al., 2013, Distributed Representations of Words and Phrases and their Compositionality (Advances in Neural Information Processing Systems 26 (NIPS 2013)), §7
Source
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado and Jeffrey Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information Processing Systems 26 (NIPS 2013). https://papers.nips.cc/paper/2013/hash/9aa42b31882ec039965f3c4923ce901b-Abstract.html