Distributed Representations of Words and Phrases and their Compositionality

4 Learning Phrases

1. How did Mikolov et al. find phrases in text?
   Score bigrams by \(\text{score}(w_i, w_j) = \frac{\text{count}(w_i w_j) - \delta}{\text{count}(w_i) \times \text{count}(w_j)}\); bigrams above a threshold become single tokens (“New York Times”, “Toronto Maple Leafs”, but not “this is”). The discount \(\delta\) prevents phrases of very rare words. 2–4 passes with decreasing thresholds form longer phrases.
   • Mikolov et al., 2013, Distributed Representations of Words and Phrases and their Compositionality (Advances in Neural Information Processing Systems 26 (NIPS 2013)), §4
2. How were phrase vectors evaluated?
   With a new phrase analogy test set (3218 questions in five categories, e.g. newspapers, NHL teams, airlines, company executives): “Montreal : Montreal Canadiens :: Toronto : ?” should give “Toronto Maple Leafs”.
   • Mikolov et al., 2013, Distributed Representations of Words and Phrases and their Compositionality (Advances in Neural Information Processing Systems 26 (NIPS 2013)), §4