Distributed Representations of Words and Phrases and their Compositionality

2 The Skip-gram Model

1. What is the Skip-gram training objective?
   Maximize the average log probability of the words around each word: \(\frac{1}{T} \sum_{t=1}^{T} \sum_{-c \le j \le c,\, j \ne 0} \log p(w_{t+j} \mid w_t)\), where \(c\) is the context size. Larger \(c\) gives more training examples and can improve accuracy, at the cost of training time.

   • Mikolov et al., 2013, Distributed Representations of Words and Phrases and their Compositionality (Advances in Neural Information Processing Systems 26 (NIPS 2013)), §2
2. How does basic Skip-gram define \(p(w_O \mid w_I)\), and why is it impractical?
   With a softmax over the vocabulary: \(p(w_O \mid w_I) = \frac{\exp(v'^\top_{w_O} v_{w_I})}{\sum_{w=1}^{W} \exp(v'^\top_w v_{w_I})}\), with separate input vectors \(v_w\) and output vectors \(v'_w\). The gradient cost is proportional to the vocabulary size \(W\), often \(10^5\)–\(10^7\).
   • Mikolov et al., 2013, Distributed Representations of Words and Phrases and their Compositionality (Advances in Neural Information Processing Systems 26 (NIPS 2013)), §2