A Neural Probabilistic Language Model

2 A Neural Model

1. How do Bengio et al. decompose \(f(w_t, \ldots, w_{t-n+1}) = \hat P(w_t \mid w_1^{t-1})\)?
   Into a mapping \(C\) from each word \(i\) to a feature vector \(C(i) \in \mathbb{R}^m\) (a \(|V| \times m\) matrix of free parameters, shared across all context positions), and a function \(g\) mapping the context’s feature vectors to a distribution over the next word: \(f(i, w_{t-1}, \ldots, w_{t-n+1}) = g(i, C(w_{t-1}), \ldots, C(w_{t-n+1}))\).
   • Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §2, pp. 1141–1142
2. Give the computation of Bengio et al.’s neural language model.
   Concatenate the context word vectors \(x = (C(w_{t-1}), \ldots, C(w_{t-n+1}))\); compute \(y = b + Wx + U \tanh(d + Hx)\), where \(W\) are optional direct connections (or zero); then \(\hat P(w_t \mid \ldots) = e^{y_{w_t}} / \sum_i e^{y_i}\) (softmax).

   • Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §2, pp. 1141–1142
3. How many free parameters does Bengio et al.’s model have, and how does it scale?
   \(\theta = (b, d, W, U, H, C)\), dominated by \(|V|(nm + h)\): linear in vocabulary size and in the order \(n\), not exponential as for n-gram tables. More sharing (time-delay or recurrent networks) could make it sublinear in \(n\).
   • Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §2, pp. 1141–1142
4. How is Bengio et al.’s neural LM trained?
   By stochastic gradient ascent on the penalized log-likelihood \(L = \frac{1}{T} \sum_t \log f(w_t, \ldots, w_{t-n+1}; \theta) + R(\theta)\), with weight decay on weights and \(C\) but not biases. Only the feature vectors of words in the current input window need updating after each example.
   • Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §2, pp. 1141–1142
5. Why did Bengio et al. mix their neural model with an interpolated trigram?
   Mixing the probability predictions (fixed weight 0.5, a learned weight, or weights conditioned on context frequency) always reduced perplexity, suggesting the two models make their errors (low probability on the observed word) in different places.
   • Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §2, pp. 1141–1142