A Neural Probabilistic Language Model

5.1 An Energy Minimization Network

1. What is the energy-minimization variant of the neural LM?
   The output word also gets a feature vector: the network reads \(x = (C(w_t), C(w_{t-1}), \ldots, C(w_{t-n+1}))\) and outputs an energy \(E = v \cdot \tanh(d + Hx) + \sum_{i=0}^{n-1} b_{w_{t-i}}\), low for likely sequences; normalizing \(e^{-E}\) over \(w_t\) gives the conditional probability. Hidden units act like products-of-experts experts.
   • Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §5.1, pp. 1150–1151
2. How could Bengio et al.’s energy-based model handle out-of-vocabulary words?
   Initialize a new word’s feature vector as the probability-weighted average of the vectors of words that could occur in the context, \(C(j) \leftarrow \sum_{i \in V} C(i)\, \hat P(i \mid w_{t-n+1}^{t-1})\), then add it to the vocabulary and renormalize.
   • Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §5.1, pp. 1150–1151