A Neural Probabilistic Language Model

1.1 Fighting the Curse of Dimensionality with Distributed Representations

1. Summarize Bengio et al.’s approach in three steps.
   (1) Associate each vocabulary word with a distributed, real-valued feature vector in \(\mathbb{R}^m\); (2) express the joint probability of word sequences in terms of these vectors; (3) learn the word vectors and the probability function’s parameters simultaneously.
   • Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §1.1, pp. 1139–1140
2. Why does a distributed word representation help a language model generalize?
   Similar words get similar feature vectors, and the probability function is smooth in those features, so a small change in features changes probabilities only slightly. Each training sentence thus raises the probability of a combinatorial number of “neighbouring” sentences.
   • Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §1.1, pp. 1139–1140