A Neural Probabilistic Language Model

1 Introduction

1. What is the curse of dimensionality in language modelling, according to Bengio et al.?
   Modelling the joint distribution of many discrete variables needs huge numbers of parameters: 10 consecutive words over a 100,000-word vocabulary have potentially \(100000^{10} - 1 = 10^{50} - 1\) free parameters. Test word sequences are likely to differ from all training sequences, and discrete spaces offer no obvious smoothness to generalize with.
   • Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §1, pp. 1137–1138
2. How do n-gram models generalize to unseen word sequences, in Bengio et al.’s view?
   By “gluing” very short overlapping pieces of length 1, 2, … up to \(n\) seen in training: backing off or interpolating to shorter contexts when a combination is unseen (Katz 1987; Jelinek and Mercer 1980).
   • Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §1, pp. 1137–1138
3. What two weaknesses of n-gram models did Bengio et al. target?
   They ignore context beyond 1 or 2 words, and they ignore similarity between words: having seen “The cat is walking in the bedroom” should make “A dog was running in a room” almost as likely, since dog/cat, the/a, room/bedroom play similar roles.
   • Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §1, pp. 1137–1138