A Neural Probabilistic Language Model

1.2 Relation to Previous Work

1. How does the neural language model differ from class-based n-grams?
   Class-based models (e.g. Brown et al. 1992) assign each word a discrete class, a hard or soft partition of the vocabulary. The neural model represents similarity with a continuous learned feature vector per word, so words can be similar along many dimensions.
   • Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §1.2, pp. 1140–1141
2. How do Bengio et al.’s word vectors differ from LSI-style vectors in information retrieval?
   LSI learns vectors from co-occurrence in documents; here the vectors are learned to represent the distribution of word sequences compactly, jointly with the model. Fixed vectors from principal components of co-occurrence counts did not work well; joint learning did.
   • Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §1.2, pp. 1140–1141