A Neural Probabilistic Language Model
Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research)
Subscriber deck — free during early access
1 cards in “5.2 Other Future Work” · whole deck
5.2 Other Future Work
- Which future directions did Bengio et al. propose that later became standard?Tree-structured (hierarchical) output to cut computation by a factor of about \(|V|/\log|V|\); computing gradients for only a subset of output words (e.g. importance sampling); recurrent networks for longer context; and multiple vectors per word for polysemy.
- Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §5.2, pp. 1151–1152
Source
- Yoshua Bengio, Réjean Ducharme, Pascal Vincent and Christian Jauvin. 2003. A Neural Probabilistic Language Model. In Journal of Machine Learning Research, pp. 1137–1155. https://www.jmlr.org/papers/v3/bengio03a.html