A Neural Probabilistic Language Model

3 Parallel Implementation

1. Why is a neural LM much more expensive to run than an n-gram model?
   An n-gram probability is a normalized relative frequency looked up directly, but the neural model must compute the output activations for every word in the vocabulary to normalize the softmax; the output layer is the main bottleneck.
   • Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §3, pp. 1143–1144
2. What did Bengio et al. learn from data-parallel training on a shared-memory machine?
   Locking parameter writes left processors mostly waiting, so they let processors write asynchronously; occasionally an update is overwritten, adding a little noise that didn’t visibly slow training.
   • Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §3, pp. 1143–1144