A Neural Probabilistic Language Model

3.2 Parameter-Parallel Processing

1. How did Bengio et al. parallelize training across a cluster?
   Across the output units, where almost all computation happens (about 99.7% for the AP News network): each CPU computes and updates a block of outputs, duplicating the cheap input and hidden computations. CPUs share only the softmax normalizer and gradients on the hidden and feature layers, via MPI Allreduce; communication was 1/15 of total time.
   • Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §3.2, pp. 1144–1147
2. How is the softmax computed stably across processors in Bengio et al.’s implementation?
   Each CPU finds the maximum \(q_i\) of its \(y_j\); an extra Allreduce shares the overall maximum \(Q\), and \(p_j = e^{y_j - Q}\) is computed, so no exponential overflows and at least one \(p_j\) is non-zero; then the sum of \(p_j\) is shared to normalize.
   • Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §3.2, pp. 1144–1147