A Neural Probabilistic Language Model

4.2 Results

1. How much did Bengio et al.’s neural LM improve test perplexity?
   On Brown: best MLP (mixed with the trigram) 252 vs 312 for the best n-gram (class-based, 500 classes), about 24% better, and 33% better than the interpolated trigram (336). On AP News: 109 vs 117 for the best Kneser-Ney model, about 8%.
   • Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §4.2, pp. 1148–1149
2. What else did Bengio et al.’s Brown corpus experiments show?
   The network benefited from more context (4 instead of 2 context words helped it, not the n-grams); hidden units helped; mixing with the trigram always helped. Without direct connections training took twice as long (20 vs 10 epochs) but generalized slightly better, perhaps as a tighter bottleneck.
   • Bengio et al., 2003, A Neural Probabilistic Language Model (Journal of Machine Learning Research), §4.2, pp. 1148–1149