Natural Language Processing (Almost) from Scratch

4.3 Training Language Models

1. How were the large ranking language models trained?
   On Wikipedia (631M words) and then Wikipedia plus Reuters RCV1 (852M words), with window size 11 and 100 hidden units, using a curriculum of growing dictionaries (5k up to 100k words), each initialized from the previous network. LM1 took about four weeks, LM2 three more. Hyperparameters were chosen by “breeding”.
   • Collobert et al., 2011, Natural Language Processing (Almost) from Scratch (Journal of Machine Learning Research), §4.3, pp. 2512–2513
2. What is “breeding” in Collobert et al.’s model selection?
   Without enough compute for grid search, train \(k\) parameter settings in parallel for some days, keep the best by validation ranking loss, and start the next round from slight variations of the winners, initialized with their parents’ embeddings so training continues online.
   • Collobert et al., 2011, Natural Language Processing (Almost) from Scratch (Journal of Machine Learning Research), §4.3, pp. 2512–2513