Natural Language Processing (Almost) from Scratch

4.5 Semi-supervised Benchmark Results

1. How were the unsupervised embeddings used for the supervised tasks, and how much did they help?
   The lookup tables of the supervised networks were initialized with the language model embeddings and then fine-tuned. Results with SLL below. The larger language model helped more, suggesting even more unlabeled data would help.

   task	benchmark	NN	NN + LM2
   POS	97.24	96.37	97.20
   chunking	94.29	90.33	93.63
   NER	89.31	81.47	88.67
   SRL	77.92	70.99	74.15

   • Collobert et al., 2011, Natural Language Processing (Almost) from Scratch (Journal of Machine Learning Research), §4.5, pp. 2514–2515