Natural Language Processing (Almost) from Scratch

5. Multi-Task Learning

1. How did Collobert et al. train several tasks jointly, and what did it give?
   Networks for different tasks share the lookup tables and first hidden (or convolution) layer, with task-specific last layers; training alternates examples from each task’s own data set. Gains were marginal; the most important “multi-task” component was the unsupervised learning of word embeddings.
   • Collobert et al., 2011, Natural Language Processing (Almost) from Scratch (Journal of Machine Learning Research), 5. Multi-Task Learning, pp. 2516–2518
2. Why does discovering shared internal representations require joint training rather than joint decoding?
   Joint decoding combines separately trained models at test time, so their extra dependency paths can’t involve unobserved variables. Shared hidden representations are unobserved, so they can only be learned by training the models together.
   • Collobert et al., 2011, Natural Language Processing (Almost) from Scratch (Journal of Machine Learning Research), 5. Multi-Task Learning, pp. 2516–2518