Deep contextualized word representations
Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018))
Subscriber deck — free during early access
1 cards in “3.3 Using biLMs for supervised NLP tasks” · whole deck
3.3 Using biLMs for supervised NLP tasks
- How is ELMo added to a supervised model?Run the frozen biLM, record all layer representations, and concatenate the learned ELMo vector with the model’s token representation: \([x_k; \text{ELMo}_k]\) goes into the task RNN. Some tasks (SNLI, SQuAD) also gain from adding ELMo at the RNN output. Dropout on ELMo and an L2 penalty pulling layer weights toward an average help.
- Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §3.3, pp. 2229–2230
Source
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2018), pp. 2227–2237. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/N18-1202 https://aclanthology.org/N18-1202/