Deep contextualized word representations

Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018))

Subscriber deck — free during early access

1 cards in “5.6 Contextual vs. sub-word information” · whole deck

5.6 Contextual vs. sub-word information

Do ELMo’s gains come from context or from character-level subword information?
Mostly context: replacing GloVe with only ELMo’s character-based token layer helped slightly (SQuAD dev F1 80.8 → 81.4), but full ELMo gave 85.3. Adding GloVe on top of ELMo gave only marginal further gains.
- Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §5.6, p. 2235

Source

Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2018), pp. 2227–2237. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/N18-1202 https://aclanthology.org/N18-1202/