Deep contextualized word representations

1 Introduction

1. What two properties should good word representations model, according to Peters et al.?
   Complex characteristics of word use (syntax and semantics) and how these uses vary across linguistic contexts, i.e. polysemy. Type embeddings like word2vec and GloVe give each word one context-independent vector.
   • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §1, pp. 2227–2228
2. What is ELMo?
   Embeddings from Language Models: each token’s representation is a function of the entire input sentence, computed from the internal states of a deep bidirectional language model (biLM) pretrained on a large corpus.
   • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §1, pp. 2227–2228
3. How is ELMo “deep” compared with earlier contextual vectors (TagLM, CoVe)?
   Earlier approaches used only the top LSTM layer. ELMo is a learned, task-specific linear combination of all biLM layers, which markedly improves performance.
   • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §1, pp. 2227–2228
4. What did adding ELMo achieve across tasks?
   Simply adding ELMo to strong existing models set a new state of the art on all six tasks tried (question answering, textual entailment, semantic role labeling, coreference, NER, sentiment), with relative error reductions of 6–20%.
   • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §1, pp. 2227–2228