Deep contextualized word representations
Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018))
Subscriber deck — free during early access
1 cards in “3.2 ELMo” · whole deck
3.2 ELMo
- Give the formula for the ELMo vector of token \(k\).\(\text{ELMo}_k^{task} = \gamma^{task} \sum_{j=0}^{L} s_j^{task} h_{k,j}^{LM}\), where \(h_{k,0}^{LM}\) is the token layer, \(h_{k,j}^{LM} = [\overrightarrow{h}_{k,j}; \overleftarrow{h}_{k,j}]\) for biLSTM layer \(j\), \(s^{task}\) are softmax-normalized weights and \(\gamma^{task}\) scales the whole vector.
The task learns the weights sj (softmax-normalized) and the scale γ; the biLM is frozen - Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §3.2, p. 2229
Source
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2018), pp. 2227–2237. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/N18-1202 https://aclanthology.org/N18-1202/