learn.ifs.name

Deep contextualized word representations

Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018))

Subscriber deck — free during early access

1 cards in “3.1 Bidirectional language models” · whole deck

3.1 Bidirectional language models

  1. What objective does the ELMo biLM optimize?
    It jointly maximizes forward and backward log-likelihoods: \(\sum_{k=1}^{N} \big(\log p(t_k \mid t_1, \ldots, t_{k-1}) + \log p(t_k \mid t_{k+1}, \ldots, t_N)\big)\), with separate LSTMs per direction but shared token representation and softmax parameters.
    • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §3.1, pp. 2228–2229

Source

  • Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2018), pp. 2227–2237. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/N18-1202 https://aclanthology.org/N18-1202/