learn.ifs.name

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019))

Subscriber deck — free during early access

1 cards in “4.4 SWAG” · whole deck

4.4 SWAG

  1. How was BERT applied to multiple-choice SWAG?
    Each of the four candidate continuations is paired with the context as (A, B); a learned vector dotted with each [CLS] representation scores the choices, normalized by a softmax. BERT-large beat ESIM+ELMo by 27.1 points and GPT by 8.3.
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §4.4, p. 4177

Source

  • Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2019), pp. 4171–4186. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/N19-1423 https://aclanthology.org/N19-1423/