BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019))
Subscriber deck — free during early access
1 cards in “5.3 Feature-based Approach with BERT” · whole deck
5.3 Feature-based Approach with BERT
- Does BERT also work as a feature extractor without fine-tuning?Yes: on CoNLL-2003 NER, feeding frozen BERT-base activations into a BiLSTM, concatenating the top four hidden layers gave 96.1 dev F1, only 0.3 behind fine-tuning the whole model. Features help when a task needs its own architecture or when representations should be computed once.
- Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §5.3, p. 4179
Source
- Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2019), pp. 4171–4186. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/N19-1423 https://aclanthology.org/N19-1423/