BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

1 Introduction

1. What are the feature-based and fine-tuning strategies for using pretrained language representations?
   Feature-based (e.g. ELMo): task-specific architectures use pretrained representations as extra input features. Fine-tuning (e.g. OpenAI GPT): add minimal task-specific parameters and fine-tune all pretrained parameters on the downstream task.
   • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §1, pp. 4171–4172
2. What limitation of earlier pretraining did BERT address?
   Standard language models are unidirectional: GPT’s left-to-right Transformer lets each token attend only to earlier tokens, which is sub-optimal for sentence-level tasks and harmful for token-level tasks like QA that need context from both directions. ELMo only concatenates independently trained left-to-right and right-to-left models (shallow bidirectionality).
   • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §1, pp. 4171–4172
3. What are BERT’s main contributions?
   Deep bidirectional pretraining via a masked language model; showing that a pretrained model fine-tuned with minimal changes beats many heavily engineered task-specific architectures on both sentence- and token-level tasks; and new state of the art on eleven NLP tasks.
   • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §1, pp. 4171–4172