BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

5.1 Effect of Pre-training Tasks

1. What did BERT’s ablation of pretraining tasks show?
   Removing NSP hurt QNLI, MNLI and SQuAD. A left-to-right LM without NSP (GPT-like) was worse than the MLM on all tasks, with large drops on MRPC and SQuAD (88.5 → 77.8 F1); adding a BiLSTM on top helped SQuAD but not GLUE, and stayed far behind bidirectional pretraining.

   BERT-base variant	MNLI	MRPC	SQuAD F1
   MLM + NSP	84.4	86.7	88.5
   no NSP	83.9	86.5	87.9
   left-to-right, no NSP	82.1	77.5	77.8

   • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §5.1, p. 4178
2. Why is concatenating separate left-to-right and right-to-left models (as in ELMo) worse than BERT, according to Devlin et al.?
   It is twice as expensive as one bidirectional model; it is unnatural for QA, since the right-to-left model can’t condition the answer on the question; and it is strictly less powerful, because a deep bidirectional model uses both contexts at every layer.
   • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §5.1, p. 4178