BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

3.1 Pre-training BERT

1. Why can’t a standard language model be trained bidirectionally, and how does BERT get around it?
   Conditioning on both sides in a deep model would let each word indirectly “see itself”, making prediction trivial. BERT’s masked language model (MLM, a cloze task) masks random input tokens and predicts only those from their full left and right context.
   • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §3.1, pp. 4174–4175
2. How does BERT choose and corrupt tokens for the masked LM?
   15% of WordPiece positions are chosen for prediction. A chosen token is replaced by [MASK] 80% of the time, by a random token 10% of the time, and left unchanged 10% of the time; the final hidden vector at that position predicts the original token with cross-entropy.
   • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §3.1, pp. 4174–4175
3. Why doesn’t BERT always replace chosen tokens with [MASK]?
   [MASK] never appears during fine-tuning, so always using it would create a pretraining/fine-tuning mismatch. Random and unchanged tokens mean the model must keep a good contextual representation of every input token, since it doesn’t know which ones will be predicted.
   • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §3.1, pp. 4174–4175
4. What is next sentence prediction (NSP) in BERT?
   A binary pretraining task: for each pair (A, B), B is the actual next sentence 50% of the time (IsNext) and a random corpus sentence otherwise (NotNext), predicted from \(C\). It targets understanding relationships between sentences, which QA and NLI need but language modelling doesn’t capture directly.
   • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §3.1, pp. 4174–4175
5. What data was BERT pretrained on, and why does document-level text matter?
   BooksCorpus (800M words) and English Wikipedia text passages (2,500M words). A document-level corpus is needed to extract long contiguous sequences; a shuffled sentence-level corpus like the Billion Word Benchmark would not do.
   • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §3.1, pp. 4174–4175