learn.ifs.name

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019))

Subscriber deck — free during early access

20 cards

1 Introduction

  1. What are the feature-based and fine-tuning strategies for using pretrained language representations?
    Feature-based (e.g. ELMo): task-specific architectures use pretrained representations as extra input features. Fine-tuning (e.g. OpenAI GPT): add minimal task-specific parameters and fine-tune all pretrained parameters on the downstream task.
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §1, pp. 4171–4172
  2. What limitation of earlier pretraining did BERT address?
    Standard language models are unidirectional: GPT’s left-to-right Transformer lets each token attend only to earlier tokens, which is sub-optimal for sentence-level tasks and harmful for token-level tasks like QA that need context from both directions. ELMo only concatenates independently trained left-to-right and right-to-left models (shallow bidirectionality).
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §1, pp. 4171–4172
  3. What are BERT’s main contributions?
    Deep bidirectional pretraining via a masked language model; showing that a pretrained model fine-tuned with minimal changes beats many heavily engineered task-specific architectures on both sentence- and token-level tasks; and new state of the art on eleven NLP tasks.
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §1, pp. 4171–4172

3 BERT

  1. What is BERT’s architecture, and what are the two standard sizes?
    A multi-layer bidirectional Transformer encoder. BERT-base: \(L = 12\) layers, hidden size \(H = 768\), \(A = 12\) heads, 110M parameters (same size as GPT). BERT-large: \(L = 24\), \(H = 1024\), \(A = 16\), 340M parameters. The feed-forward size is \(4H\).
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §3, pp. 4173–4174
  2. How does BERT represent its input?
    WordPiece tokens (30,000 vocabulary), always starting with [CLS], whose final hidden state \(C\) serves as the sequence representation for classification. Sentence pairs are packed into one sequence, separated by [SEP], and each token’s input is the sum of its token, segment (A or B) and position embeddings.
    inputtokensegmentposition
    [CLS]E[CLS]EAE0
    theEtheEAE1
    catEcatEAE2
    satEsatEAE3
    [SEP]E[SEP]EAE4
    itEitEBE5
    purrEpurrEBE6
    ##edE##edEBE7
    [SEP]E[SEP]EBE8
    Each row’s three embeddings are summed; purred is split into the WordPieces purr ##ed.
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §3, pp. 4173–4174

3.1 Pre-training BERT

  1. Why can’t a standard language model be trained bidirectionally, and how does BERT get around it?
    Conditioning on both sides in a deep model would let each word indirectly “see itself”, making prediction trivial. BERT’s masked language model (MLM, a cloze task) masks random input tokens and predicts only those from their full left and right context.
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §3.1, pp. 4174–4175
  2. How does BERT choose and corrupt tokens for the masked LM?
    15% of WordPiece positions are chosen for prediction. A chosen token is replaced by [MASK] 80% of the time, by a random token 10% of the time, and left unchanged 10% of the time; the final hidden vector at that position predicts the original token with cross-entropy.
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §3.1, pp. 4174–4175
  3. Why doesn’t BERT always replace chosen tokens with [MASK]?
    [MASK] never appears during fine-tuning, so always using it would create a pretraining/fine-tuning mismatch. Random and unchanged tokens mean the model must keep a good contextual representation of every input token, since it doesn’t know which ones will be predicted.
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §3.1, pp. 4174–4175
  4. What is next sentence prediction (NSP) in BERT?
    A binary pretraining task: for each pair (A, B), B is the actual next sentence 50% of the time (IsNext) and a random corpus sentence otherwise (NotNext), predicted from \(C\). It targets understanding relationships between sentences, which QA and NLI need but language modelling doesn’t capture directly.
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §3.1, pp. 4174–4175
  5. What data was BERT pretrained on, and why does document-level text matter?
    BooksCorpus (800M words) and English Wikipedia text passages (2,500M words). A document-level corpus is needed to extract long contiguous sequences; a shuffled sentence-level corpus like the Billion Word Benchmark would not do.
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §3.1, pp. 4174–4175

3.2 Fine-tuning BERT

  1. How is BERT fine-tuned for different tasks?
    Plug task inputs into the A/B slots (sentence pairs, hypothesis–premise, question–passage, or text with an empty B) and outputs into a new layer: token representations for tagging or span prediction, \(C\) for classification. All parameters are fine-tuned end to end; self-attention over the packed pair gives cross-attention between the texts.
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §3.2, p. 4175
  2. How expensive is fine-tuning BERT compared with pretraining?
    Cheap: every result in the paper could be reproduced in at most an hour on a Cloud TPU or a few hours on a GPU from the same pretrained model. GLUE tasks used batch size 32, 3 epochs, and the best of four learning rates (5e-5 to 2e-5).
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §3.2, p. 4175

4.1 GLUE

  1. How is BERT applied to GLUE classification, and how well did it do?
    The only new parameters are \(W \in \mathbb{R}^{K \times H}\), with loss \(\log(\text{softmax}(C W^\top))\). BERT-large scored 80.5 on the GLUE leaderboard vs 72.8 for OpenAI GPT, with a 4.6-point absolute gain on MNLI (86.7%), and beat BERT-base on every task, especially small ones.
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §4.1, pp. 4175–4176

4.2 SQuAD v1.1

  1. How does BERT predict an answer span for SQuAD?
    It learns a start vector \(S\) and an end vector \(E\): \(P_i^{\text{start}} = \frac{e^{S \cdot T_i}}{\sum_j e^{S \cdot T_j}}\), likewise for the end; a span \((i, j)\) with \(j \ge i\) scores \(S \cdot T_i + E \cdot T_j\), and the best is predicted. Training maximizes the log-likelihood of the correct start and end. Test F1 reached 93.2 (ensemble with TriviaQA).
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §4.2, pp. 4176–4177

4.3 SQuAD v2.0

  1. How does BERT handle unanswerable questions in SQuAD 2.0?
    No-answer is treated as a span starting and ending at [CLS], with score \(s_{\text{null}} = S \cdot C + E \cdot C\). A non-null answer is predicted only if the best span score exceeds \(s_{\text{null}} + \tau\), with \(\tau\) tuned on dev for F1. This gave 83.1 test F1, +5.1 over the previous best.
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §4.3, p. 4177

4.4 SWAG

  1. How was BERT applied to multiple-choice SWAG?
    Each of the four candidate continuations is paired with the context as (A, B); a learned vector dotted with each [CLS] representation scores the choices, normalized by a softmax. BERT-large beat ESIM+ELMo by 27.1 points and GPT by 8.3.
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §4.4, p. 4177

5.1 Effect of Pre-training Tasks

  1. What did BERT’s ablation of pretraining tasks show?
    Removing NSP hurt QNLI, MNLI and SQuAD. A left-to-right LM without NSP (GPT-like) was worse than the MLM on all tasks, with large drops on MRPC and SQuAD (88.5 → 77.8 F1); adding a BiLSTM on top helped SQuAD but not GLUE, and stayed far behind bidirectional pretraining.
    BERT-base variantMNLIMRPCSQuAD F1
    MLM + NSP84.486.788.5
    no NSP83.986.587.9
    left-to-right, no NSP82.177.577.8
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §5.1, p. 4178
  2. Why is concatenating separate left-to-right and right-to-left models (as in ELMo) worse than BERT, according to Devlin et al.?
    It is twice as expensive as one bidirectional model; it is unnatural for QA, since the right-to-left model can’t condition the answer on the question; and it is strictly less powerful, because a deep bidirectional model uses both contexts at every layer.
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §5.1, p. 4178

5.2 Effect of Model Size

  1. What did BERT show about model size and small downstream tasks?
    Larger models were strictly better on all GLUE tasks tried, even MRPC with only 3,600 examples. Scaling up had long helped large tasks like MT and LM; BERT showed it also helps very small tasks when the model is sufficiently pretrained and fine-tuned with few new parameters.
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §5.2, pp. 4178–4179

5.3 Feature-based Approach with BERT

  1. Does BERT also work as a feature extractor without fine-tuning?
    Yes: on CoNLL-2003 NER, feeding frozen BERT-base activations into a BiLSTM, concatenating the top four hidden layers gave 96.1 dev F1, only 0.3 behind fine-tuning the whole model. Features help when a task needs its own architecture or when representations should be computed once.
    • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §5.3, p. 4179

Source

  • Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2019), pp. 4171–4186. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/N19-1423 https://aclanthology.org/N19-1423/