BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

3.2 Fine-tuning BERT

1. How is BERT fine-tuned for different tasks?
   Plug task inputs into the A/B slots (sentence pairs, hypothesis–premise, question–passage, or text with an empty B) and outputs into a new layer: token representations for tagging or span prediction, \(C\) for classification. All parameters are fine-tuned end to end; self-attention over the packed pair gives cross-attention between the texts.
   • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §3.2, p. 4175
2. How expensive is fine-tuning BERT compared with pretraining?
   Cheap: every result in the paper could be reproduced in at most an hour on a Cloud TPU or a few hours on a GPU from the same pretrained model. GLUE tasks used batch size 32, 3 epochs, and the best of four learning rates (5e-5 to 2e-5).
   • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §3.2, p. 4175