BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019))
Subscriber deck — free during early access
1 cards in “5.2 Effect of Model Size” · whole deck
5.2 Effect of Model Size
- What did BERT show about model size and small downstream tasks?Larger models were strictly better on all GLUE tasks tried, even MRPC with only 3,600 examples. Scaling up had long helped large tasks like MT and LM; BERT showed it also helps very small tasks when the model is sufficiently pretrained and fine-tuned with few new parameters.
- Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §5.2, pp. 4178–4179
Source
- Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2019), pp. 4171–4186. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/N19-1423 https://aclanthology.org/N19-1423/