BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

4.1 GLUE

1. How is BERT applied to GLUE classification, and how well did it do?
   The only new parameters are \(W \in \mathbb{R}^{K \times H}\), with loss \(\log(\text{softmax}(C W^\top))\). BERT-large scored 80.5 on the GLUE leaderboard vs 72.8 for OpenAI GPT, with a 4.6-point absolute gain on MNLI (86.7%), and beat BERT-base on every task, especially small ones.
   • Devlin et al., 2019, BERT (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §4.1, pp. 4175–4176