GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

1 Introduction

1. What is the GLUE benchmark, and what is it meant to encourage?
   The General Language Understanding Evaluation: nine existing English sentence and sentence-pair NLU tasks, an online evaluation platform with private test labels, and a diagnostic set. Because some tasks have little training data, it favours models that share general linguistic knowledge across tasks and learn sample-efficiently.
   • Wang et al., 2019, GLUE (7th International Conference on Learning Representations (ICLR 2019)), §1
2. What constraints does GLUE place on model architecture?
   None beyond processing single sentences and sentence pairs and producing predictions for every task: models can use any representation or cross-sentence interaction, unlike SentEval, which only evaluates sentence-to-vector encoders.
   • Wang et al., 2019, GLUE (7th International Conference on Learning Representations (ICLR 2019)), §1
3. Why did GLUE reuse existing datasets instead of creating new ones?
   Existing datasets were already implicitly agreed upon by the community as challenging and interesting; GLUE obtained private test labels for several of them so the benchmark could be used fairly.
   • Wang et al., 2019, GLUE (7th International Conference on Learning Representations (ICLR 2019)), §1