GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

4 Diagnostic Dataset

1. What is GLUE’s diagnostic dataset?
   About 1100 expert-constructed NLI examples (from naturally occurring sentences, minimally edited) tagged with linguistic phenomena in four coarse categories: lexical semantics, predicate–argument structure, logic, and knowledge and common sense. Each pair is labelled in both directions.
   • Wang et al., 2019, GLUE (7th International Conference on Learning Representations (ICLR 2019)), §4
2. How is the GLUE diagnostic set evaluated, and how was it checked for artifacts?
   With \(R_3\), a three-class generalization of the Matthews correlation, since labels are unbalanced (42% entailment, 35% neutral, 23% contradiction). Hypothesis-only fastText classifiers trained on SNLI and MNLI scored near chance on it (32.7% and 36.4%); human annotators reached \(R_3 = 0.80\).
   • Wang et al., 2019, GLUE (7th International Conference on Learning Representations (ICLR 2019)), §4
3. How should GLUE diagnostic results be used?
   As an analysis tool for error analysis, comparing models and building adversarial examples, not as a benchmark: performance should be compared between models but not between categories, and it doesn’t predict downstream performance.
   • Wang et al., 2019, GLUE (7th International Conference on Learning Representations (ICLR 2019)), §4