GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

The General Language Understanding Evaluation: nine existing English sentence and sentence-pair NLU tasks, an online evaluation platform with private test labels, and a diagnostic set. Because some tasks have little training data, it favours models that share general linguistic knowledge across tasks and learn sample-efficiently.

None beyond processing single sentences and sentence pairs and producing predictions for every task: models can use any representation or cross-sentence interaction, unlike SentEval, which only evaluates sentence-to-vector encoders.

Existing datasets were already implicitly agreed upon by the community as challenging and interesting; GLUE obtained private test labels for several of them so the benchmark could be used fairly.

decaNLP recasts ten tasks (including summarization and text-to-SQL) as question answering, pursuing the ambitious goal of unifying NLU as QA. GLUE rewards good performance on a circumscribed set of classification and regression tasks with current methods, and adds a leaderboard and diagnostic toolkit.

Grouped by type, with their metrics:

type	tasks (metric)
single sentence	CoLA acceptability (Matthews corr.), SST-2 sentiment (acc.)
similarity, paraphrase	MRPC (acc./F1), QQP (acc./F1), STS-B similarity score (Pearson/Spearman)
inference	MNLI (matched/mismatched acc.), QNLI, RTE, WNLI (acc.)

The Corpus of Linguistic Acceptability: sentences from linguistics books and articles labelled grammatical or not. Its classes are unbalanced, and Matthews correlation (from −1 to 1, 0 for uninformed guessing) suits unbalanced binary classification.

Their classes are unbalanced (MRPC 68% positive, QQP 63% negative), so accuracy alone is misleading; STS-B, a regression task predicting 1–5 similarity scores, uses Pearson and Spearman correlation.

Each question is paired with each sentence of its context paragraph (dropping low-overlap pairs), and the task is whether the sentence contains the answer. This drops the need to select the exact span, but also the assumptions that the answer is present and that lexical overlap is a reliable cue.

The Winograd Schema Challenge recast as NLI: replace the ambiguous pronoun with each candidate referent and ask whether the result is entailed. The test set is imbalanced (65% not entailment) and the development set is adversarial (hypotheses shared with training examples get opposite labels), so memorization hurts.

By combining RTE1, RTE2, RTE3 and RTE5 (news and Wikipedia text) and collapsing neutral and contradiction into “not entailment” for a two-class task; it has only about 2.5k training examples.

Predictions on the test sets are uploaded to gluebenchmark.com; the score is the macro-average of per-task scores, where tasks with two metrics contribute their unweighted average.

About 1100 expert-constructed NLI examples (from naturally occurring sentences, minimally edited) tagged with linguistic phenomena in four coarse categories: lexical semantics, predicate–argument structure, logic, and knowledge and common sense. Each pair is labelled in both directions.

With \(R_3\), a three-class generalization of the Matthews correlation, since labels are unbalanced (42% entailment, 35% neutral, 23% contradiction). Hypothesis-only fastText classifiers trained on SNLI and MNLI scored near chance on it (32.7% and 36.4%); human annotators reached \(R_3 = 0.80\).

As an analysis tool for error analysis, comparing models and building adversarial examples, not as a benchmark: performance should be compared between models but not between categories, and it doesn’t predict downstream performance.

A two-layer BiLSTM encoder with max pooling over GloVe embeddings, with sentence pairs as \([u; v; |u - v|; u * v]\); variants with cross-sentence attention, ELMo and CoVe; trained per task or multi-task (tasks sampled in proportion to data size). Also pretrained sentence encoders: CBoW, Skip-Thought, InferSent, DisSent, GenSen.

Multi-task training beat single-task training for models with attention or ELMo; the best was multi-task BiLSTM + attention + ELMo with a 70.0 average. ELMo helped consistently, especially on single-sentence tasks; no model beat the majority class on WNLI (65.1%).

Gains rose from CBoW to Skip-Thought, InferSent and GenSen, and GenSen beat all but the two best GLUE-trained models. Sentence encoders did badly on CoLA but best on STS-B.

Overall performance was low (best \(R_3\) of 28): models did reasonably on universal quantification and morphological negation, where lexical cues suffice, but struggled with double negation (without ELMo), restrictivity, and downward monotonicity.

They treat hypernym/hyponym substitution and word deletion as signals of entailment in the upward direction even in downward-monotone contexts (e.g. under negation), consistent with relying on the premise–hypothesis subsequence heuristic.

Joint training and transfer (attention plus ELMo) helped, but absolute performance remained low and models failed, often spectacularly, on many linguistic phenomena, so how to design general-purpose NLU models was still open.