GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

3.3 Inference Tasks

1. How was QNLI built from SQuAD?
   Each question is paired with each sentence of its context paragraph (dropping low-overlap pairs), and the task is whether the sentence contains the answer. This drops the need to select the exact span, but also the assumptions that the answer is present and that lexical overlap is a reliable cue.
   • Wang et al., 2019, GLUE (7th International Conference on Learning Representations (ICLR 2019)), §3.3
2. What is WNLI, and what are its quirks?
   The Winograd Schema Challenge recast as NLI: replace the ambiguous pronoun with each candidate referent and ask whether the result is entailed. The test set is imbalanced (65% not entailment) and the development set is adversarial (hypotheses shared with training examples get opposite labels), so memorization hurts.
   • Wang et al., 2019, GLUE (7th International Conference on Learning Representations (ICLR 2019)), §3.3
3. How is RTE built in GLUE?
   By combining RTE1, RTE2, RTE3 and RTE5 (news and Wikipedia text) and collapsing neutral and contradiction into “not entailment” for a two-class task; it has only about 2.5k training examples.
   • Wang et al., 2019, GLUE (7th International Conference on Learning Representations (ICLR 2019)), §3.3