learn.ifs.name

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Wang et al., 2019, GLUE (7th International Conference on Learning Representations (ICLR 2019))

Subscriber deck — free during early access

2 cards in “7 Analysis” · whole deck

7 Analysis

  1. What did the GLUE diagnostic set reveal about the baselines?
    Overall performance was low (best \(R_3\) of 28): models did reasonably on universal quantification and morphological negation, where lexical cues suffice, but struggled with double negation (without ELMo), restrictivity, and downward monotonicity.
    • Wang et al., 2019, GLUE (7th International Conference on Learning Representations (ICLR 2019)), §7
  2. Why did attention models fail on downward-monotone diagnostic examples?
    They treat hypernym/hyponym substitution and word deletion as signals of entailment in the upward direction even in downward-monotone contexts (e.g. under negation), consistent with relying on the premise–hypothesis subsequence heuristic.
    • Wang et al., 2019, GLUE (7th International Conference on Learning Representations (ICLR 2019)), §7

Source

  • Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy and Samuel R. Bowman. 2019. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In 7th International Conference on Learning Representations (ICLR 2019). https://openreview.net/forum?id=rJ4km2R5t7