Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data

2 Large LMs: Hype and analysis

1. What problem with how NLP papers talk about large LMs did Bender and Koller point out?
   Terms like “understands sentence relationships”, “machine comprehension” or “recall factual knowledge” are gross overclaims if read as human-analogous abilities, and otherwise should be defined as technical terms; imprudent terminology feeds AI hype in the popular press.
   • Bender & Koller, 2020, Climbing towards NLU (Proceedings of the 58th Annual Meeting of ACL), §2, pp. 5185–5186
2. What evidence suggested that BERT’s “reasoning” is sometimes a mirage?
   Niven and Kao (2019): BERT’s strong argument reasoning results fall to chance once adversarial examples negate cues so label-specific lexical cues are balanced. McCoy et al. (2019): BERT’s MNLI success relies on overlap heuristics, and on data built to defeat them it falls well below chance.
   • Bender & Koller, 2020, Climbing towards NLU (Proceedings of the 58th Annual Meeting of ACL), §2, pp. 5185–5186