Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data

8 On climbing the right hills

1. What is the “right hill” worry in Bender and Koller?
   Bottom-up, NLP is climbing fast with steady state-of-the-art gains; top-down, the question is whether this hill leads to human-analogous NLU. Past paradigms (grammar-based, statistical) also felt like constant progress before their shortcomings showed.
   • Bender & Koller, 2020, Climbing towards NLU (Proceedings of the 58th Annual Meeting of ACL), §8, pp. 5191–5192
2. Which best practices did Bender and Koller recommend for “less error-prone mountaineering”?
   Cultivate humility towards language and ask top-down questions; be aware of task limitations (artificial or crowdsourced data sets; superhuman scores suggest artifacts); value careful creation of new tasks; evaluate meaning across tasks; and thoroughly analyse both errors and successes.
   • Bender & Koller, 2020, Climbing towards NLU (Proceedings of the 58th Annual Meeting of ACL), §8, pp. 5191–5192
3. Do Bender and Koller’s arguments apply to fine-tuning BERT on reading comprehension or NLI?
   Not directly: such data sets specify semantic relations between texts, a training signal beyond pure form, so a model might learn some aspects of meaning from them, though McCoy et al. and Niven and Kao warn that gains often come from data idiosyncrasies.
   • Bender & Koller, 2020, Climbing towards NLU (Proceedings of the 58th Annual Meeting of ACL), §8, pp. 5191–5192