Annotation Artifacts in Natural Language Inference Data
Gururangan et al., 2018, Annotation Artifacts in Natural Language Inference Data (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies, Volume 2 (Short Papers))
Subscriber deck — free during early access
2 cards in “1 Introduction” · whole deck
1 Introduction
- What is the main finding of “Annotation Artifacts in Natural Language Inference Data”?The crowdsourcing protocol of SNLI and MultiNLI leaves clues in the hypotheses, so the label can often be predicted from the hypothesis alone, without the premise; NLI models’ success had been overestimated and the task remains hard.
- Gururangan et al., 2018, Annotation Artifacts in Natural Language Inference Data (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies, Volume 2 (Short Papers)), §1, p. 107
- How were SNLI and MultiNLI hypotheses created, and why does that matter here?Crowdworkers saw a premise and wrote three hypotheses: one definitely true (entailment), one that might be true (neutral), one definitely not true (contradiction). Writing to a label invites quick, repeatable heuristics that leave label-specific traces in the hypothesis text.
- Gururangan et al., 2018, Annotation Artifacts in Natural Language Inference Data (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies, Volume 2 (Short Papers)), §1, p. 107
Source
- Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman and Noah A. Smith. 2018. Annotation Artifacts in Natural Language Inference Data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 107–112. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/N18-2017 https://aclanthology.org/N18-2017/