Annotation Artifacts in Natural Language Inference Data
Gururangan et al., 2018, Annotation Artifacts in Natural Language Inference Data (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies, Volume 2 (Short Papers))
Subscriber deck — free during early access
4 cards in “4 Re-evaluating NLI Models” · whole deck
4 Re-evaluating NLI Models
- How did Gururangan et al. split NLI test sets to test models?Into Easy examples, which the hypothesis-only classifier labelled correctly, and Hard examples, which it got wrong, then evaluated models trained on the original training sets on full, Hard and Easy subsets.
- Gururangan et al., 2018, Annotation Artifacts in Natural Language Inference Data (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies, Volume 2 (Short Papers)), §4, p. 110
- How did strong NLI models perform on Easy versus Hard examples?Much worse on Hard: on SNLI, DIIN scored 93.4% on Easy but 72.7% on Hard (86.5% overall); on MultiNLI matched 87.6% vs 64.1%. Similar gaps for DAM and ESIM. The bulk of their success came from examples solvable by artifacts.
SNLI full Hard Easy DAM 84.7 69.4 92.4 ESIM 85.8 71.3 92.6 DIIN 86.5 72.7 93.4 - Gururangan et al., 2018, Annotation Artifacts in Natural Language Inference Data (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies, Volume 2 (Short Papers)), §4, p. 110
- Why isn’t simply removing Easy examples a complete fix for annotation artifacts?Hard examples may carry new artifacts (removing contradictions with no leaves no mostly in other classes), and Easy examples contain genuine inference phenomena (animal really is a hypernym of dog). Artifacts don’t make examples wrong; the label distribution is skewed, so balancing artifacts across labels may be better.
- Gururangan et al., 2018, Annotation Artifacts in Natural Language Inference Data (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies, Volume 2 (Short Papers)), §4, p. 110
- What did Gururangan et al. suggest for collecting future NLI data?Change prompts or worker training to encourage a wide range of strategies, or use baseline or adversarial systems during collection to flag examples that rely on over-represented heuristics.
- Gururangan et al., 2018, Annotation Artifacts in Natural Language Inference Data (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies, Volume 2 (Short Papers)), §4, p. 110
Source
- Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman and Noah A. Smith. 2018. Annotation Artifacts in Natural Language Inference Data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 107–112. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/N18-2017 https://aclanthology.org/N18-2017/