Annotation Artifacts in Natural Language Inference Data

5 Discussion

1. Which other datasets were found to have similar artifacts?
   SICK (negation, overlap and hypernymy predict labels), CNN/Daily Mail (problems from automatic annotation), the ROC Stories cloze task (endings alone suffice), relation inference benchmarks biased toward the method used to build them, and other NLI sets (Poliak et al. 2018, hypothesis-only baselines).
   • Gururangan et al., 2018, Annotation Artifacts in Natural Language Inference Data (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies, Volume 2 (Short Papers)), §5, pp. 110–111
2. What practical lesson does this paper teach about evaluating on crowdsourced benchmarks?
   Check for shortcuts with partial-input baselines (e.g. hypothesis-only); a model scoring well may be exploiting annotation artifacts rather than solving the task, so report results on hard subsets and build more challenging benchmarks.
   • Gururangan et al., 2018, Annotation Artifacts in Natural Language Inference Data (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies, Volume 2 (Short Papers)), §5, pp. 110–111