Annotation Artifacts in Natural Language Inference Data

2 Annotation Artifacts are Common

1. What are annotation artifacts?
   Patterns in a dataset produced by the framing of the annotation task and the strategies annotators adopt, here in how crowdworkers phrase hypotheses for each label, which correlate with labels independently of the actual inference.
   • Gururangan et al., 2018, Annotation Artifacts in Natural Language Inference Data (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies, Volume 2 (Short Papers)), §2, pp. 107–108
2. How accurate is a hypothesis-only classifier on SNLI and MultiNLI?
   A bag-of-words-and-bigrams fastText classifier that never sees the premise got 67.0% on SNLI (majority class 34.3%) and about 53% on MultiNLI (53.9% matched, 52.3% mismatched; majority about 35%). Other premise-oblivious classifiers gave similar results.
   • Gururangan et al., 2018, Annotation Artifacts in Natural Language Inference Data (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies, Volume 2 (Short Papers)), §2, pp. 107–108