Annotation Artifacts in Natural Language Inference Data

3.1 Lexical Choice

1. How did Gururangan et al. find words indicative of each NLI class?
   With pointwise mutual information between word and class over training hypotheses, \(\text{PMI}(\text{word}, \text{class}) = \log \frac{p(\text{word}, \text{class})}{p(\text{word}, \cdot)\, p(\cdot, \text{class})}\), with aggressive add-100 smoothing to emphasize highly discriminative correlations.
   • Gururangan et al., 2018, Annotation Artifacts in Natural Language Inference Data (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies, Volume 2 (Short Papers)), §3.1, pp. 108–109
2. Which lexical artifacts mark entailed hypotheses?
   Generic words generalizing over specific premise words (animal for dog, instrument for guitar, outdoors/outside for beach), approximate quantities replacing exact numbers (some, at least, various), and gender-neutral references (person, human).
   • Gururangan et al., 2018, Annotation Artifacts in Natural Language Inference Data (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies, Volume 2 (Short Papers)), §3.1, pp. 108–109
3. Which lexical artifacts mark neutral hypotheses?
   Modifiers (tall, sad, popular) and superlatives (first, favorite, most) that add plausible but unsupported information, and cause or purpose clauses, raising the frequency of discourse markers like because.
   • Gururangan et al., 2018, Annotation Artifacts in Natural Language Inference Data (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies, Volume 2 (Short Papers)), §3.1, pp. 108–109
4. Which lexical artifacts mark contradictions?
   Negation words (nobody, no, never, nothing) and words that contradict almost any premise, such as sleeping (contradicts any activity), naked (any clothing description) and cat (many SNLI images show dogs).
   • Gururangan et al., 2018, Annotation Artifacts in Natural Language Inference Data (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies, Volume 2 (Short Papers)), §3.1, pp. 108–109
5. How might the SNLI annotation guidelines have caused some artifacts?
   The guideline example (“Two dogs are running through a field.”) had the entailment “There are animals outdoors.” (generic word, scenery word, number removed), the neutral “Some puppies are running to catch a stick.” (purpose clause), and a contradiction about pets on a couch, likely priming workers.
   • Gururangan et al., 2018, Annotation Artifacts in Natural Language Inference Data (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies, Volume 2 (Short Papers)), §3.1, pp. 108–109
6. Are the artifact words very frequent?
   No: even the most discriminative words appear in only a small share of hypotheses (e.g. nobody in 0.1% of contradictions, no in a few percent), so the artifacts are diverse, and workers used many different heuristics.
   • Gururangan et al., 2018, Annotation Artifacts in Natural Language Inference Data (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies, Volume 2 (Short Papers)), §3.1, pp. 108–109