A large annotated corpus for learning natural language inference

2.1 Data collection

1. How were SNLI hypotheses collected?
   On Mechanical Turk, workers saw a Flickr30k image caption as the premise and wrote three alternative captions: one definitely true (entailment), one that might be true (neutral), and one definitely false (contradiction), balancing the classes. About 2,500 workers contributed.

   invented example — premise: “A woman is riding a bicycle down a hill.”
   entailment	A person is outdoors.
   neutral	The woman is late for work.
   contradiction	The woman is sitting on a sofa.

   • Bowman et al., 2015, A large annotated corpus for learning natural language inference (Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §2.1, pp. 634–635
2. Why did SNLI use image captions as premises but never show the images?
   Flickr30k captions are literal scene descriptions written by crowdworkers, well suited to the task. Not using the images ensures that each label can be recovered from the text alone.
   • Bowman et al., 2015, A large annotated corpus for learning natural language inference (Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §2.1, pp. 634–635
3. How do SNLI premises and hypotheses differ in length?
   Premises vary a lot (mean 14.1 tokens); hypotheses tend to be as short as possible while still allowing a clear judgment (mean 8.3 tokens, clustering around seven words). Most sentences of both kinds are syntactically complete rather than fragments.
   • Bowman et al., 2015, A large annotated corpus for learning natural language inference (Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §2.1, pp. 634–635