A large annotated corpus for learning natural language inference
Bowman et al., 2015, A large annotated corpus for learning natural language inference (Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP))
Subscriber deck — free during early access
2 cards in “2.2 Data validation” · whole deck
2.2 Data validation
- How was SNLI validated, and how reliable are the labels?About 10% of pairs (56,951) got four more labels from trusted workers. A gold label needs at least 3 of 5 votes; 98% of pairs got one and 58% were unanimous. Individual labels matched gold 89.0% of the time; Fleiss κ was 0.70 overall (contradiction 0.77, entailment 0.72, neutral 0.60).
- Bowman et al., 2015, A large annotated corpus for learning natural language inference (Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §2.2, pp. 635–636
- What happens to SNLI pairs without a gold label?Pairs where no label got three of five votes (about 2% of validated pairs) get the placeholder label “-”; they are distributed but excluded from training and evaluation in the paper.
- Bowman et al., 2015, A large annotated corpus for learning natural language inference (Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §2.2, pp. 635–636
Source
- Samuel R. Bowman, Gabor Angeli, Christopher Potts and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 632–642. doi:10.18653/v1/D15-1075 https://aclanthology.org/D15-1075/