A large annotated corpus for learning natural language inference

Deciding whether a hypothesis sentence is entailed by, contradicts, or is neutral to (independent of) a premise. Entailment and contradiction are central to meaning, from the lexicon to whole texts, so NLI is a testing ground for representations that must support logical and commonsense inference.

They were too small for data-hungry models (RTE sets under a thousand examples each, SICK 4,500 training pairs), many sentences were algorithmically generated (introducing spurious patterns), and labels suffered from indeterminacies of event and entity coreference.

The Stanford Natural Language Inference corpus: 570,152 sentence pairs labeled entailment, contradiction or neutral, two orders of magnitude larger than other resources of its kind, with all sentences and labels written by humans in a grounded, naturalistic task.

“A boat sank in the Pacific Ocean” / “A boat sank in the Atlantic Ocean” is a contradiction if both describe the same event, neutral otherwise. Assuming coreference everywhere makes unrelated pairs contradictions; never assuming it leaves contradictions only for universal claims.

Both sentences describe the same grounded scenario (a photo) from the same perspective: workers saw an image caption (not the photo) and wrote new captions about that same scene, so entities and events are naturally taken to corefer.

On Mechanical Turk, workers saw a Flickr30k image caption as the premise and wrote three alternative captions: one definitely true (entailment), one that might be true (neutral), and one definitely false (contradiction), balancing the classes. About 2,500 workers contributed.

invented example — premise: “A woman is riding a bicycle down a hill.”
entailment	A person is outdoors.
neutral	The woman is late for work.
contradiction	The woman is sitting on a sofa.

Flickr30k captions are literal scene descriptions written by crowdworkers, well suited to the task. Not using the images ensures that each label can be recovered from the text alone.

Premises vary a lot (mean 14.1 tokens); hypotheses tend to be as short as possible while still allowing a clear judgment (mean 8.3 tokens, clustering around seven words). Most sentences of both kinds are syntactically complete rather than fragments.

About 10% of pairs (56,951) got four more labels from trusted workers. A gold label needs at least 3 of 5 votes; 98% of pairs got one and 58% were unanimous. Individual labels matched gold 89.0% of the time; Fleiss κ was 0.70 overall (contradiction 0.77, entailment 0.72, neutral 0.60).

Pairs where no label got three of five votes (about 2% of validated pairs) get the placeholder label “-”; they are distributed but excluded from training and evaluation in the paper.

Prespecified train/development/test splits with 10k validated examples each in development and test; each Flickr30k caption appears in only one split. It includes Stanford PCFG parses and is licensed CC BY-SA, like Flickr30k.

On 2-class entailment (majority baseline 66% on SNLI): an edit-distance system 71.9%, a classifier 72.2%, and the classifier with WordNet, VerbOcean and tree features 75.0%, showing the task is hard for standard systems.

Unlexicalized: BLEU of hypothesis vs premise, length difference, and word overlap counts. Lexicalized: hypothesis unigrams and bigrams, cross-unigrams (pairs of words across premise and hypothesis sharing a POS tag), and cross-bigrams (pairs of bigrams whose second words share a POS tag).

With enough data, a simple lexicalized model with no alignment or tree transformations is strong: 78.2% test accuracy (3-class), vs 71.6% with unigrams only and 50.4% unlexicalized. It nearly memorized training data (99.7%). Cross-bigrams help learn structures like negation and adjective modification.

Each sentence is encoded separately into a 100-d vector (no cross-sentence context); the two vectors are concatenated and fed to three 200-d tanh layers and a 3-way softmax, trained jointly. Word embeddings start from 300-d GloVe and are fine-tuned.

Test accuracy: sum of word vectors 75.3%, plain RNN 72.2%, LSTM 77.6%, essentially equal to the lexicalized classifier (78.2%), with a much smaller train–test gap, suggesting higher-capacity neural models would pay off.

The corpus size is crucial for both the LSTM and the lexicalized model, and more data would likely help both; the LSTM’s steeper slope hints it may overtake the constrained lexicalized model on still larger data. The lexicalized model beat the unlexicalized one with only 100 examples.

Misanalysed lexical relations even for frequent pairs (beach/surf, sprinter/runner), phrasal semantics, world knowledge (“driver leaps from a burning car” → “escaping danger”), and shortcuts through lexical cues and expected parallel structure: all models predicted entailment for “A man bit a dog” from a premise where a dog bites a man.

An LSTM trained on SNLI and then fine-tuned on SICK (new softmax and embeddings for unseen words, AdaDelta accumulators transferred to lower learning rates) reached 80.8%, the best reported for an unaugmented neural model, vs 71.3% trained on SICK alone and 46.7% for SNLI without fine-tuning; the state of the art was 84.6%.

It made it possible to train representation-learning models for sentence meaning that compete with hand-engineered NLI systems: a neural model was competitive on NLI benchmarks for the first time, and SNLI became a standard training set and testbed.