SQuAD: 100,000+ Questions for Machine Comprehension of Text

High-quality datasets (MCTest, ProcessBank) were too small for data-intensive models, while large ones (CNN/Daily Mail, Children’s Book Test) were semi-synthetic cloze tasks that don’t share the characteristics of real reading comprehension questions.

The Stanford Question Answering Dataset: 107,785 crowdsourced questions on 536 Wikipedia articles, where every answer is a span of the corresponding paragraph, almost two orders of magnitude larger than earlier manually labelled RC datasets.

Systems must choose among all spans of the passage, not a short list of answer choices, and spans are much easier to evaluate than free-form answers. But span questions are more constrained than the interpretive questions of standardized tests.

Cloze answers are single words or entities merely suggested by the passage, and the data are generated automatically. SQuAD answers are often non-entity, longer phrases entailed by the passage, written by humans as questions.

Answer extraction usually exploits redundancy, since an answer appears in many documents; in SQuAD a system has only a single reading passage.

From the top 10,000 English Wikipedia articles by PageRank, 536 were sampled at random; paragraphs over 500 characters were kept (23,215 paragraphs), and articles were split 80/10/10 into train, development and test.

Crowdworkers asked and answered up to 5 questions per paragraph, highlighting the answer span, and were encouraged to use their own words: a reminder appeared for every paragraph, and copy-paste from the paragraph was disabled.

To measure human performance and make evaluation robust to the choice of span: at least two more workers selected the shortest answering span for each question. 2.6% of questions were marked unanswerable by at least one of them.

Dates and other numbers 19.8%, proper nouns (person, location, other entity) 32.6%, common noun phrases 31.8%, and adjective phrases, verb phrases, clauses and others 15.8%: many answers go beyond named entities.

All involve some lexical or syntactic divergence between question and answer sentence: syntactic variation 64.1%, synonymy 33.3%, multiple-sentence reasoning (e.g. anaphora) 13.6%, world knowledge 9.1%; 6.1% were ambiguous.

Find anchors (word–lemma pairs shared by both); take the unlexicalized dependency path from the anchor to the wh-word in the question and from the anchor to the answer span in the sentence; the divergence is the minimum edit distance (insertions and deletions) between these paths over all anchors.

Only constituents of the sentence’s constituency parse, not all \(O(L^2)\) spans. Ignoring punctuation and articles, 77.3% of correct answers are constituents, an effective ceiling for these methods.

Matching word and bigram TF-IDF sums (left of, right of and inside the span, and whole sentence), dependency root matches, lengths, span word frequencies, constituent label and span POS tags (optionally with the wh-word), lexicalized question–sentence lemma pairs, and dependency tree paths: about 180 million features.

Exact match: the prediction equals any gold answer, ignoring punctuation and articles. F1: the bag-of-tokens overlap F1 between prediction and gold answer, maximized over gold answers and averaged over questions.

Exact match 0. After dropping articles: prediction {monsoon, trough}, gold {movement, of, monsoon, trough}; overlap 2, so precision 2/2 = 1, recall 2/4 = 0.5, \(F_1 = \frac{2 \cdot 1 \cdot 0.5}{1 + 0.5} \approx 0.67\).

Treat the second answer to each question as the human prediction and the others as gold: 77.0% exact match and 86.8% F1 on test. Mismatches were mostly about including or excluding non-essential phrases, not about the answer itself.

On test F1: random guess 4.3%, sliding window 19.7–20.0%, logistic regression 51.0%, humans 86.8%. The model found the right sentence 79.3% of the time, so most of the difficulty lies in picking the exact span.

Lexicalized and dependency tree path features: removing both cut dev F1 from 51.0% to 35.8%; removing lexicalized features alone gave 45.4%, dependency paths alone 46.4%. Other feature groups mattered little.

The model did best on dates and numbers (few plausible candidates, single tokens), worse on entities and worst on other types (47.6% of the data), and degraded steadily as syntactic divergence grew. Human performance was nearly uniform across types and insensitive to divergence.

Within about four months, Wang and Jiang’s Match-LSTM with an answer pointer reached 70.3% F1, more than halving the gap between the 51.0% logistic regression baseline and 86.8% human performance.