SQuAD: 100,000+ Questions for Machine Comprehension of Text

4 Dataset Analysis

1. What answer types does SQuAD contain?
   Dates and other numbers 19.8%, proper nouns (person, location, other entity) 32.6%, common noun phrases 31.8%, and adjective phrases, verb phrases, clauses and others 15.8%: many answers go beyond named entities.
   • Rajpurkar et al., 2016, SQuAD (Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §4, pp. 2386–2387
2. What reasoning do SQuAD questions require, according to a manual analysis of 192 examples?
   All involve some lexical or syntactic divergence between question and answer sentence: syntactic variation 64.1%, synonymy 33.3%, multiple-sentence reasoning (e.g. anaphora) 13.6%, world knowledge 9.1%; 6.1% were ambiguous.
   • Rajpurkar et al., 2016, SQuAD (Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §4, pp. 2386–2387
3. How did Rajpurkar et al. measure syntactic divergence between a question and its answer sentence?
   Find anchors (word–lemma pairs shared by both); take the unlexicalized dependency path from the anchor to the wh-word in the question and from the anchor to the answer span in the sentence; the divergence is the minimum edit distance (insertions and deletions) between these paths over all anchors.
   • Rajpurkar et al., 2016, SQuAD (Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §4, pp. 2386–2387