SQuAD: 100,000+ Questions for Machine Comprehension of Text

1 Introduction

1. What shortcomings of existing reading comprehension datasets did SQuAD address?
   High-quality datasets (MCTest, ProcessBank) were too small for data-intensive models, while large ones (CNN/Daily Mail, Children’s Book Test) were semi-synthetic cloze tasks that don’t share the characteristics of real reading comprehension questions.
   • Rajpurkar et al., 2016, SQuAD (Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §1, pp. 2383–2384
2. What is SQuAD v1.0?
   The Stanford Question Answering Dataset: 107,785 crowdsourced questions on 536 Wikipedia articles, where every answer is a span of the corresponding paragraph, almost two orders of magnitude larger than earlier manually labelled RC datasets.
   • Rajpurkar et al., 2016, SQuAD (Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §1, pp. 2383–2384
3. What are the advantages and limits of span-based answers in SQuAD?
   Systems must choose among all spans of the passage, not a short list of answer choices, and spans are much easier to evaluate than free-form answers. But span questions are more constrained than the interpretive questions of standardized tests.
   • Rajpurkar et al., 2016, SQuAD (Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §1, pp. 2383–2384