SQuAD: 100,000+ Questions for Machine Comprehension of Text

3 Dataset Collection

1. How were SQuAD passages chosen?
   From the top 10,000 English Wikipedia articles by PageRank, 536 were sampled at random; paragraphs over 500 characters were kept (23,215 paragraphs), and articles were split 80/10/10 into train, development and test.
   • Rajpurkar et al., 2016, SQuAD (Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §3, pp. 2385–2386
2. How did SQuAD reduce lexical copying between questions and passages?
   Crowdworkers asked and answered up to 5 questions per paragraph, highlighting the answer span, and were encouraged to use their own words: a reminder appeared for every paragraph, and copy-paste from the paragraph was disabled.
   • Rajpurkar et al., 2016, SQuAD (Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §3, pp. 2385–2386
3. Why did SQuAD collect additional answers for development and test questions?
   To measure human performance and make evaluation robust to the choice of span: at least two more workers selected the shortest answering span for each question. 2.6% of questions were marked unanswerable by at least one of them.
   • Rajpurkar et al., 2016, SQuAD (Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §3, pp. 2385–2386