SQuAD: 100,000+ Questions for Machine Comprehension of Text

6.3 Model Performance

1. How did the SQuAD baselines perform?
   On test F1: random guess 4.3%, sliding window 19.7–20.0%, logistic regression 51.0%, humans 86.8%. The model found the right sentence 79.3% of the time, so most of the difficulty lies in picking the exact span.
   • Rajpurkar et al., 2016, SQuAD (Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §6.3, pp. 2389–2391
2. Which features mattered most for the SQuAD logistic regression model?
   Lexicalized and dependency tree path features: removing both cut dev F1 from 51.0% to 35.8%; removing lexicalized features alone gave 45.4%, dependency paths alone 46.4%. Other feature groups mattered little.
   • Rajpurkar et al., 2016, SQuAD (Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §6.3, pp. 2389–2391
3. How did model and human performance vary with answer type and syntactic divergence?
   The model did best on dates and numbers (few plausible candidates, single tokens), worse on entities and worst on other types (47.6% of the data), and degraded steadily as syntactic divergence grew. Human performance was nearly uniform across types and insensitive to divergence.
   • Rajpurkar et al., 2016, SQuAD (Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §6.3, pp. 2389–2391