SQuAD: 100,000+ Questions for Machine Comprehension of Text

6.1 Model Evaluation

1. How are SQuAD answers scored?
   Exact match: the prediction equals any gold answer, ignoring punctuation and articles. F1: the bag-of-tokens overlap F1 between prediction and gold answer, maximized over gold answers and averaged over questions.
   • Rajpurkar et al., 2016, SQuAD (Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §6.1, p. 2389
2. Gold answer “movement of the monsoon trough”, prediction “the monsoon trough”: compute exact match and F1.
   Exact match 0. After dropping articles: prediction {monsoon, trough}, gold {movement, of, monsoon, trough}; overlap 2, so precision 2/2 = 1, recall 2/4 = 0.5, \(F_1 = \frac{2 \cdot 1 \cdot 0.5}{1 + 0.5} \approx 0.67\).
   • Rajpurkar et al., 2016, SQuAD (Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP)), §6.1, p. 2389