Information Retrieval and Retrieval-Augmented Generation

11.6 Evaluating question answering

1. How are question-answering systems evaluated?
   Multiple-choice (e.g. MMLU): exact match, the % of predictions matching the gold answer exactly. Free-text answers (e.g. Natural Questions): token F1, treating prediction and gold as bags of tokens, computing F1 per question and averaging.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.6, p. 271