Information Retrieval and Retrieval-Augmented Generation

11.5 Datasets

1. Along which dimensions do question-answering datasets vary?
   The questions’ origin (natural information-seeking vs probing/testing), answer format (multiple choice vs free form), whether passages are provided (open vs closed book), and prompting setup (zero-shot vs few-shot).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.5, pp. 269–270
2. Name natural information-seeking QA datasets.
   Natural Questions (Google queries with long paragraph and short span answers from Wikipedia), MS MARCO (Bing queries with human answers and passages, for retrieval ranking and QA), DuReader (Chinese), TyDi QA (typologically diverse languages).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.5, p. 270
3. What is the TyDi QA task?
   Given a question and a Wikipedia article’s passages in one of several typologically diverse languages, (a) select the passage containing the answer (or NULL) and (b) mark the minimal answer span (or NULL).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.5, p. 270
4. Why is MMLU a probing rather than an information-seeking dataset?
   Its questions come from exams for humans (GRE, medical licensing, AP…), designed to test knowledge, not to reflect people’s information needs.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.5, p. 270
5. What is the difference between reading comprehension, open-book and closed-book QA?
   Reading comprehension: extract the answer from a given document. Open-book QA: answer given one or more documents (e.g. via RAG). Closed-book QA: answer directly from the LM with no retrieval.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.5, p. 270