Information Retrieval and Retrieval-Augmented Generation

11.2 Evaluating IR systems

1. Define precision and recall for a set of retrieved documents.
   Precision = relevant returned / all returned (\(|R|/|T|\)). Recall = relevant returned / all relevant documents in the collection (\(|R|/|U|\)).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.2, p. 261
2. Why aren’t plain precision and recall enough for ranked retrieval?
   They ignore order: we want metrics that prefer systems ranking relevant documents higher.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.2, p. 261
3. Define precision and recall at rank \(k\).
   Precision\(_k\) = relevant documents in top \(k\) / \(k\); Recall\(_k\) = relevant documents in top \(k\) / total relevant documents.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.2, p. 261
4. How do precision and recall behave as you go down a ranked list?
   Recall never decreases (it rises at each relevant document); precision jumps up at relevant documents and down otherwise. They are plotted as a precision-recall curve.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.2, p. 262
5. What is interpolated precision, and why use it?
   Precision at recall level \(r\) is set to the maximum precision at any recall ≥ \(r\): \(\text{IntPrecision}(r) = \max_{i \ge r}\text{Precision}(i)\). It allows averaging over queries at 11 standard recall levels (0, .1, …, 1) and smooths the curve, giving systems the benefit of the doubt.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.2, p. 263
6. What does the shape of a precision-recall curve tell you about a system?
   Higher precision at all recall levels is better; systems high on the left favor precision, those high on the right favor recall.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.2, p. 263
7. What is average precision (AP), and mean average precision (MAP)?
   AP: average of the precision values measured at the ranks where relevant documents occur, \(\frac{1}{|R_r|}\sum_{d \in R_r}\text{Precision}_r(d)\). MAP: the mean of AP over a set of queries.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.2, p. 263
8. Compute AP if relevant documents appear at ranks 1, 3 and 4 (and those are all the relevant ones).
   Precisions at those ranks: 1/1, 2/3, 3/4. AP = (1 + .667 + .75)/3 ≈ .81.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 11, §11.2, p. 263