Logistic Regression and Text Classification

4.9 Evaluation: precision, recall, F-measure

1. What are gold labels?
   The human-defined correct labels that a system’s outputs are compared against.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.9, p. 112
2. What is a confusion matrix?
   A table comparing system output with gold labels, whose cells count true positives, false positives, false negatives and true negatives (for binary tasks).

   	gold +	gold −
   system +	true positive	false positive
   system −	false negative	true negative

   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.9, p. 113
3. Why is accuracy a poor metric for text classification with unbalanced classes?
   A classifier that labels a million tweets as “not about pie” when only 100 are about pie gets 99.99% accuracy while finding nothing. Accuracy rewards ignoring rare classes.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.9, p. 113
4. Define precision and recall.
   Precision = tp / (tp + fp): the fraction of items the system labeled positive that really are positive. Recall = tp / (tp + fn): the fraction of truly positive items the system found.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.9, p. 113
5. What is the recall of the “nothing is about pie” classifier?
   0 (0 true positives out of 100), exposing it as useless despite its accuracy.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.9, p. 114
6. Define the F-measure and F1.
   \(F_\beta = \frac{(\beta^2 + 1)PR}{\beta^2 P + R}\); \(\beta > 1\) favors recall, \(\beta < 1\) precision. With \(\beta = 1\): \(F_1 = \frac{2PR}{P + R}\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.9, p. 114
7. Why does F-measure use a harmonic mean of precision and recall?
   The harmonic mean is closer to the smaller of the two values than the arithmetic mean, so it weighs the weaker one more heavily: a conservative combination.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.9, p. 114
8. How are precision and recall computed for each class in a multiclass confusion matrix?
   For class \(c\): precision = correct \(c\) / all items the system labeled \(c\) (its row); recall = correct \(c\) / all items whose gold label is \(c\) (its column).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.9, pp. 114–115
9. What is the difference between macroaveraging and microaveraging?
   Macroaveraging computes the metric per class and averages over classes. Microaveraging pools all decisions into one confusion matrix and computes the metric from it.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.9, p. 115
10. When is macroaveraging more appropriate than microaveraging?
    When performance on all classes matters equally: the microaverage is dominated by the most frequent class, while the macroaverage reflects small classes.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.9, p. 115
11. Why is microaveraged precision always equal to microaveraged recall?
    In the pooled matrix every false positive for one class is a false negative for another, so the total fp equals the total fn.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.9, p. 115