learn.ifs.name

Logistic Regression and Text Classification

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4

11 cards in “4.11 Statistical significance testing” · whole deck

4.11 Statistical significance testing

  1. What is the effect size \(\delta(x)\) when comparing systems A and B?
    The performance difference on test set \(x\): \(\delta(x) = M(A, x) - M(B, x)\) for a metric \(M\) such as F1 or accuracy.
  2. Why isn’t \(\delta(x) > 0\) enough to conclude that A is better than B?
    A might be better just by accident on this particular test set; we want to know whether the advantage would hold on other test sets.
  3. What are the null and alternative hypotheses when testing whether A beats B?
    \(H_0: \delta(x) \le 0\) (A is not better than B) and \(H_1: \delta(x) > 0\). We try to reject \(H_0\).
  4. Define the p-value.
    The probability, assuming the null hypothesis is true, of seeing a \(\delta\) as large as the one observed or larger: \(P(\delta(X) \ge \delta(x) \mid H_0)\).
  5. When is a result statistically significant?
    When the p-value is below a chosen threshold (commonly .05 or .01), so we reject the null hypothesis.
  6. Why does NLP usually use non-parametric tests rather than t-tests or ANOVAs?
    Parametric tests assume distributions (e.g. normality) of the test statistic that usually don’t hold for NLP metrics. Non-parametric tests build a distribution by sampling many versions of the experiment.
  7. Which two non-parametric tests are common in NLP?
    Approximate randomization and the bootstrap test.
  8. What is a paired test?
    A comparison of two sets of aligned observations, e.g. systems A and B evaluated on the same test items, so each item’s result for A is paired with its result for B.
  9. What is bootstrapping, and what does the bootstrap test assume?
    Repeatedly drawing samples with replacement from an observed set to create many virtual test sets. It assumes only that the sample is representative of the population; it works with any metric (accuracy, F1, BLEU…).
  10. How does the paired bootstrap test compute a p-value?
    Create \(b\) virtual test sets of size \(n\) by sampling items of \(x\) with replacement, and compute \(\delta(x^{(i)})\) for each. Since these samples are biased toward A by \(\delta(x)\), count how often \(\delta(x^{(i)}) \ge 2\delta(x)\): \(p \approx \frac{1}{b}\sum_i \mathbb{1}(\delta(x^{(i)}) \ge 2\delta(x))\).
  11. Why does the bootstrap compare \(\delta(x^{(i)})\) to \(2\delta(x)\) rather than to 0?
    The bootstrap samples are drawn from \(x\), which is already biased by \(\delta(x)\) in favor of A, so their expected \(\delta\) is \(\delta(x)\), not 0. We ask how often they exceed that expectation by \(\delta(x)\) or more.

Source