learn.ifs.name

Logistic Regression and Text Classification

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4

3 cards in “4.10 Test sets and cross-validation” · whole deck

4.10 Test sets and cross-validation

  1. What problem does cross-validation solve?
    With fixed training/dev/test splits, the test or dev set may be too small to be representative. Cross-validation lets all the data be used for both training and testing.
  2. How does \(k\)-fold cross-validation work?
    Partition the data into \(k\) disjoint folds. For each fold, train on the other \(k-1\) folds and measure the error on that fold; average the \(k\) error rates. With \(k = 10\), 10 models are each trained on 90% of the data.
    run 1run 2run 3…test setblue = dev fold, rest = training
    Common practice: cross-validate inside the training data, keep a separate test set.
  3. Why is cross-validation usually done only inside the training set, with a separate test set?
    If all data is used for testing, the whole corpus must stay blind: you couldn’t look at any of it to design features without peeking at test data. A fixed test set lets you inspect the training data.

Source