learn.ifs.name

Logistic Regression and Text Classification

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4

6 cards in “4.14 Regularization” · whole deck

4.14 Regularization

  1. What is overfitting in logistic regression?
    Weights fit the training data too perfectly, e.g. huge weights for features that happen to occur in only one class, modeling noise that accidentally correlates with the class, so the model generalizes poorly.
  2. How does regularization prevent overfitting?
    It adds a penalty term \(R(\theta)\) for large weights: \(\hat{\theta} = \arg\max_\theta \sum_i \log P(y^{(i)} \mid x^{(i)}) - \alpha R(\theta)\). A solution using smaller weights is preferred over one that fits slightly better with large weights; \(\alpha\) sets the strength.
  3. Is the bias term regularized?
    Normally not: it acts as a threshold that handles uncentered data and class priors.
  4. What are L2 and L1 regularization?
    L2 (ridge): \(R(\theta) = \|\theta\|_2^2 = \sum_j \theta_j^2\), the squared Euclidean norm. L1 (lasso): \(R(\theta) = \|\theta\|_1 = \sum_j |\theta_j|\), the Manhattan distance.
  5. How do the solutions preferred by L1 and L2 regularization differ, and which is easier to optimize?
    L2 prefers many small weights and is easier to optimize (derivative \(2\theta\)). L1 prefers sparse solutions with many weights exactly zero (fewer features) but its derivative is discontinuous at 0.
  6. What are the Bayesian interpretations of L1 and L2 regularization?
    L1 corresponds to a Laplace prior on the weights; L2 to a Gaussian prior with mean 0, i.e. a preference for weights near 0.

Source