learn.ifs.name

Logistic Regression and Text Classification

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4

8 cards in “4.2 The sigmoid function” · whole deck

4.2 The sigmoid function

  1. What does a weight \(w_i\) in logistic regression represent?
    How important feature \(x_i\) is to the decision: positive weights give evidence for the positive class, negative weights for the negative class (e.g. awesome high positive, abysmal very negative).
  2. What is the bias term (intercept)?
    A real number added to the weighted sum of the features.
  3. How does logistic regression compute the score \(z\)?
    \(z = \sum_i w_i x_i + b = w \cdot x + b\): the weighted sum of the evidence for the class. It ranges from \(-\infty\) to \(\infty\).
  4. Define the sigmoid (logistic) function.
    \(\sigma(z) = \frac{1}{1 + e^{-z}} = \frac{1}{1 + \exp(-z)}\). It maps any real number into (0, 1).
    z10σ(0) = 0.5
    Nearly linear around 0; squashes large |z| toward 0 or 1.
  5. What properties make the sigmoid suitable for turning a score into a probability?
    It maps reals to (0, 1); it is nearly linear around 0 but flattens at the ends, squashing outliers toward 0 or 1; and it is differentiable, which is handy for learning.
  6. How are \(P(y=1)\) and \(P(y=0)\) computed in binary logistic regression?
    \(P(y=1) = \sigma(w \cdot x + b)\) and \(P(y=0) = 1 - \sigma(w \cdot x + b) = \sigma(-(w \cdot x + b))\), using \(1 - \sigma(x) = \sigma(-x)\).
  7. What is the logit, and why is \(z\) called that?
    The logit function is the inverse of the sigmoid: the log odds, \(\text{logit}(p) = \ln \frac{p}{1-p}\). Calling \(z = w \cdot x + b\) the logit reminds us that the sigmoid interprets it as a log odds.
  8. What does \(\hat{y}\) (“y hat”) denote in binary logistic regression?
    The classifier’s output \(\sigma(w \cdot x + b)\): the estimated probability that the observation is in the positive class.

Source