Logistic Regression and Text Classification

4.5 The cross-entropy loss function

1. What is conditional maximum likelihood estimation?
   Choosing the parameters \(w, b\) that maximize the log probability of the true labels \(y\) in the training data given the observations \(x\). The resulting loss is the negative log likelihood, i.e. cross-entropy loss.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.5, p. 101
2. How can \(p(y \mid x)\) be written for a binary classifier, and why?
   \(p(y \mid x) = \hat{y}^{\,y}(1 - \hat{y})^{1-y}\): a Bernoulli distribution that reduces to \(\hat{y}\) when \(y = 1\) and to \(1 - \hat{y}\) when \(y = 0\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.5, p. 101
3. Write the binary cross-entropy loss.
   \(L_{CE}(\hat{y}, y) = -\log p(y \mid x) = -[\,y \log \hat{y} + (1-y)\log(1-\hat{y})\,]\), with \(\hat{y} = \sigma(w \cdot x + b)\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.5, p. 102
4. For the review with \(P(+ \mid x) = 0.70\), what is the loss if the gold label is positive, and if it is negative?
   Positive: \(-\log 0.70 \approx 0.36\). Negative: \(-\log 0.30 \approx 1.2\). The loss is smaller when the model is right.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.5, p. 102
5. Why is the negative log probability of the correct class a sensible loss?
   It goes from 0 (probability 1, no loss) to infinity (probability 0). And since the class probabilities sum to 1, raising the correct one necessarily lowers the incorrect one.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.5, p. 102
6. Why is it called the cross-entropy loss?
   The formula is the cross-entropy between the true distribution \(y\) and the estimated distribution \(\hat{y}\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.5, p. 102