Logistic Regression and Text Classification

4.8 Learning in multinomial logistic regression

1. What is the cross-entropy loss for multinomial logistic regression?
   \(L_{CE}(\hat{y}, y) = -\sum_{k=1}^{K} y_k \log \hat{y}_k = -\log \hat{y}_c\): since \(y\) is one-hot, only the correct class \(c\) contributes. Hence also called the negative log likelihood loss.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.8, p. 112
2. What is the gradient of the multinomial cross-entropy loss with respect to \(w_{k,i}\)?
   \(\frac{\partial L_{CE}}{\partial w_{k,i}} = -(y_k - \hat{y}_k)\,x_i\): the difference between the true value for class \(k\) (0 or 1) and its predicted probability, times the input.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.8, p. 112