Logistic Regression and Text Classification

4.3 Classification with logistic regression

1. What is the decision boundary in binary logistic regression?
   0.5: output class 1 if \(P(y=1 \mid x) > 0.5\), otherwise 0.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.3, p. 97
2. Give examples of features for binary sentiment classification of a review.
   Count of positive-lexicon words, count of negative-lexicon words, whether “no” occurs, count of 1st/2nd person pronouns, whether “!” occurs, and the log of the document length.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.3, p. 97
3. With weights [2.5, −5.0, −1.2, 0.5, 2.0, 0.7], \(b = 0.1\) and features [3, 2, 1, 3, 0, 4.19], what does the sentiment classifier output?
   \(z = 7.5 - 10 - 1.2 + 1.5 + 0 + 2.93 + 0.1 \approx 0.83\), so \(P(+ \mid x) = \sigma(0.83) \approx 0.70\) and \(P(- \mid x) \approx 0.30\): positive.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.3, pp. 97–98
4. What does it mean that \(w_2 = -5.0\) while \(w_1 = 2.5\) in the sentiment example?
   Negative-lexicon words count against the positive class, and each is about twice as important as a positive word.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.3, p. 98
5. What features might a period-disambiguation classifier (end-of-sentence vs not) use?
   Whether the current word is lowercase (likely EOS), whether it is in an abbreviation dictionary (likely not EOS), and combinations like “word is St. and previous word is capitalized” (likely a street name, not EOS).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.3, p. 98
6. What are feature interactions and feature templates?
   Feature interactions are complex features combining primitive ones. Feature templates are abstract specifications that generate features automatically (e.g. one feature for every word bigram before a period), usually hashed to integer ids; the resulting feature space is sparse.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.3, p. 98
7. Why do recent NLP systems avoid hand-designed features?
   Designing features takes extensive human effort, so systems use representation learning: learning features automatically from the input.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.3, pp. 98–99
8. How do you standardize (z-score) a feature, and how do you normalize it to [0, 1]?
   Standardize: \(x_i' = \frac{x_i - \mu_i}{\sigma_i}\) (zero mean, unit standard deviation over the dataset). Normalize: \(x_i' = \frac{x_i - \min(x_i)}{\max(x_i) - \min(x_i)}\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.3, p. 99
9. Why scale input features?
   Features with very different ranges are hard to compare, and comparable ranges speed up gradient descent, especially in large neural networks. Count features that follow a Zipfian distribution are often log-transformed.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.3, p. 99
10. How do you compute logistic regression outputs for a whole batch of examples at once?
    Stack the examples as rows of a matrix \(X\) of shape \([m \times f]\): \(\hat{y} = \sigma(Xw + b)\), with \(w\) of shape \([f \times 1]\), \(b\) the bias repeated \(m\) times, and \(\sigma\) applied element-wise. One matrix multiply replaces a for-loop.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.3, p. 100