learn.ifs.name

Logistic Regression and Text Classification

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4

9 cards in “4.7 Multinomial logistic regression” · whole deck

4.7 Multinomial logistic regression

  1. When do we need multinomial logistic regression? What else is it called?
    With more than two classes: 3-way sentiment, part-of-speech tags, named-entity types, or the \(|V|\)-way next-word prediction of LLMs. Also called softmax regression (older name: maxent classifier).
  2. What is hard classification?
    Each observation belongs to exactly one of the \(K\) classes.
  3. What is a one-hot vector?
    A vector with one element equal to 1 (the correct class) and all others 0; used to represent the gold label \(y\) in multiclass classification.
  4. Define the softmax function.
    For \(z = [z_1, \ldots, z_K]\): \(\text{softmax}(z_i) = \frac{\exp(z_i)}{\sum_{j=1}^{K}\exp(z_j)}\). It maps \(K\) arbitrary values to a probability distribution (each in [0, 1], summing to 1).
    logits zsoftmax(z)0.61.1−1.53.2−1.1.06.10.01.82.01
    The largest score takes most of the probability mass.
  5. Like the sigmoid, what does softmax do to its inputs?
    It squashes values toward 0 or 1: if one input is larger than the others, its probability is pushed toward 1 and the smaller inputs are suppressed. Its inputs are also called logits.
  6. How does multinomial logistic regression compute class probabilities?
    With a separate weight vector \(w_k\) and bias \(b_k\) for each class, packed into a matrix \(W\) of shape \([K \times f]\) and vector \(b\): \(\hat{y} = \text{softmax}(Wx + b)\).
  7. Why can each row \(w_k\) of the weight matrix be seen as a prototype of class \(k\)?
    The dot product acts as a similarity function, so logistic regression learns a template vector per class and assigns inputs to the class whose template they are most similar to.
  8. How do features behave differently in multinomial logistic regression than in binary?
    Each class has its own weight for each feature, so a feature can be evidence for or against each class separately, e.g. “!” might have positive weights for + and − but a negative weight for neutral.
  9. What does the feature notation \(f(x, y)\) express?
    That a feature (with its single weight) depends on both the input and the output class, e.g. \(f_5(x, +)\), \(f_5(x, -)\), \(f_5(x, 0)\). It is used later for CRFs.

Source