learn.ifs.name

Logistic Regression and Text Classification

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4

6 cards in “4.1 Machine learning and classification” · whole deck

4.1 Machine learning and classification

  1. What is the goal of classification?
    Take an input observation \(x\), extract useful features, and assign it one of a fixed set of discrete classes \(Y = \{y_1, \ldots, y_M\}\).
  2. What are the weaknesses of handwritten rules and of prompting an LLM for classification?
    Rules are fragile as data change, and hard to write when features interact (e.g. negation: “don’t love”). LLMs may hallucinate and may not be able to explain why they chose a class.
  3. What is supervised machine learning?
    Learning a classifier from a labeled training set \(\{(x^{(1)}, y^{(1)}), \ldots, (x^{(m)}, y^{(m)})\}\), where each input comes with its correct output (the supervision signal), so that it maps new inputs to their correct classes.
  4. What does a probabilistic classifier give beyond a class label, and why is that useful?
    The probability of the observation being in each class. This distribution is useful for downstream decisions; avoiding early discrete decisions helps when combining systems.
  5. What are the four components of a probabilistic machine-learning classifier like logistic regression?
    (1) A feature representation of the input; (2) a classification function computing \(P(y \mid x)\) (sigmoid or softmax); (3) an objective function to optimize, usually a loss such as cross-entropy; (4) an optimization algorithm, e.g. stochastic gradient descent.
  6. What happens in the training and test phases of logistic regression?
    Training: learn the weights \(w\) and bias \(b\) with SGD and the cross-entropy loss. Test: compute \(P(y \mid x)\) for each class and return the most probable label.

Source