learn.ifs.name

Logistic Regression and Text Classification

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4

14 cards in “4.6 Gradient descent” · whole deck

4.6 Gradient descent

  1. What is the goal of gradient descent in logistic regression?
    Find the parameters \(\hat{\theta} = \arg\min_\theta \frac{1}{m}\sum_i L_{CE}(f(x^{(i)}; \theta), y^{(i)})\), with \(\theta = \{w, b\}\).
  2. What is the intuition of gradient descent?
    Find the direction in parameter space in which the loss rises most steeply (the gradient) and move in the opposite direction, like walking downhill along the steepest slope into a canyon.
  3. Why is gradient descent guaranteed to find the minimum for logistic regression but not for neural networks?
    Logistic regression’s loss is convex: it has at most one minimum and no local minima. Multi-layer network losses are non-convex, so gradient descent may get stuck in local minima.
  4. What is the gradient of a function?
    A vector pointing in the direction of greatest increase; its components are the partial derivatives with respect to each variable. For one variable it is just the slope.
  5. Write the gradient descent update for a single weight \(w\).
    \(w^{t+1} = w^t - \eta \frac{d}{dw} L(f(x; w), y)\), where \(\eta\) is the learning rate.
    w¹slope < 0w_minwloss
    The slope at w¹ is negative, so the step moves w to the right.
  6. How does the gradient generalize to many parameters?
    It is the vector of partial derivatives \(\nabla L = [\partial L/\partial w_1, \ldots, \partial L/\partial w_n, \partial L/\partial b]\), and the update is \(\theta^{t+1} = \theta^t - \eta \nabla L(f(x; \theta), y)\).
  7. What is the gradient of the cross-entropy loss with respect to weight \(w_j\) in logistic regression?
    \(\frac{\partial L_{CE}}{\partial w_j} = (\hat{y} - y)\,x_j = [\sigma(w \cdot x + b) - y]\,x_j\): the prediction error times the input value. For the bias: \(\hat{y} - y\).
  8. What is stochastic gradient descent?
    An online algorithm that computes the gradient after each (randomly chosen) training example and nudges \(\theta\) in the opposite direction, rather than waiting for the whole dataset.
  9. How are logistic regression weights initialized, and when does SGD stop?
    To 0 for logistic regression (small random values for neural networks). Stop at convergence (gradient norm below \(\epsilon\)) or when progress halts, e.g. loss goes up on a held-out set.
  10. What happens if the learning rate is too high or too low, and how is it usually scheduled?
    Too high: steps overshoot the minimum. Too low: training takes too long. Commonly it starts higher and is slowly decreased over iterations (\(\eta_k\)).
  11. What is the difference between parameters and hyperparameters?
    Parameters (weights, biases) are learned from the training set; hyperparameters (learning rate, regularization strength…) are chosen by the designer and affect how the algorithm works.
  12. Do one SGD step from \(w_1 = w_2 = b = 0\), \(\eta = 0.1\), on a positive example with \(x = [3, 2]\).
    \(\hat{y} = \sigma(0) = 0.5\), so the gradient is \((0.5 - 1)[3, 2, 1] = [-1.5, -1.0, -0.5]\). New parameters: \(\theta - 0.1 \cdot \text{gradient} = [0.15, 0.1, 0.05]\).
  13. What are batch, mini-batch and stochastic gradient descent?
    Batch: gradient over the whole training set (excellent direction, slow). Stochastic: one random example at a time (fast but choppy). Mini-batch: groups of e.g. 512 or 1024 examples, which can be processed in parallel as one vectorized computation.
  14. What is the mini-batch cost and its gradient?
    The average loss over the \(m\) examples, \(\text{Cost} = \frac{1}{m}\sum_i L_{CE}(\hat{y}^{(i)}, y^{(i)})\), with gradient \(\frac{1}{m}\sum_i (\hat{y}^{(i)} - y^{(i)}) x_j^{(i)}\), or in matrix form \(\frac{1}{m}(\hat{y} - y)^\top X\).

Source