Logistic Regression and Text Classification

4.15 Deriving the gradient

1. What is the derivative of the sigmoid?
   \(\frac{d\sigma(z)}{dz} = \sigma(z)(1 - \sigma(z))\)
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.15, p. 123
2. State the chain rule of derivatives.
   For \(f(x) = u(v(x))\): \(\frac{df}{dx} = \frac{du}{dv} \cdot \frac{dv}{dx}\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.15, p. 123
3. Sketch how the logistic regression gradient \((\sigma(w \cdot x + b) - y)x_j\) is derived.
   Differentiate \(-[y \log \sigma + (1-y)\log(1-\sigma)]\) using \(\frac{d}{dx}\ln x = \frac{1}{x}\) and the chain rule, getting \(-\big[\frac{y}{\sigma} - \frac{1-y}{1-\sigma}\big]\frac{\partial \sigma}{\partial w_j}\). Then substitute \(\frac{\partial \sigma}{\partial w_j} = \sigma(1-\sigma)x_j\); the \(\sigma(1-\sigma)\) terms cancel, leaving \(-(y - \sigma)x_j\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 4, §4.15, p. 124