learn.ifs.name

Neural Networks

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6

15 cards in “6.6 Training neural nets” · whole deck

6.6 Training neural nets

  1. What three things does training a neural network require?
    A loss function (cross-entropy), an optimization algorithm (gradient descent), and a way to compute the gradient of the loss with respect to every parameter (error backpropagation).
  2. What loss is used for a neural classifier?
    Cross-entropy: binary \(-[y\log\hat{y} + (1-y)\log(1-\hat{y})]\) with a sigmoid output; for \(K\) classes, \(-\log\hat{y}_c\) for the correct class \(c\), i.e. the negative log likelihood.
  3. Why can’t we just use the logistic regression gradient for a deep network?
    It only gives correct updates for the last weight layer; the loss is computed at the end, but we need derivatives with respect to weights deep in early layers.
  4. What is error backpropagation?
    The algorithm (Rumelhart et al., 1986) for computing gradients in deep networks; it is an instance of backward differentiation on computation graphs.
  5. What is a computation graph?
    A representation of a mathematical computation broken down into separate operations, each a node, with edges carrying outputs to the next operations.
  6. Run the forward pass of \(L(a, b, c) = c(a + 2b)\) at \(a = 3\), \(b = 1\), \(c = -2\).
    \(d = 2b = 2\), \(e = a + d = 5\), \(L = c \cdot e = -10\).
    abcd = 2be = a + dL = c · e3125−2−10∂L/∂a = −2∂L/∂b = −4∂L/∂c = 5
    Black: forward values. Orange: gradients from the backward pass.
  7. What is the intuition of backward differentiation?
    Pass gradients back from the final node: each node multiplies the upstream gradient it receives by its local gradient (of its output with respect to each input) to get the downstream gradient for earlier nodes (chain rule).
  8. Compute \(\partial L/\partial a\), \(\partial L/\partial b\), \(\partial L/\partial c\) for \(L = c(a + 2b)\) at \(a = 3, b = 1, c = -2\).
    \(\partial L/\partial c = e = 5\). \(\partial L/\partial e = c = -2\), so \(\partial L/\partial a = \frac{\partial L}{\partial e}\frac{\partial e}{\partial a} = -2 \cdot 1 = -2\) and \(\partial L/\partial b = \frac{\partial L}{\partial e}\frac{\partial e}{\partial d}\frac{\partial d}{\partial b} = -2 \cdot 1 \cdot 2 = -4\).
  9. Why is the forward pass needed before the backward pass?
    It computes the intermediate values (like \(d\) and \(e\)) that the local derivatives depend on.
  10. What are the derivatives of the sigmoid, tanh and ReLU?
    \(\sigma'(z) = \sigma(z)(1 - \sigma(z))\); \(\tanh'(z) = 1 - \tanh^2(z)\); ReLU’(z) = 0 for \(z < 0\), 1 for \(z \ge 0\) (undefined at 0, by convention 1).
  11. For a sigmoid output with cross-entropy loss, what is \(\partial L/\partial z\) at the output?
    \(a^{[2]} - y\): \(\frac{\partial L}{\partial a}\frac{\partial a}{\partial z} = \big(-\frac{y}{a} + \frac{1-y}{1-a}\big)a(1-a) = a - y\).
  12. How should neural network weights and inputs be initialized and prepared, compared with logistic regression?
    Network training is non-convex. Weights must be initialized with small random numbers (not 0, as in logistic regression), and inputs are normalized to zero mean and unit variance.
  13. What is dropout?
    A regularization method: during training, at each update, each unit’s output is set to zero with probability \(p\) (and the rest renormalized), randomly dropping units and their connections.
  14. Name typical neural network hyperparameters, and how they are set.
    Learning rate, mini-batch size, architecture (number of layers, hidden units per layer, activation functions), regularization (e.g. dropout rate), optimizer variant (e.g. Adam). They are tuned on a devset, not learned by gradient descent.
  15. What role do frameworks like PyTorch and TensorFlow play?
    They build networks as computation graphs, making gradient computation and parallelization on GPUs easy.

Source