learn.ifs.name

Neural Networks

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6

7 cards in “6.1 Units” · whole deck

6.1 Units

  1. What does a single neural unit compute?
    A weighted sum of its inputs plus a bias, \(z = w \cdot x + b\), then a non-linear activation function: \(a = f(z)\).
    x₁x₂x₃w₁w₂w₃Σ+ bzfa = y
    Weighted sum, bias, non-linearity.
  2. What is the activation value of a unit?
    The output \(a = f(z)\) of its non-linear function. For a single-unit network it is also the network output \(y\).
  3. Compute the output of a sigmoid unit with \(w = [0.2, 0.3, 0.9]\), \(b = 0.5\), on \(x = [0.5, 0.6, 0.1]\).
    \(z = 0.1 + 0.18 + 0.09 + 0.5 = 0.87\), so \(y = \sigma(0.87) = \frac{1}{1 + e^{-0.87}} \approx 0.70\).
  4. What is the tanh activation function, and how does it compare to the sigmoid?
    \(\tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}}\), a variant of the sigmoid ranging from −1 to +1. It is almost always better than the sigmoid; it is smoothly differentiable and maps outliers toward the mean.
  5. What is the ReLU?
    The rectified linear unit, \(\text{ReLU}(z) = \max(z, 0)\): equal to \(z\) when positive, 0 otherwise. The simplest and perhaps most commonly used activation.
    tanh: −1 … 1ReLU: max(z, 0)
  6. What does it mean for a sigmoid or tanh unit to be saturated, and why is that a problem?
    For large |z| its output is extremely close to its limit and its derivative is close to 0. Since training multiplies gradients across layers, near-zero gradients shrink the error signal until it is too small to learn from: the vanishing gradient problem.
  7. Why don’t ReLUs suffer from the vanishing gradient problem?
    They are close to linear: for positive \(z\) the derivative is 1, not close to 0.

Source