Neural Networks

6.3 Feedforward neural networks

1. Why is the name multi-layer perceptron (MLP) a misnomer?
   Modern multilayer networks don’t use perceptrons (step functions) but units with non-linearities like ReLU or sigmoid; the name just stuck.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.3, p. 157
2. What three kinds of nodes does a simple feedforward network have?
   Input units, hidden units, and output units.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.3, p. 157
3. What does it mean that a layer is fully connected?
   Each unit takes input from all units of the previous layer: there is a link between every pair of units in adjacent layers.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.3, p. 157
4. What does element \(W_{ji}\) of a hidden layer’s weight matrix represent?
   The weight of the connection from input unit \(x_i\) to hidden unit \(h_j\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.3, p. 157
5. How is a hidden layer computed in matrix form, and what are the shapes?
   \(h = \sigma(Wx + b)\), with the activation applied element-wise. For input \(x \in \mathbb{R}^{n_0}\) and \(n_1\) hidden units: \(W\) is \([n_1 \times n_0]\), \(b, h \in \mathbb{R}^{n_1}\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.3, p. 158
6. How does the output layer of a classification network produce probabilities?
   It multiplies the hidden representation by a weight matrix, \(z = Uh\) (\(U\) of shape \([n_2 \times n_1]\)), then normalizes with softmax: \(y = \text{softmax}(z)\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.3, p. 159
7. In what sense is a one-hidden-layer network like multinomial logistic regression?
   It builds a hidden representation \(h\) of the input and runs multinomial logistic regression on it. But the features in \(h\) are induced by the network rather than designed by hand, layers can be stacked, and various activations are used.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.3, p. 159
8. Write the equations of a 2-layer feedforward classifier.
   \(h = \sigma(Wx + b)\), \(z = Uh\), \(y = \text{softmax}(z)\).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.3, p. 159
9. How are layers counted in a feedforward network?
   The input layer is not counted, the output layer is. So a network with one hidden layer is a 2-layer network, and logistic regression is a 1-layer network.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.3, p. 159
10. How is the forward pass of an \(n\)-layer network computed, in layer notation?
    With \(a^{[0]} = x\): for \(i = 1 \ldots n\), \(z^{[i]} = W^{[i]}a^{[i-1]} + b^{[i]}\) and \(a^{[i]} = g^{[i]}(z^{[i]})\); output \(\hat{y} = a^{[n]}\). Intermediate \(g\) is usually ReLU or tanh, the last is softmax.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.3, p. 160
11. What are the logits of a network?
    The unnormalized scores \(z^{[n]}\) in the final layer, right before the softmax.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.3, p. 160
12. Why do neural networks need non-linear activation functions?
    Without them, stacked layers collapse into one: \(W^{[2]}(W^{[1]}x + b^{[1]}) + b^{[2]} = W'x + b'\). A multilayer linear network is just a single-layer network with different weights.
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.3, p. 160
13. How can the bias be represented without an explicit bias vector?
    Add a dummy input \(x_0 = 1\) to each layer; its weight \(W_{j0}\) plays the role of the bias \(b_j\): \(h = \sigma(Wx)\).
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.3, p. 161