Neural Networks

6.4 Feedforward networks for classification

1. How do you turn a logistic regression sentiment classifier with hand-built features into a neural one, and what does it gain?
   Add a hidden layer between the features and the softmax output: \(h = \sigma(Wx + b)\), \(z = Uh\), \(\hat{y} = \text{softmax}(z)\). The hidden layer can represent non-linear interactions between features.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.4, p. 162
2. How are a whole test set’s outputs computed with a feedforward network?
   Vectorize: stack inputs as rows of \(X\) \([m \times d]\), then \(H = \sigma(XW^\top + b)\), \(Z = HU^\top\), \(\hat{Y} = \text{softmax}(Z)\), with output \([m \times d_o]\). Always keep track of matrix shapes, since orderings like \(WX\) vs \(XW\) vary.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 6, §6.4, p. 163