learn.ifs.name

RNNs and LSTMs

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 14

6 cards in “14.3 RNNs for other NLP tasks” · whole deck

14.3 RNNs for other NLP tasks

  1. How is an RNN used for sequence labeling, e.g. part-of-speech tagging?
    Input word embeddings; at each step a softmax over the tagset gives tag probabilities; pick the most likely tag per step; train with cross-entropy.
  2. How is an RNN used for sequence classification?
    Run the text through the RNN and pass the last hidden state \(h_n\), a compressed representation of the whole sequence, to a feedforward network with a softmax over classes. Only the final classification produces a loss.
  3. What is end-to-end training?
    Using the loss from the downstream task to adjust weights all the way through the network (classifier and RNN).
  4. What alternative to the last hidden state can represent a sequence for classification?
    Pooling all hidden states, e.g. the element-wise mean \(\frac{1}{n}\sum_i h_i\) or element-wise max.
  5. How does an RNN LM generate text autoregressively?
    Feed <s>, sample a word from the softmax, feed that word’s embedding as the next input, sample again, until </s> or a length limit.
  6. How is autoregressive generation adapted to tasks like translation or summarization?
    Prime the generator with a task-appropriate context instead of just <s>: the source sentence for translation, the long text for summarization.

Source