learn.ifs.name

Transformers and Pretraining

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 7

7 cards in “Historical notes” · whole deck

Historical notes

  1. How did the IBM group’s original notion of “language model” differ from just an n-gram model?
    Jelinek’s team used it for the whole set of linguistic influences on word-sequence probabilities (grammar, semantics, discourse, speaker characteristics), contrasted with the acoustic model.
  2. Which two lines of work did Bengio et al.’s neural language model combine?
    Discriminatively trained self-supervised word predictors (Rosenfeld’s maxent LMs at IBM) and pretrained embeddings (LSA). Bengio’s network learned embeddings as part of predicting the next word.
  3. Which neural LM first beat traditional 5-gram models?
    Probably the RNN language model of Mikolov et al. (2010, 2011).
  4. Why was ELMo influential?
    It pretrained a biLSTM with a language-model objective on a large corpus, then fine-tuned and added task heads, making it clear that language models could be a general solution for NLP problems.
  5. What did GPT-2 (Radford et al., 2019) show?
    That a transformer-based autoregressive LM could perform many NLP tasks, like summarization and question answering, zero-shot.
  6. What is a foundation model?
    A general term for using large language model technology across domains (vision, speech, genetics…), where the elements need not be words.
  7. Which two lines of research did the transformer draw on?
    Attention (encoder-decoder attention from Graves 2013 and Bahdanau et al. 2015, extended to self-attention) and memory networks, which contributed the key–query–value terminology for matching a query against keys in an associative memory.

Source