learn.ifs.name

Deep contextualized word representations

Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018))

Subscriber deck — free during early access

20 cards

1 Introduction

  1. What two properties should good word representations model, according to Peters et al.?
    Complex characteristics of word use (syntax and semantics) and how these uses vary across linguistic contexts, i.e. polysemy. Type embeddings like word2vec and GloVe give each word one context-independent vector.
    • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §1, pp. 2227–2228
  2. What is ELMo?
    Embeddings from Language Models: each token’s representation is a function of the entire input sentence, computed from the internal states of a deep bidirectional language model (biLM) pretrained on a large corpus.
    • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §1, pp. 2227–2228
  3. How is ELMo “deep” compared with earlier contextual vectors (TagLM, CoVe)?
    Earlier approaches used only the top LSTM layer. ELMo is a learned, task-specific linear combination of all biLM layers, which markedly improves performance.
    • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §1, pp. 2227–2228
  4. What did adding ELMo achieve across tasks?
    Simply adding ELMo to strong existing models set a new state of the art on all six tasks tried (question answering, textual entailment, semantic role labeling, coreference, NER, sentiment), with relative error reductions of 6–20%.
    • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §1, pp. 2227–2228

2 Related work

  1. How does ELMo differ from CoVe (McCann et al. 2017)?
    CoVe takes contextual vectors from a supervised machine translation encoder, so it is limited by parallel data; ELMo uses a language model trained on plentiful monolingual text (about 30M sentences) and combines all layers, not just the top.
  2. How does ELMo’s use of pretraining differ from fine-tuning pretrained encoders (Dai and Le 2015)?
    After pretraining, ELMo freezes the biLM weights and adds task-specific capacity on top, so large, rich biLM representations can serve tasks whose data only supports a small supervised model.

3.1 Bidirectional language models

  1. What objective does the ELMo biLM optimize?
    It jointly maximizes forward and backward log-likelihoods: \(\sum_{k=1}^{N} \big(\log p(t_k \mid t_1, \ldots, t_{k-1}) + \log p(t_k \mid t_{k+1}, \ldots, t_N)\big)\), with separate LSTMs per direction but shared token representation and softmax parameters.
    • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §3.1, pp. 2228–2229

3.2 ELMo

  1. Give the formula for the ELMo vector of token \(k\).
    \(\text{ELMo}_k^{task} = \gamma^{task} \sum_{j=0}^{L} s_j^{task} h_{k,j}^{LM}\), where \(h_{k,0}^{LM}\) is the token layer, \(h_{k,j}^{LM} = [\overrightarrow{h}_{k,j}; \overleftarrow{h}_{k,j}]\) for biLSTM layer \(j\), \(s^{task}\) are softmax-normalized weights and \(\gamma^{task}\) scales the whole vector.
    char. token layerbiLSTM layer 1biLSTM layer 2word tk× s0× s1× s2Σ× γELMok
    The task learns the weights sj (softmax-normalized) and the scale γ; the biLM is frozen

3.3 Using biLMs for supervised NLP tasks

  1. How is ELMo added to a supervised model?
    Run the frozen biLM, record all layer representations, and concatenate the learned ELMo vector with the model’s token representation: \([x_k; \text{ELMo}_k]\) goes into the task RNN. Some tasks (SNLI, SQuAD) also gain from adding ELMo at the RNN output. Dropout on ELMo and an L2 penalty pulling layer weights toward an average help.
    • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §3.3, pp. 2229–2230

3.4 Pre-trained bidirectional language model architecture

  1. Describe ELMo’s pretrained biLM.
    A purely character-based token layer (2048 character n-gram convolution filters, two highway layers, projection to 512), then 2 biLSTM layers with 4096 units and 512-dimensional projections plus a residual connection, trained 10 epochs on the 1B Word Benchmark (average perplexity 39.7). Character input gives representations even for words outside the training vocabulary.
  2. Why was the ELMo biLM sometimes fine-tuned before use?
    Fine-tuning the biLM (language modelling only) on domain-specific data significantly lowered perplexity and improved downstream performance, a form of domain transfer; the biLM is then frozen for the task.

4 Evaluation

  1. How much did ELMo improve each task?
    Over strong baselines on test sets:
    taskbaseline+ ELMorel. error ↓
    SQuAD (F1)81.185.824.9%
    SNLI (acc.)88.088.75.8%
    SRL (F1)81.484.617.2%
    Coref (avg F1)67.270.49.8%
    NER (F1)90.1592.2221%
    SST-5 (acc.)51.454.76.8%
    • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §4, pp. 2230–2232

5.1 Alternate layer weighting schemes

  1. Does combining all biLM layers beat using only the top layer?
    Yes. On SQuAD dev, the top layer alone added 3.9 F1 over the baseline; averaging all layers added 0.3 more, and learning individual layer weights (small regularization \(\lambda = 0.001\)) another 0.2. SNLI and SRL showed the same trend; CoVe benefited similarly but less.

5.2 Where to include ELMo?

  1. Where in a task model should ELMo be included?
    Always at the input; for SNLI and SQuAD, adding it at the output of the task biRNN too helped, perhaps because their attention layers can then attend to biLM representations directly. For SRL and coreference, input only was best.
    • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §5.2, pp. 2232–2233

5.3 What information is captured by the biLM’s representations?

  1. How do ELMo’s contextual vectors handle a polysemous word like play?
    GloVe neighbours of play mix parts of speech and are mostly sports-related (playing, game, player). Nearest-neighbour sentences by biLM context vectors match both part of speech and sense: a baseball “play” finds another sports “play”, a Broadway “play” finds a theatre “play”.
    • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §5.3, pp. 2233–2234
  2. Which biLM layers encode syntax and which encode word sense?
    Used directly for prediction, the first biLSTM layer was better for POS tagging (97.3% on PTB, competitive with task-specific taggers) and the second layer for word sense disambiguation (69.0 F1 via nearest-neighbour senses). Lower layers capture syntax, higher layers context-dependent meaning; both beat CoVe’s layers.
    • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §5.3, pp. 2233–2234
  3. Why does it help to expose all biLM layers to the task model?
    Different layers carry different kinds of information (syntax lower, semantics higher), so letting each task learn its own mixture lets it choose the most useful types of semi-supervision.
    • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §5.3, pp. 2233–2234

5.4 Sample efficiency

  1. How does ELMo affect sample efficiency?
    Strongly: the SRL model without ELMo peaked after 486 epochs, while with ELMo it passed that score at epoch 10 (98% fewer updates). Gains were largest for small training sets; with 1% of the SRL data, ELMo matched the baseline trained on 10%.

5.5 Visualization of learned weights

  1. What layer weights did tasks learn for ELMo?
    At the input, tasks favoured the first biLSTM layer, strongly so for coreference and SQuAD; at the output, weights were fairly balanced with a slight preference for lower layers.

5.6 Contextual vs. sub-word information

  1. Do ELMo’s gains come from context or from character-level subword information?
    Mostly context: replacing GloVe with only ELMo’s character-based token layer helped slightly (SQuAD dev F1 80.8 → 81.4), but full ELMo gave 85.3. Adding GloVe on top of ELMo gave only marginal further gains.

Source

  • Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2018), pp. 2227–2237. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/N18-1202 https://aclanthology.org/N18-1202/