Deep contextualized word representations

2 Related work

1. How does ELMo differ from CoVe (McCann et al. 2017)?
   CoVe takes contextual vectors from a supervised machine translation encoder, so it is limited by parallel data; ELMo uses a language model trained on plentiful monolingual text (about 30M sentences) and combines all layers, not just the top.
   • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §2, p. 2228
2. How does ELMo’s use of pretraining differ from fine-tuning pretrained encoders (Dai and Le 2015)?
   After pretraining, ELMo freezes the biLM weights and adds task-specific capacity on top, so large, rich biLM representations can serve tasks whose data only supports a small supervised model.
   • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §2, p. 2228