Deep contextualized word representations

3.4 Pre-trained bidirectional language model architecture

1. Describe ELMo’s pretrained biLM.
   A purely character-based token layer (2048 character n-gram convolution filters, two highway layers, projection to 512), then 2 biLSTM layers with 4096 units and 512-dimensional projections plus a residual connection, trained 10 epochs on the 1B Word Benchmark (average perplexity 39.7). Character input gives representations even for words outside the training vocabulary.
   • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §3.4, p. 2230
2. Why was the ELMo biLM sometimes fine-tuned before use?
   Fine-tuning the biLM (language modelling only) on domain-specific data significantly lowered perplexity and improved downstream performance, a form of domain transfer; the biLM is then frozen for the task.
   • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §3.4, p. 2230