Deep contextualized word representations

5.3 What information is captured by the biLM’s representations?

1. How do ELMo’s contextual vectors handle a polysemous word like play?
   GloVe neighbours of play mix parts of speech and are mostly sports-related (playing, game, player). Nearest-neighbour sentences by biLM context vectors match both part of speech and sense: a baseball “play” finds another sports “play”, a Broadway “play” finds a theatre “play”.
   • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §5.3, pp. 2233–2234
2. Which biLM layers encode syntax and which encode word sense?
   Used directly for prediction, the first biLSTM layer was better for POS tagging (97.3% on PTB, competitive with task-specific taggers) and the second layer for word sense disambiguation (69.0 F1 via nearest-neighbour senses). Lower layers capture syntax, higher layers context-dependent meaning; both beat CoVe’s layers.
   • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §5.3, pp. 2233–2234
3. Why does it help to expose all biLM layers to the task model?
   Different layers carry different kinds of information (syntax lower, semantics higher), so letting each task learn its own mixture lets it choose the most useful types of semi-supervision.
   • Peters et al., 2018, Deep contextualized word representations (Proceedings of the 2018 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2018)), §5.3, pp. 2233–2234