A Structural Probe for Finding Syntax in Word Representations
Hewitt & Manning, 2019, A Structural Probe for Finding Syntax in Word Representations (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019))
Subscriber deck — free during early access
2 cards in “4 Results” · whole deck
4 Results
- How did ELMo and BERT score with the structural probe compared with baselines?UUAS: LINEAR 48.9, best baseline PROJ0 59.8; ELMo layer 1 77.0, BERT-base layer 7 79.8, BERT-large layer 15 82.5. Depth root%: PROJ0 64.4 vs BERT-large layer 16 90.1. BERT-large beat BERT-base, which beat ELMo, with clear differences between layers.
- Hewitt & Manning, 2019, A Structural Probe for Finding Syntax in Word Representations (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §4, p. 4132
- How do we know the structural probe doesn’t just “learn to parse” any informative representation?Non-contextual baselines (ELMo0, DECAY0) barely beat a linear chain, and PROJ0, with all of ELMo’s capacity but no training, improved only by simple deviations from linearity; the probe is linear and constrained, unlike parser-based probes.
- Hewitt & Manning, 2019, A Structural Probe for Finding Syntax in Word Representations (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §4, p. 4132
Source
- John Hewitt and Christopher D. Manning. 2019. A Structural Probe for Finding Syntax in Word Representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2019), pp. 4129–4138. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/N19-1419 https://aclanthology.org/N19-1419/