A Structural Probe for Finding Syntax in Word Representations
Hewitt & Manning, 2019, A Structural Probe for Finding Syntax in Word Representations (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019))
Subscriber deck — free during early access
1 cards in “4.1 Analysis of linear transformation rank” · whole deck
4.1 Analysis of linear transformation rank
- How many dimensions does syntax need in ELMo and BERT, according to the structural probe?Surprisingly few: increasing the probe’s rank \(k\) beyond 64 or 128 gave no further gains in parsing accuracy, and all three models needed roughly the same rank.
- Hewitt & Manning, 2019, A Structural Probe for Finding Syntax in Word Representations (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §4.1, p. 4133
Source
- John Hewitt and Christopher D. Manning. 2019. A Structural Probe for Finding Syntax in Word Representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT 2019), pp. 4129–4138. License: CC BY 4.0. Flashcards adapted from this work. doi:10.18653/v1/N19-1419 https://aclanthology.org/N19-1419/