A Structural Probe for Finding Syntax in Word Representations

1 Introduction

1. What question does the structural probe answer that earlier syntactic probes didn’t?
   Whether entire syntax trees are embedded in a model’s word representations: earlier probes tested pieces of syntactic knowledge (like POS or labels of given structures) but not whether whole parse trees are represented.
   • Hewitt & Manning, 2019, A Structural Probe for Finding Syntax in Word Representations (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §1, p. 4129
2. What is the structural probe hypothesis?
   That there is a linear transformation of the representation space under which squared L2 distance between two words’ vectors equals the number of edges between them in the parse tree, and squared L2 norm equals a word’s depth in the tree, even though the model was never trained on trees.
   • Hewitt & Manning, 2019, A Structural Probe for Finding Syntax in Word Representations (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §1, p. 4129
3. What did the structural probe find for ELMo and BERT?
   Such transformations exist for ELMo and BERT but not for baselines, and in a low-rank space: entire dependency trees are embedded implicitly in deep models’ vector geometry, analogous to word analogies being encoded as vector offsets.
   • Hewitt & Manning, 2019, A Structural Probe for Finding Syntax in Word Representations (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §1, p. 4129