A Structural Probe for Finding Syntax in Word Representations

2.1 The structural probe

1. Define the structural probe’s distance.
   For a linear map \(B \in \mathbb{R}^{k \times m}\) (equivalently the inner product with \(A = B^\top B\)), \(d_B(h_i, h_j)^2 = (B(h_i - h_j))^\top (B(h_i - h_j))\). \(B\) is trained so that these squared distances match tree distances for all word pairs.
   • Hewitt & Manning, 2019, A Structural Probe for Finding Syntax in Word Representations (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §2.1, p. 4130
2. What loss trains the structural distance probe?
   \(\min_B \sum_\ell \frac{1}{|s^\ell|^2} \sum_{i,j} \big| d_{T^\ell}(w_i^\ell, w_j^\ell) - d_B(h_i^\ell, h_j^\ell)^2 \big|\), the L1 difference between tree distance and predicted squared distance over all word pairs, normalized by the number of pairs per sentence.
   • Hewitt & Manning, 2019, A Structural Probe for Finding Syntax in Word Representations (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §2.1, p. 4130
3. Why did Hewitt and Manning use squared distances?
   Approximating tree distances and depths with squared vector distances and norms consistently worked better, and a metric and its square encode the same trees.
   • Hewitt & Manning, 2019, A Structural Probe for Finding Syntax in Word Representations (Proceedings of the 2019 Conference of the North American Chapter of ACL: Human Language Technologies (NAACL-HLT 2019)), §2.1, p. 4130