BERT Rediscovers the Classical NLP Pipeline

3.1 Scalar Mixing Weights

1. How do scalar mixing weights show which BERT layers matter for a task?
   As in ELMo, the probe pools layers as \(h_{i,\tau} = \gamma_\tau \sum_{\ell=0}^{L} s_\tau^{(\ell)} h_i^{(\ell)}\) with \(s_\tau = \text{softmax}(a_\tau)\), learned jointly with the probe on training data; higher weights suggest a layer holds more task-relevant information. The summary is the center of gravity \(\bar{E}_s[\ell] = \sum_\ell \ell \cdot s_\tau^{(\ell)}\).
   • Tenney et al., 2019, BERT Rediscovers the Classical NLP Pipeline (Proceedings of the 57th Annual Meeting of ACL), §3.1, pp. 4594–4595