BERT Rediscovers the Classical NLP Pipeline

4 Results

1. In what order are linguistic tasks encoded across BERT-large’s layers?
   Both metrics agree: POS tags earliest, then constituents and dependencies, then semantic roles and coreference. Basic syntax appears earlier in the network, high-level semantics in higher layers, consistent with Peters et al. (2018b).
   • Tenney et al., 2019, BERT Rediscovers the Classical NLP Pipeline (Proceedings of the 57th Annual Meeting of ACL), §4, p. 4596
2. How does the localization of syntactic and semantic information differ in BERT?
   Syntactic information is localizable: mixing weights concentrate on a few layers. Semantic information (relations, proto-roles) is spread across the network: weights are close to uniform and examples are resolved gradually across nearly all layers.
   • Tenney et al., 2019, BERT Rediscovers the Classical NLP Pipeline (Proceedings of the 57th Annual Meeting of ACL), §4, p. 4596
3. Why do differential scores peak early (layers 1–7) while mixing weights peak later (layers 9–20) in BERT-large?
   Many examples can be resolved early from shallow heuristic cues, while hard examples need later layers; the highest mixing weights fall on or just after the last layers that still improve F1 for the task.
   • Tenney et al., 2019, BERT Rediscovers the Classical NLP Pipeline (Proceedings of the 57th Annual Meeting of ACL), §4, p. 4596
4. How does BERT-base compare with BERT-large in the probing results?
   The same general ordering holds, with a “stretching effect”: each task’s representations concentrate at the same layers relative to the top of the model.
   • Tenney et al., 2019, BERT Rediscovers the Classical NLP Pipeline (Proceedings of the 57th Annual Meeting of ACL), §4, p. 4596