BERT Rediscovers the Classical NLP Pipeline

2 Model

1. What is edge probing?
   A common format for structured prediction tasks: a probing classifier gets one or two spans, \(s_1 = [i_1, j_1)\) and optionally \(s_2 = [i_2, j_2)\), and predicts a label (e.g. constituent or relation type), seeing only the contextual vectors within the spans, so it must rely on the encoder for information about the spans’ roles.
   • Tenney et al., 2019, BERT Rediscovers the Classical NLP Pipeline (Proceedings of the 57th Annual Meeting of ACL), §2, pp. 4593–4594
2. Which eight probing tasks did Tenney et al. use?
   Part-of-speech tags, constituents, dependencies, entities, semantic role labeling, coreference, semantic proto-roles (SPR), and relation classification (SemEval), all scored with micro-averaged F1.
   • Tenney et al., 2019, BERT Rediscovers the Classical NLP Pipeline (Proceedings of the 57th Annual Meeting of ACL), §2, pp. 4593–4594
3. Why are BERT’s weights frozen when probing?
   To learn how the network represents language as a result of pretraining: fine-tuning would let the encoder rearrange its internal representations to suit the probing task.
   • Tenney et al., 2019, BERT Rediscovers the Classical NLP Pipeline (Proceedings of the 57th Annual Meeting of ACL), §2, pp. 4593–4594
4. What are the limits of inspection-based probing, according to Tenney et al.?
   Not finding a pattern doesn’t prove it isn’t there, and finding one doesn’t say how the model uses it, so structural analysis should be combined with behavioural studies.
   • Tenney et al., 2019, BERT Rediscovers the Classical NLP Pipeline (Proceedings of the 57th Annual Meeting of ACL), §2, pp. 4593–4594