BERT Rediscovers the Classical NLP Pipeline

4.1 Per-Example Analysis

1. How can BERT revise lower-level decisions? Give an example from Tenney et al.
   In “he smoked toronto in the playoffs …”, early layers tag toronto as a city (GPE); once the semantic role is resolved (toronto is what got “smoked”, ARG1), the entity type is revised to ORG, the sports team. The pipeline order holds on average but not for every example.
   • Tenney et al., 2019, BERT Rediscovers the Classical NLP Pipeline (Proceedings of the 57th Annual Meeting of ACL), §4.1, pp. 4596–4597
2. What happened to “china today” in Tenney et al.’s per-example analysis?
   In “china today blacked out a cnn interview”, today was first tagged as a common noun, date and temporal modifier; higher layers reinterpreted “china today” as a proper noun (a TV network), updating the entity type to ORG and the semantic role to agent (ARG0).
   • Tenney et al., 2019, BERT Rediscovers the Classical NLP Pipeline (Proceedings of the 57th Annual Meeting of ACL), §4.1, pp. 4596–4597