learn.ifs.name

BERT Rediscovers the Classical NLP Pipeline

Tenney et al., 2019, BERT Rediscovers the Classical NLP Pipeline (Proceedings of the 57th Annual Meeting of ACL)

Subscriber deck — free during early access

2 cards in “1 Introduction” · whole deck

1 Introduction

  1. What is the main finding of “BERT Rediscovers the Classical NLP Pipeline”?
    BERT represents the steps of the traditional NLP pipeline in an interpretable, localizable way, and the layers responsible for each step appear in the expected order: POS tagging, parsing, NER, semantic roles, then coreference. On individual sentences, though, it can revise lower-level decisions using higher-level information.
    lowhighPOS tagsconstituents, dependenciesentitiessemantic rolescoreferencesyntax: localizedsemantics: spread out
    Schematic order of where each kind of information is encoded across BERT’s layers (aggregate trend)
  2. How does structural probing differ from behavioural analysis of models?
    Behavioural work designs controlled test sets and analyses errors to infer what abstractions a model uses; structural probing inspects the network’s internal representations directly to find localizable regions associated with particular linguistic decisions. Tenney et al. take the structural route.

Source