Introduction

1.5 Linguistic structure and interpretability

1. What does syntactic structure describe? Give an example.
   Abstract grammatical relations between words or phrases, e.g. that a verb has a subject and a direct object: in “She announced this in January”, She is the nsubj and this the dobj of announced. Detecting these relations is parsing.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.5, p. 14
2. What does it mean for phrases to corefer?
   They refer to the same entity in our mental model of the world, e.g. “Victoria Chen”, “CFO of Megabucks Banking”, “her” and “the 38-year-old”.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.5, p. 14
3. What role does linguistic structure play in modern NLP, now that parsing is no longer a required step for extracting meaning?
   It is crucial for LLM interpretability and analysis, and for applications in the cognitive and social sciences.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.5, p. 14
4. What is interpretability?
   The study of how LLMs work, by uncovering the model’s internal linguistic structures or computations.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.5, p. 14
5. What linguistic structure has interpretability research found inside LMs?
   Embeddings implicitly represent syntactic parse trees, semantic information and coreference (Hewitt & Manning, 2019). Some attention heads track grammar, e.g. heads where direct objects attend to their verbs, or where a mention attends to its antecedent.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.5, p. 14
6. What are common ground and grounding in conversation?
   Common ground is the mutual understanding conversational partners try to establish. Grounding is linguistically signaling understanding or lack of it, e.g. asking a clarification question (“Which one?”).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.5, p. 15
7. What have linguistic analyses of conversations revealed about current LLMs’ grounding behavior?
   They are poor at grounding: they avoid asking clarification questions and so make wrong assumptions about what users want. Such analysis also helps detect sycophancy and overconfidence.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.5, p. 15
8. Give an example of NLP tools serving the social sciences.
   Card et al. (2022) parsed political speeches to study how US politicians talk about immigrant groups: as contributing (hardworking, contribute) versus to be excluded (deport, reject).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.5, p. 15
9. What is computational linguistics, as sometimes distinguished from NLP?
   Using computation of linguistic structure to answer questions about language itself, e.g. how language changes over time or across individuals, or how children acquire it.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.5, p. 15
10. How have researchers used LLMs to generate insights about language learning?
    By training models on artificial languages with particular syntactic structures to see which kinds of languages are easier or harder for LLMs to learn (Papadimitriou & Jurafsky, 2023; Kallini et al., 2024).
    • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 1, §1.5, p. 15