learn.ifs.name

Discourse Coherence

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 25

6 cards in “25.4 Representation learning models for local coherence” · whole deck

25.4 Representation learning models for local coherence

  1. What are lexical chains and TextTiling?
    Lexical chains (Morris and Hirst): sequences of thesaurus-related words through a discourse (pine, bush, trees, trunk), whose number and density track topic structure. TextTiling (Hearst): cosine between word-count vectors of neighboring spans; high within a subtopic, low across a boundary.
  2. How does LSA coherence (Foltz et al. 1998) score a text?
    Embed each sentence as the sum of its word LSA embeddings, and average the cosine over adjacent pairs: \(coherence(T) = \frac{1}{n-1}\sum_{i=1}^{n-1}\cos(s_i, s_{i+1})\).
  3. What idea do modern neural coherence models take from Barzilay and Lapata (2005)?
    Self-supervision: train to distinguish natural discourses from ones with scrambled sentences, letting representation learning discover the features that matter, instead of hand-labeled RST/PDTB relations.
  4. How is the local coherence discriminator (LCD) trained?
    Positive examples are consecutive pairs \((s_i, s_{i+1})\); negatives pair \(s_i\) with a random sentence \(s'\) from the same document. A margin loss \(\max(0, \eta - f^+ + f^-)\) pushes the positive score above the negative; document coherence is the average pair score.
  5. What features does LCD compute for a sentence pair \(s, t\)?
    From sentence embeddings: the concatenation \([s; t]\), the difference \(s - t\), the absolute difference \(|s - t|\), and the elementwise product \(s \odot t\), fed to a one-layer feedforward network that outputs a coherence score.
  6. What is the predictability baseline for coherence (Xu et al. 2019)?
    Train a language model and compare each sentence’s log likelihood with and without the preceding context: the gain measures how much the context makes the sentence predictable.

Source