Natural Language Processing (Almost) from Scratch

3.4.2 Sentence-Level Log-Likelihood

1. What is the sentence-level log-likelihood (SLL) criterion?
   A path of tags gets score \(s = \sum_t ([A]_{[i]_{t-1}, [i]_t} + [f_\theta]_{[i]_t, t})\), adding trained transition scores to network scores; the probability of the true path is its score normalized over all paths with logadd, computed in linear time by a forward recursion. Decoding uses Viterbi.
   • Collobert et al., 2011, Natural Language Processing (Almost) from Scratch (Journal of Machine Learning Research), §3.4.2, pp. 2506–2507
2. How does SLL relate to a CRF?
   It maximizes the same likelihood as a linear-chain CRF, but with network scores instead of a linear model. The scores aren’t normalized per position, so they act like CRF potentials and avoid the label-bias problem; the non-linear network learns features for each task.
   • Collobert et al., 2011, Natural Language Processing (Almost) from Scratch (Journal of Machine Learning Research), §3.4.2, pp. 2506–2507