Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

2 The label bias problem

1. Explain the label bias problem with the rib/rob example.
   A finite-state model branches at r into a path for rib (state 1) and one for rob (state 4). On input r i b, both states 1 and 4 have a single outgoing transition, so state 4 must pass on all its mass though it has rarely seen i: the two paths stay about equally likely regardless of the observations, and whichever word was slightly more frequent in training always wins.

   • Lafferty et al., 2001, Conditional Random Fields (Proceedings of the 18th International Conference on Machine Learning (ICML)), §2
2. What remedies did Bottou propose for label bias, and why are they unsatisfactory?
   Changing the state structure, e.g. collapsing states 1 and 4 to delay branching (determinization, not always possible and can explode combinatorially); or starting fully connected and letting training find structure, which discards valuable prior structural knowledge.
   • Lafferty et al., 2001, Conditional Random Fields (Proceedings of the 18th International Conference on Machine Learning (ICML)), §2
3. What property must a proper solution to label bias have?
   It must score whole state sequences at once, letting some transitions “vote” more strongly than others depending on the observations: score mass isn’t conserved, and transitions can amplify or dampen the mass they receive.
   • Lafferty et al., 2001, Conditional Random Fields (Proceedings of the 18th International Conference on Machine Learning (ICML)), §2