Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

5 Experiments

1. How did CRFs and MEMMs compare on data generated from the rib/rob network?
   With identical topology and only symbol-identity features, the CRF erred about 5% of the time while the MEMM erred about 42%, failing to discriminate the two branches: a direct demonstration of label bias.
   • Lafferty et al., 2001, Conditional Random Fields (Proceedings of the 18th International Conference on Machine Learning (ICML)), §5
2. What did the mixed-order synthetic experiments show?
   Data came from HMMs mixing first- and second-order dependencies; as data became more second-order, all first-order models erred more. CRFs typically beat MEMMs by a wide margin, HMMs beat MEMMs, and CRFs beat HMMs on mostly second-order data: CRFs are more robust to inaccurate modelling assumptions.
   • Lafferty et al., 2001, Conditional Random Fields (Proceedings of the 18th International Conference on Machine Learning (ICML)), §5
3. What did the Penn Treebank POS tagging experiments show?
   With the same tag–word and tag–tag parameters, HMM beat MEMM (label bias) and CRF beat HMM. Adding orthographic features (capitalization, numbers, hyphens, suffixes like -ing, -ed, -tion) helped MEMMs and CRFs greatly, especially on out-of-vocabulary words.
   • Lafferty et al., 2001, Conditional Random Fields (Proceedings of the 18th International Conference on Machine Learning (ICML)), §5
4. How was slow CRF training from the uniform distribution addressed in the POS experiments?
   Initialize the CRF with the trained MEMM’s parameters; it then converged in about 1,000 iterations, while training from all-zero parameters had not converged after 2,000.
   • Lafferty et al., 2001, Conditional Random Fields (Proceedings of the 18th International Conference on Machine Learning (ICML)), §5