A Maximum Entropy Approach to Natural Language Processing

3.2 Statistics, features and constraints

1. What is a feature in a maximum entropy model? Give an example.
   A binary-valued (indicator) function of a context \(x\) and output \(y\), e.g. \(f(x, y) = 1\) if \(y = \text{en}\) and April follows in, else 0. Any statistic of the sample can be expressed as the expected value of such a function.
   • Berger et al., 1996, A Maximum Entropy Approach to Natural Language Processing (Computational Linguistics), §3.2
2. What is a constraint in a maximum entropy model?
   An equation requiring the model’s expected value of a feature to equal its empirical expected value: \(p(f) = \tilde{p}(f)\), i.e. \(\sum_{x,y} \tilde{p}(x)\, p(y \mid x)\, f(x, y) = \sum_{x,y} \tilde{p}(x, y)\, f(x, y)\). (Berger et al. insist on not confusing features with constraints.)
   • Berger et al., 1996, A Maximum Entropy Approach to Natural Language Processing (Computational Linguistics), §3.2