A Maximum Entropy Approach to Natural Language Processing

3.3 The maximum entropy principle

1. How is the uniformity of a conditional model measured?
   By its conditional entropy \(H(p) = -\sum_{x,y} \tilde{p}(x)\, p(y \mid x) \log p(y \mid x)\), between 0 (no uncertainty) and \(\log |\mathcal{Y}|\) (uniform over outputs).
   • Berger et al., 1996, A Maximum Entropy Approach to Natural Language Processing (Computational Linguistics), §3.3
2. State the maximum entropy principle formally.
   Given the set \(\mathcal{C}\) of models satisfying the constraints, choose \(p_* = \arg\max_{p \in \mathcal{C}} H(p)\). This model always exists and is unique.
   • Berger et al., 1996, A Maximum Entropy Approach to Natural Language Processing (Computational Linguistics), §3.3
3. What can happen geometrically when linear constraints are imposed on the probability simplex?
   No constraint: any model allowed. One constraint: a line (hyperplane) of models. Two consistent constraints may pin down a single model, or be inconsistent. Constraints from a training sample are consistent by construction, and in practice leave an infinite set of models, from which maximum entropy picks one.

   • Berger et al., 1996, A Maximum Entropy Approach to Natural Language Processing (Computational Linguistics), §3.3