A Maximum Entropy Approach to Natural Language Processing

3.5 Relation to maximum likelihood

1. How are maximum entropy and maximum likelihood related?
   The dual function \(\Psi(\lambda)\) is exactly the log-likelihood \(L_{\tilde{p}}(p_\lambda) = \sum_{x,y} \tilde{p}(x, y) \log p_\lambda(y \mid x)\) of the training data. So the maximum entropy model subject to the constraints is the exponential model that maximizes the likelihood of the training sample.

   primal	maximum entropy: \(\arg\max_{p \in \mathcal{C}} H(p)\), constrained search over models
   dual	maximum likelihood: \(\arg\max_\lambda \Psi(\lambda)\), unconstrained search over \(\lambda\)

   • Berger et al., 1996, A Maximum Entropy Approach to Natural Language Processing (Computational Linguistics), §3.5
2. Why is this duality an extra justification for maximum entropy?
   Even if choosing the most uniform model isn’t compelling in itself, the same model is the one in its parametric family that best accounts for the training sample.
   • Berger et al., 1996, A Maximum Entropy Approach to Natural Language Processing (Computational Linguistics), §3.5