A Maximum Entropy Approach to Natural Language Processing

2 A maximum entropy overview

1. Model an expert translating English in into one of {dans, en, à, au cours de, pendant}, knowing only that set. What does maximum entropy choose?
   The uniform model: each phrase gets 1/5. Models like p(dans) = 1 assume more than we know.
   • Berger et al., 1996, A Maximum Entropy Approach to Natural Language Processing (Computational Linguistics), §2
2. Add the fact that dans or en was chosen 30% of the time. What is the maximum entropy model now?
   Spread probability as evenly as the constraints allow: p(dans) = p(en) = 3/20, and p(à) = p(au cours de) = p(pendant) = 7/30. Adding a third fact (dans or à half the time) makes the most uniform model no longer obvious, which motivates a precise measure of uniformity.
   • Berger et al., 1996, A Maximum Entropy Approach to Natural Language Processing (Computational Linguistics), §2
3. State the maximum entropy principle informally.
   Model all that is known and assume nothing about what is unknown: among models consistent with all the facts, choose the most uniform one.
   • Berger et al., 1996, A Maximum Entropy Approach to Natural Language Processing (Computational Linguistics), §2
4. How is maximum entropy related to Laplace and Jaynes?
   Laplace’s Principle of Insufficient Reason: with no information to distinguish two events, treat them as equally likely. Jaynes: the maximum-entropy distribution agrees with everything known but carefully avoids assuming anything not known.
   • Berger et al., 1996, A Maximum Entropy Approach to Natural Language Processing (Computational Linguistics), §2