A Maximum Entropy Approach to Natural Language Processing

5.2 Context-dependent word models

1. How were training events built for the context-dependent translation model of in?
   From Hansard sentence pairs: compute the Viterbi alignment, and for each occurrence of in take the context \(x\) = the six surrounding English words and the output \(y\) = the aligned French word.
   • Berger et al., 1996, A Maximum Entropy Approach to Natural Language Processing (Computational Linguistics), §5.2
2. What feature templates did the in translation model use?

   template	f(x, y) = 1 iff …
   1	y = ◇ (no context)
   2	y = ◇ and the next word is □
   3	y = ◇ and the previous word is □
   4	y = ◇ and □ is among the 3 previous words
   5	y = ◇ and □ is among the 3 next words

   ◇ is a French translation, □ an English word. Template 1 alone reproduces the context-free distribution.
   • Berger et al., 1996, A Maximum Entropy Approach to Natural Language Processing (Computational Linguistics), §5.2
3. What did the feature selection algorithm learn for translating in?
   First a template-1 constraint for each translation seen, then context features chosen by approximate gain, e.g. that dans is more likely when speech is among the next three words; the run model showed a distinct “Hansard flavor” of parliamentary language.
   • Berger et al., 1996, A Maximum Entropy Approach to Natural Language Processing (Computational Linguistics), §5.2