Neural Machine Translation by Jointly Learning to Align and Translate

Bahdanau et al., 2015, Neural Machine Translation by Jointly Learning to Align and Translate (3rd International Conference on Learning Representations (ICLR 2015))

Subscriber deck — free during early access

2 cards in “5.2.1 Alignment” · whole deck

5.2.1 Alignment

What did visualizing attention weights \(\alpha_{ij}\) reveal?
English–French alignments are largely monotonic (strong diagonal), with sensible non-monotonic exceptions: for “European Economic Area” → “zone économique européenne”, the model aligned zone with Area, jumping over two words, then looked back one word at a time.
Schematic weights (stronger colour = larger α): adjective–noun order is reversed in French
- Bahdanau et al., 2015, Neural Machine Translation by Jointly Learning to Align and Translate (3rd International Conference on Learning Representations (ICLR 2015)), §5.2.1
Why is a soft alignment better than a hard one for “the man” → “l’ homme”?
A hard alignment maps the → l’, but choosing between le, la, les and l’ depends on the following word. Soft alignment lets the model look at both the and man. It also handles phrases of different lengths without mapping words to or from NULL.
- Bahdanau et al., 2015, Neural Machine Translation by Jointly Learning to Align and Translate (3rd International Conference on Learning Representations (ICLR 2015)), §5.2.1

Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In 3rd International Conference on Learning Representations (ICLR 2015). https://arxiv.org/abs/1409.0473