Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau et al., 2015, Neural Machine Translation by Jointly Learning to Align and Translate (3rd International Conference on Learning Representations (ICLR 2015))
Subscriber deck — free during early access
2 cards in “1 Introduction” · whole deck
1 Introduction
- What bottleneck in encoder–decoder NMT did Bahdanau, Cho and Bengio identify?The encoder must compress all the information of a source sentence into one fixed-length vector, which makes long sentences hard; Cho et al. had shown that basic encoder–decoder performance deteriorates rapidly as input length grows.
- Bahdanau et al., 2015, Neural Machine Translation by Jointly Learning to Align and Translate (3rd International Conference on Learning Representations (ICLR 2015)), §1
- What is the key idea of “jointly learning to align and translate”?Encode the source into a sequence of vectors instead of one, and each time a target word is generated, (soft-)search for the source positions where the most relevant information is, building a context vector from them. The decoder no longer depends on a single fixed-length vector.
- Bahdanau et al., 2015, Neural Machine Translation by Jointly Learning to Align and Translate (3rd International Conference on Learning Representations (ICLR 2015)), §1
Source
- Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In 3rd International Conference on Learning Representations (ICLR 2015). https://arxiv.org/abs/1409.0473