Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks

1 Introduction

1. What problem does Connectionist Temporal Classification (CTC) solve?
   Training RNNs to label unsegmented sequences directly, e.g. transcribing speech into phonemes. Standard objectives are defined per time step, so RNNs needed pre-segmented training data and post-processing of their outputs into label sequences.
   • Graves et al., 2006, Connectionist Temporal Classification (Proceedings of the 23rd International Conference on Machine Learning (ICML 2006)), §1, pp. 369–370
2. What drawbacks of HMMs and CRFs for sequence labelling did Graves et al. list?
   They need task-specific knowledge (HMM state models, CRF input features); they need explicit, often questionable dependency assumptions for tractable inference (e.g. independent observations in HMMs); and standard HMM training is generative although labelling is discriminative.
   • Graves et al., 2006, Connectionist Temporal Classification (Proceedings of the 23rd International Conference on Machine Learning (ICML 2006)), §1, pp. 369–370
3. How did hybrid HMM–RNN systems use RNNs, and what was wrong with that?
   The HMM models long-range sequential structure, segments the data during training and turns the network’s local frame classifications into label sequences. Hybrids inherit the HMMs’ drawbacks and don’t exploit RNNs’ full power for sequence modelling.
   • Graves et al., 2006, Connectionist Temporal Classification (Proceedings of the 23rd International Conference on Machine Learning (ICML 2006)), §1, pp. 369–370
4. What is the basic idea of CTC?
   Interpret the network outputs as a probability distribution over all possible label sequences given the input sequence, and train by directly maximizing the probability of the correct labelling. The objective is differentiable, so standard backpropagation through time works.
   • Graves et al., 2006, Connectionist Temporal Classification (Proceedings of the 23rd International Conference on Machine Learning (ICML 2006)), §1, pp. 369–370