Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks

4.2 Maximum Likelihood Training

1. What is the CTC error signal with respect to the unnormalized outputs \(u_k^t\)?
   \(\frac{\partial O^{ML}}{\partial u_k^t} = y_k^t - \frac{1}{y_k^t Z_t} \sum_{s \in lab(\mathbf{z}, k)} \hat\alpha_t(s)\hat\beta_t(s)\), where \(lab(\mathbf{z}, k)\) are the positions of label \(k\) in \(\mathbf{z}'\) and \(Z_t\) normalizes: the output minus the posterior probability of \(k\) at time \(t\) over all valid alignments.
   • Graves et al., 2006, Connectionist Temporal Classification (Proceedings of the 23rd International Conference on Machine Learning (ICML 2006)), §4.2, pp. 373–374
2. How does the CTC error signal evolve during training?
   Initially, with small random weights, it is determined by the target sequence only; as the network starts predicting, the error localizes around its predictions; once it strongly predicts the correct labelling, the error virtually disappears.
   • Graves et al., 2006, Connectionist Temporal Classification (Proceedings of the 23rd International Conference on Machine Learning (ICML 2006)), §4.2, pp. 373–374