Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks

3.1 From Network Outputs to Labellings

1. What does a CTC network’s output layer look like?
   A softmax with one more unit than there are labels: \(y_k^t\) is the probability of label \(k\) at time \(t\), and the extra unit is the probability of a blank (no label). The probability of a path \(\pi\) of length \(T\) over labels plus blank is \(p(\pi \mid \mathbf{x}) = \prod_{t=1}^{T} y_{\pi_t}^t\), assuming outputs are conditionally independent given the network’s internal state.
   • Graves et al., 2006, Connectionist Temporal Classification (Proceedings of the 23rd International Conference on Machine Learning (ICML 2006)), §3.1, pp. 370–371
2. What is the CTC collapsing map \(\mathcal{B}\)? Give an example.
   It maps a path to a labelling by first merging repeated labels and then removing blanks: \(\mathcal{B}(a{-}ab{-}) = \mathcal{B}({-}aa{-}{-}abb) = aab\). A new label is emitted whenever the network switches from blank to a label or between different labels; a blank between two equal labels keeps them separate.
   • Graves et al., 2006, Connectionist Temporal Classification (Proceedings of the 23rd International Conference on Machine Learning (ICML 2006)), §3.1, pp. 370–371
3. How does CTC define the probability of a labelling?
   As the sum over all paths that collapse to it: \(p(\mathbf{l} \mid \mathbf{x}) = \sum_{\pi \in \mathcal{B}^{-1}(\mathbf{l})} p(\pi \mid \mathbf{x})\). Summing over alignments means no segmentation is needed.
   • Graves et al., 2006, Connectionist Temporal Classification (Proceedings of the 23rd International Conference on Machine Learning (ICML 2006)), §3.1, pp. 370–371
4. Why do CTC networks’ outputs look like spikes?
   A trained CTC network typically predicts each label as a short spike, separated by strongly predicted blanks, instead of aligning labels with manual segment boundaries as a framewise network must. Labels that always occur together (e.g. the closure dcl and the stop d) may appear as a double spike.
   • Graves et al., 2006, Connectionist Temporal Classification (Proceedings of the 23rd International Conference on Machine Learning (ICML 2006)), §3.1, pp. 370–371