Robust Speech Recognition via Large-Scale Weak Supervision

2.1 Data Processing

1. How did Whisper’s authors filter internet transcripts?
   They removed machine-generated transcripts (detected by heuristics such as missing punctuation or all-caps text), dropped pairs whose spoken and transcript languages disagreed (keeping English transcripts of non-English audio as translation data), fuzzy de-duplicated transcripts, and after training an initial model inspected high-error data sources to remove misaligned or partial ones.
   • Radford et al., 2023, Robust Speech Recognition via Large-Scale Weak Supervision (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §2.1
2. Why does Whisper predict raw transcript text without normalization?
   A sequence-to-sequence model can learn the mapping from speech to naturally written text, so no separate inverse text normalization step (e.g. turning “twenty dollars” into “$20”) is needed in the pipeline.
   • Radford et al., 2023, Robust Speech Recognition via Large-Scale Weak Supervision (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §2.1