Robust Speech Recognition via Large-Scale Weak Supervision

1 Introduction

1. What weakness of self-supervised speech models like wav2vec 2.0 motivated Whisper?
   They learn excellent audio encoders from unlabeled speech but have no comparably good decoder, so each use needs supervised fine-tuning, which is complex and risks exploiting dataset-specific quirks that don’t generalize. Whisper aims for a system that works “out of the box” without fine-tuning.
   • Radford et al., 2023, Robust Speech Recognition via Large-Scale Weak Supervision (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §1
2. What is the core idea of Whisper?
   Scale weakly supervised training, on imperfect transcripts found on the internet, to 680,000 hours of labeled audio, an order of magnitude more than earlier weakly supervised datasets, and make it multilingual and multitask; the resulting models transfer zero-shot to standard benchmarks.
   • Radford et al., 2023, Robust Speech Recognition via Large-Scale Weak Supervision (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §1
3. How is Whisper’s 680,000-hour dataset composed?
   Mostly English speech recognition, plus 117,000 hours in 96 other languages and 125,000 hours of X→English speech translation.
   • Radford et al., 2023, Robust Speech Recognition via Large-Scale Weak Supervision (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §1