Robust Speech Recognition via Large-Scale Weak Supervision

3.3 English Speech Recognition

1. Why can models that are “superhuman” on LibriSpeech still be worse than humans elsewhere, according to Radford et al.?
   Human performance measures out-of-distribution generalization (people get no training on the test distribution), while models trained on LibriSpeech are measured in-distribution. The same test set thus measures two different abilities.
   • Radford et al., 2023, Robust Speech Recognition via Large-Scale Weak Supervision (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §3.3
2. What is effective robustness, and how did Whisper fare on it?
   How much better a model does on out-of-distribution datasets than expected from its score on a reference dataset (here LibriSpeech). A zero-shot Whisper model and a supervised model with nearly equal LibriSpeech WER differed hugely elsewhere: Whisper made 55.2% fewer errors on average across other datasets, roughly matching a human’s robustness.
   • Radford et al., 2023, Robust Speech Recognition via Large-Scale Weak Supervision (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §3.3