Robust Speech Recognition via Large-Scale Weak Supervision

2.3 Multitask Format

1. How does a single Whisper model perform several speech tasks?
   Tasks are specified as a token sequence for the decoder: a start token, a language token (or a no-speech token), a task token (transcribe or translate), an optional no-timestamps token, then the text (with interleaved time tokens if timestamps are on), and an end token. Earlier transcript text can be given as context.

   • Radford et al., 2023, Robust Speech Recognition via Large-Scale Weak Supervision (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §2.3
2. How does Whisper predict timestamps?
   As extra vocabulary tokens for times relative to the 30-second segment, quantized to 20 ms; a start time token precedes each caption’s text and an end time token follows it.
   • Radford et al., 2023, Robust Speech Recognition via Large-Scale Weak Supervision (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §2.3
3. What odd behaviour did early Whisper models show, and how was it fixed?
   They often “transcribed” plausible but wrong speaker names, because many training transcripts name the speaker; brief fine-tuning on transcripts without speaker annotations removed it.
   • Radford et al., 2023, Robust Speech Recognition via Large-Scale Weak Supervision (Proceedings of the 40th International Conference on Machine Learning (ICML 2023), PMLR 202), §2.3