Text-to-Speech

17.4 TTS evaluation

1. What is MOS?
   Mean opinion score: listeners rate synthesized utterances on how good they are, usually on a 1–5 scale; systems are compared on the same sentences (e.g. with paired t-tests).
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.4, p. 393
2. What is CMOS?
   Comparative MOS: listeners hear the same sentence from two systems and give a preference, from −3 (much worse than the reference) to +3 (better), over many sentences in random order.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.4, p. 394
3. What automatic metrics complement human TTS evaluation?
   Run the output through ASR and compute word error rate (robustness/intelligibility), and use a speaker verification system to score similarity between the output voice and the enrolled voice.
   • Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 17, §17.4, p. 394