learn.ifs.name

Automatic Speech Recognition

Jurafsky & Martin, Speech and Language Processing, 3rd ed. draft, ch. 16

16 cards in “16.6 Word error rate” · whole deck

16.6 Word error rate

  1. What is the word error rate (WER) based on?
    How much the recognizer's hypothesized word string differs from a reference transcription, via the minimum edit distance in words (substitutions, insertions, deletions).
  2. WER formula?
    \(\mathrm{WER} = 100 \times \frac{\text{Insertions} + \text{Substitutions} + \text{Deletions}}{\text{Total words in correct transcript}}\)
  3. Can WER exceed 100%? Why?
    Yes, because insertions are counted: the hypothesis can contain more errors than there are reference words.
  4. Compute WER: 6 substitutions, 3 insertions, 1 deletion, 13 reference words.
    100 × (6 + 3 + 1)/13 = 76.9%.
  5. What is sclite?
    The standard free NIST script for computing WER: aligns reference and hypothesis sentences, computes WER, and supports error analysis.
  6. What extra analyses does sclite provide?
    Confusion matrices of commonly misrecognized words, statistics on frequently inserted/deleted words, error rates by speaker, and sentence error rate.
  7. What is sentence error rate?
    The percentage of sentences with at least one word error.
  8. List standard English text normalization rules before computing WER.
    (1) Remove metalanguage in brackets. (2) Remove or standardize filled pauses (uh, um, err). (3) Standardize contractions (I'm / I am). (4) Normalize non-standard words ($100 → one hundred dollars). (5) Unify US and UK spellings.
  9. What is the standard significance test for comparing WERs?
    The Matched-Pair Sentence Segment Word Error (MAPSSWE) test (Gillick & Cox, 1989).
  10. How does MAPSSWE work?
    A parametric test on the difference in word error counts between two systems, per segment, averaged across many segments.
  11. How are segments defined in MAPSSWE, and why?
    As regions bounded on both sides by words both recognizers got correct (or turn/utterance boundaries), so errors in different segments are roughly independent (ASR uses trigram-like LMs). Many short segments help justify normality and maximize power.
  12. MAPSSWE: what is Zᵢ?
    \(Z_i = N_A^i - N_B^i\): the difference in the number of errors systems A and B make on segment i.
  13. MAPSSWE mean and variance estimates?
    \(\hat\mu_z = \sum_i Z_i / n\); \(\sigma_z^2 = \frac{1}{n-1}\sum_{i=1}^{n}(Z_i-\mu_z)^2\)
  14. MAPSSWE test statistic W and decision rule?
    \(W = \frac{\hat\mu_z}{\sigma_z/\sqrt{n}}\). For n > 50, W is approximately standard normal. Reject H₀: μ_z = 0 if 2·P(Z ≥ |w|) ≤ 0.05 (two-tailed) or P(Z ≥ |w|) ≤ 0.05 (one-tailed).
  15. Why is McNemar's test unsuitable for continuous ASR?
    It requires independent errors, but in continuous speech errors on a word strongly depend on errors on neighboring words.
  16. What's a known weakness of WER as a metric?
    It weights every word equally; content words like “Tuesday” arguably matter more than function words like “a” or “of”, but no agreed alternative works across all applications.

Source