Phonetics and Speech Feature Extraction

One symbol represents a whole word. The earliest known writing systems (Sumerian, Chinese, Mayan) were mainly logographic.

Some symbols were used purely for their sound. The cuneiform sign pronounced ba meaning “ration” could also function simply as the sound /ba/. Early Chinese oracle-bone characters also contain phonetic elements.

Syllabic (Japanese hiragana), alphabetic (Roman alphabet), consonantal (Semitic writing systems).

A West Semitic script, presumed to have been modified by Western Semitic mercenaries from a cursive form of Egyptian hieroglyphs.

From a cursive form of Chinese phonetic characters, which were used in Chinese to phonetically represent Sanskrit in Buddhist scriptures that came to China in the Tang dynasty.

Speech recognition: transcribing waveforms into text.
Text-to-speech (TTS): converting text into waveforms.

The study of the speech sounds used in the world's languages: how they are produced in the vocal tract, how they are realized acoustically, and how they can be digitized and processed.

A speech sound, represented with a symbol adapted from the Roman alphabet. A word's pronunciation is represented as a string of phones.

The International Phonetic Alphabet: the evolving standard for transcribing the world's languages, first developed in 1888.

A simple phonetic alphabet (Shoup, 1980) using ASCII symbols to represent an American-English subset of the IPA.

Relatively opaque: one letter can represent very different sounds. c is [k] in cougar but [s] in cell. Spanish is much more transparent.

c (cougar), x (fox [f aa k s]), ck (jackal), cc (raccoon).

English pronunciations for 160,595 wordforms, with syllabification, stress, and morphological and part-of-speech information.

Open-source; about 134,000 wordforms.

About 110,000 words, fine-grained; gives syllabifications, stress, and pronunciations for dozens of English dialects. Free for research.

Egyptian Arabic, Dutch, English, German, Japanese, Korean, Mandarin, Spanish.

A collection of waveforms hand-labeled with the corresponding string of phones.

Texas Instruments, MIT and SRI. 6300 read sentences: 10 sentences each from 630 speakers, drawn from a set of 2342 sentences (some chosen for dialect shibboleths, others to maximize diphone coverage).

Hand-labeled phone sequences were automatically aligned with the wavefile, and the automatic phone boundaries were then manually hand-corrected.

A transcription in which each phone is associated with a start and end time in the waveform.

Palatalization of [d] in had, unreleased final stop in dark, glottalization of final [t] in suit to [q], and the flap of [t] in water.

3.5 hours of sentences from the Switchboard corpus, time-aligned at the syllable level.

Vowel reduction (they're, of), coda deletion (kind, right), and resyllabification (the [v] of of becomes the onset of in).

A phonetically transcribed corpus of spontaneous American speech: about 300,000 words from 40 talkers.

The Kiel corpus (German); Mandarin corpora from the Chinese Academy of Social Sciences.

IPA [ŋ] — as in baking [b ey k ix ng]

IPA [θ] — as in thick

IPA [ð] — as in those

IPA [ʃ] — as in squash

IPA [ʒ] — as in ambrosia

IPA [tʃ] — as in cherry

IPA [dʒ] — as in jar

IPA [j] — as in yellow

IPA [i] — as in lily (2nd vowel) [l ih l iy]

IPA [ɪ] — as in lily (1st vowel)

IPA [eɪ] — as in daisy

IPA [ɛ] — as in pen

IPA [æ] — as in aster

IPA [ɑ] — as in poppy

IPA [ɔ] — as in orchid

IPA [ʊ] — as in wood

IPA [oʊ] — as in lotus

IPA [u] — as in tulip

IPA [ʌ] — as in butter

IPA [ɝ] — as in bird

IPA [aɪ] — as in iris

IPA [aʊ] — as in flower

IPA [ɔɪ] — as in soil

IPA [ə (schwa)] — as in pita (2nd vowel)

IPA [ɾ (tap/flap)] — as in lotus [l ow dx ax s], water

IPA [ʔ (glottal stop)] — as in glottalized final [t] of suit

The study of how phones are produced as the organs in the mouth, throat and nose modify the airflow from the lungs.

Expelled from the lungs through the trachea (windpipe), through the larynx, then out the mouth or nose.

The “Adam's apple” or voice box; it contains the vocal folds.

Vocal folds (non-technically “vocal cords”): two small folds of muscle in the larynx that can move together or apart. The glottis is the space between them.

When they are close together (but not tightly closed) as air passes through. If far apart, they don't vibrate.

Voiced: made with vocal folds together and vibrating — [b], [d], [g], [v], [z], all English vowels.
Unvoiced (voiceless): no vocal fold vibration — [p], [t], [k], [f], [s].

The area above the trachea; it consists of the oral tract and the nasal tract.

Sounds made by air passing through the nose; they use both the oral and nasal tracts as resonating cavities. English [m], [n], [ng].

Consonants: made by restricting or blocking airflow; can be voiced or unvoiced.
Vowels: less obstruction, usually voiced, generally louder and longer-lasting.

Sounds like [y] and [w] with properties of both: voiced like vowels, but short and less syllabic like consonants.

A consonant's point of maximum restriction of airflow.

Main restriction formed by the two lips coming together: [p] (possum), [b] (bear), [m] (marmot).

Bottom lip pressed against the upper teeth, air flowing through the space in the upper teeth: [f], [v].

Tongue against the teeth (behind the teeth, tip slightly between them): [th] (thing), [dh] (though).

The portion of the roof of the mouth just behind the upper teeth. [s], [z], [t], [d] are made with the tongue tip against it.

Both dental and alveolar places of articulation.

Blade of the tongue against the rising back of the alveolar ridge: [sh] (shrimp), [ch] (china), [zh] (Asian), [jh] (jar).

Front of the tongue placed up close to the palate (roof of the mouth), as in yak.

The velum (soft palate) is a movable muscular flap at the very back of the roof of the mouth. Velars [k] (cuckoo), [g] (goose), [ng] (kingfisher) press the back of the tongue against it.

By closing the glottis (bringing the vocal folds together).

How the airflow restriction is made (e.g. complete stoppage vs partial blockage). Place + manner is usually enough to uniquely identify a consonant.

A consonant where airflow is completely blocked for a short time, then explosively released. Closure = period of blockage; release = the explosion. Also called plosives.

Voiced: [b], [d], [g]. Unvoiced: [p], [t], [k].

By lowering the velum so air passes into the nasal cavity: [n], [m], [ng].

Airflow is constricted but not cut off completely; the turbulent airflow produces a characteristic “hissing” sound.

Labiodental [f], [v]; dental [th], [dh]; alveolar [s], [z]; palato-alveolar [sh], [zh].

Tongue at the back of the alveolar ridge, forcing air through a groove formed in the tongue.

The higher-pitched fricatives: in English [s], [z], [sh], [zh].

Stops followed immediately by fricatives: [ch] (chicken), [jh] (giraffe).

The two articulators are close together but not close enough to cause turbulent airflow: [y] (yellow), [w] (wood), [r], [l].

(1) Tip of the tongue extended and close to the palate, or (2) the whole tongue bunched up near the palate.

It's made with the tongue tip against the alveolar ridge or teeth, with one or both sides of the tongue lowered so air flows over them.

[dx]: a quick motion of the tongue against the alveolar ridge, e.g. the middle consonant of lotus [l ow dx ax s] in American English. Many U.K. speakers use [t] instead.

Vowel height, frontness/backness (location of the tongue's highest point), and lip rounding.

Roughly with the height of the highest part of the tongue — though it's really a more accurate reflection of acoustic facts than of actual tongue position.

Front: tongue raised toward the front (e.g. [iy]). Back: tongue raised toward the back (e.g. [uw]).

[ih]. Both are front vowels.

Classified by the height of the highest point of the tongue: comparatively high, mid, or low.

A vowel in which the tongue position changes markedly during its production. English is particularly rich in diphthongs (e.g. [ey], [ay], [aw], [oy], [ow]).

Diphthongs are represented as a path; monophthongs as a point.

Vowels pronounced with the lips rounded (as for whistling): [uw], [ao], [ow].

A vowel-like (sonorant) sound together with the surrounding consonants most closely associated with it. E.g. dog = 1 syllable; catnip = [k ae t] + [n ih p].

Nucleus: the vowel at the core of a syllable. Onset: initial consonants (if any). Coda: optional consonant(s) following the nucleus.

An onset with more than one consonant, e.g. strike [s t r ay k].

The nucleus plus the coda.

ham: onset [h], nucleus [ae], coda [m].
green: onset [g r], nucleus [iy], coda [n].
eggs: no onset; nucleus [eh], coda [g z].

The task of automatically breaking a word into syllables.

Constraints on which phones can follow each other in a language. E.g. [zdr] cannot be an English syllable onset. Can be modeled with a language model or finite-state model of phone sequences.

The study of intonational and rhythmic aspects of language, especially the use of F0, energy and duration to convey pragmatic, affective, or conversation-interactional meaning.

Energy → loudness. F0 → pitch.

Marking discourse structure (e.g. statements vs questions), marking saliency of words/phrases, paralinguistic functions (emotion like happiness, surprise, anger), and managing conversational turn-taking.

The study of verse metrical structure.

Perceptual salience to the listener. Made by saying it louder, slower (longer duration), or with higher/more variable F0.

A linguistic marker representing prominence; prominent words/syllables are said to bear a pitch accent.

A dictionary property of a word: the syllable that will be louder or longer if the word is accented. Pitch accent must be realized on the lexically stressed syllable (e.g. surPRISED, not SURprised).

Vowels marked 0 (unstressed) or 1 (stressed): counter [K AW1 N T ER0], table [T EY1 B AH0 L].

Noun: [K AA1 N T EH0 N T] (first syllable stressed). Adjective: [K AA0 N T EH1 N T] (second syllable stressed).

Unstressed vowels weakened further, with an incomplete articulatory gesture. Schwa [ax] is the most common, e.g. 2nd vowel of parakeet [p ae r ax k iy t].

No. Any vowel (diphthongs especially) can keep full quality unstressed, e.g. [iy] stressed in eat, unstressed in carry [k ae r iy].

Accented, stressed, full vowel, reduced vowel.

The idea that an utterance has a prosodic phrase structure (words group together, with breaks/disjunctures), analogous to syntactic phrase structure.

Two main intonation phrases, split at the comma. Within the first, lesser boundaries give intermediate phrases: I wanted | to go | to London.

They are often correlated with syntactic constituents.

Sequence models taking raw text or text annotated with features (e.g. parse trees) make a break/no-break decision at each word boundary; trained on data like the Boston University Radio News Corpus.

The rise and fall of its F0 over time.

Question rise: final F0 rise indicating a yes-no question. Final fall: final F0 drop indicating a declarative.

A short F0 rise after each noun in a comma-separated list.

Contradiction and surprise.

Tone and Break Indices: a theory/transcription system of intonation (Silverman et al., 1992). Each word may bear one of 5 pitch accents; each intonational phrase ends in one of 4 boundary tones. Versions exist for many languages.

H* peak accent; L* low accent; L*+H scooped accent; L+H* rising peak accent; H+!H* step down.

L-L% final fall (declarative); L-H% continuation rise; H-H% question rise (yes-no question); H-L% final level plateau.

The tone ending each intonational phrase, representing utterance-final aspects of tune.

\(y = A \sin(2\pi f t)\) — A = amplitude, f = frequency.

The number of times per second a wave repeats (cycles per second), measured in hertz (Hz).

Its maximum value on the y-axis.

The time for one cycle to complete: \(T = 1/f\). A 10 Hz wave has T = 0.1 s.

Changes in air pressure over time: compression or rarefaction of air molecules (e.g. at a microphone or eardrum).

Compression: higher-than-normal air pressure. Rarefaction: lower-than-normal (uncompressed) air pressure.

Sampling and quantization.

Sampling: measuring the signal's amplitude at particular times. Sampling rate: number of samples per second.

One for the positive part and one for the negative part; with fewer, the wave's frequency is completely missed.

The maximum frequency measurable at a given sampling rate: half the sample rate.

20,000 Hz, since most speech information is below 10,000 Hz.

The telephone network transmits only frequencies below 4,000 Hz (e.g. Switchboard).

16,000 Hz.

Representing real-valued amplitudes as integers; the gap between integers is a minimum granularity (quantum size), and values closer than it are represented identically.

8-bit: −128 to 127. 16-bit: −32768 to 32767.

Telephone: 8 kHz, 8-bit samples. Microphone: 16 kHz, 16-bit samples.

Separate signals, e.g. for stereo or two-party conversations; stored in the same file or separate files.

A log compression format for telephone speech. Human hearing is more sensitive at small intensities, so the log represents small values faithfully at the cost of more error on large values.

\(F(x) = \frac{\mathrm{sgn}(x)\log(1+\mu|x|)}{\log(1+\mu)},\ -1 \le x \le 1\), with µ = 255 for 8 bits.

Linear (unlogged, uncompressed) sample values. PCM = pulse code modulation.

The .wav format is a subset of Microsoft's RIFF, a general format of nested chunks of data and control info. A simple .wav has a 44-byte header (with format chunk) before the data chunk.

Microsoft .wav, Apple AIFF (both with headers), and headerless “raw” files.

10 / 0.03875 ≈ 258 Hz.

Vocal fold vibration: each major amplitude peak corresponds to an opening of the vocal folds.

The frequency of vocal fold vibration, i.e. the frequency of the complex wave.

A plot of F0 over time. Automatic pitch tracking counts pulses in voiced regions, so it fails where there is no voicing or insufficient sound.

Air pressure variation, in Pascals (Pa). Positive = above atmospheric pressure, zero = normal, negative = rarefaction.

Positive and negative values would mostly cancel. RMS squares each value before averaging, then takes the square root.

\(\text{RMS} = \sqrt{\frac{1}{N}\sum_{i=1}^{N} x_i^2}\)

\(\text{Power} = \frac{1}{N}\sum_{i=1}^{N} x_i^2\) — related to the square of the amplitude.

Power normalized to the human auditory threshold, in dB: \(\text{Intensity} = 10\log_{10}\frac{1}{NP_0}\sum_{i=1}^{N} x_i^2\), with \(P_0 = 2\times10^{-5}\) Pa.

At each vowel, with an especially high peak on the word long.

The perceptual correlate (mental sensation) of fundamental frequency.

Most accurate and roughly linear between 100 and 1000 Hz; above 1000 Hz, logarithmic (high-frequency differences are compressed).

A unit of pitch: pairs of sounds perceptually equidistant in pitch are separated by an equal number of mels (Stevens et al. 1937; Stevens & Volkmann 1940).

\(m = 1127 \ln\left(1 + \frac{f}{700}\right)\)

The perceptual correlate of power. Nonlinear because (1) the ear has greater resolution at low power, and (2) perceived loudness also depends on frequency.

Algorithms for automatically extracting F0 (a slight abuse of terminology).

Correlates the signal with itself at various offsets; the offset with the highest correlation gives the signal's period.

Praat (Boersma & Weenink): augmented autocorrelation pitch tracking, spectral and formant analysis, and a scripting language.

They are voiced (regular amplitude peaks), relatively long, and relatively loud.

A period of silence or near-silence (closure) followed by a slight burst of amplitude (release).

A noisy, irregular waveform from turbulent air, unlike the regular voicing of vowels.

Every complex wave can be represented as a sum of many sine waves of different frequencies.

A representation of each of a signal's frequency components and their amplitudes (frequency on x-axis, amplitude on y-axis).

Two spikes, at 10 Hz and 100 Hz.

A complex wave ≈234 Hz (10 cycles in 0.0427 s), a smaller wave ≈936 Hz (4× per cycle), and a tiny wave ≈1872 Hz.

Around 930 Hz, 1860 Hz, and 3020 Hz.

A logarithmic measure of amplitude.

Spectral peaks are characteristic “signatures” of different phones — like spectral lines identifying chemical elements in stars.

The inner ear computes a spectrum of the incoming waveform.

A visualization of how a waveform's frequencies change over time. x = time, y = frequency (Hz), darkness = amplitude.

A spectrum shows frequency components at one point in time; a spectrogram shows them over time (a collection of spectra placed end to end).

A frequency band particularly amplified by the vocal tract; appears as a dark bar (spectral peak) in a spectrogram.

F1: low for [ih] (~470 Hz), higher for [ae] and [uh] (~800 Hz).
F2: highest for [ih], middle for [ae], lowest for [uh].

[iy]: F1 = 540 Hz (much lower than [ae]), F2 = 2581 Hz (much higher than [ae]).

F1 and F2 largely determine vowel identity (though they vary by speaker). Higher formants reflect general characteristics of the speaker's vocal tract.

Nasals [n], [m], [ng] and liquids [l], [r].

Reduction and coarticulation processes.

Explains a sound's acoustics as pulses from the glottis (the source) shaped by the vocal tract (the filter).

A wave whose frequency is a multiple of the fundamental. A 115 Hz glottal vibration has harmonics at 230, 345, 460 Hz… each generally weaker.

Any cavity (like a tube) resonates at frequencies determined by its shape, amplifying some frequencies and damping others.

Moving the tongue and articulators changes the vocal tract's shape, so different harmonics of the same fundamental are amplified. Formants are where the tract amplifies harmonics.

[iy]: 268 / 2416 Hz. [ae]: 903 / 1695 Hz. [uw]: 295 / 817 Hz.

A vector representing the information in a small time window of the signal; speech processing often first converts the waveform into a sequence of these.

The raw waveform (processed by convolutional networks) or a higher-level representation such as the log mel spectrum.

No. E.g. if testing on 8 kHz Switchboard, downsample training data to 8 kHz; if any training corpus is telephone speech, downsample all wideband corpora to 8 kHz.

Yes, higher sampling rates produce higher ASR accuracy.

The sample at time index n in the digitized, quantized waveform.

Stationary: statistical properties constant within a region. Speech is non-stationary overall, but roughly stationary within a small window.

A function that is non-zero inside a region and zero elsewhere; run across the signal and multiplied by it to produce a windowed waveform.

The speech extracted from each window.

Window (frame) size in ms, frame stride (shift/offset) between windows, and window shape.

25 ms windows with a 10 ms stride.

\(y[n] = w[n]\,s[n]\)

It abruptly cuts off the signal at its boundaries, creating discontinuities that cause problems in Fourier analysis.

Shrinks signal values toward zero at the window boundaries, avoiding discontinuities.

Rectangular: \(w[n]=1\) for \(0\le n\le L-1\), else 0.
Hamming: \(w[n]=0.54-0.46\cos\left(\frac{2\pi n}{L}\right)\) for \(0\le n\le L-1\), else 0.

Discrete Fourier transform: extracts spectral information for discrete frequency bands from a sampled signal. Input: windowed signal x[n]…x[m]. Output: for each of N bands, a complex number X[k] giving magnitude and phase.

\(e^{j\theta} = \cos\theta + j\sin\theta\), with j the imaginary unit.

\(X[k] = \sum_{n=0}^{N-1} x[n]\, e^{-j\frac{2\pi}{N}kn}\)

Fast Fourier transform: a very efficient algorithm for computing the DFT; only works for N that are powers of 2.

Human hearing is less sensitive at high frequencies. Low-frequency info (formants) is crucial for vowels/nasals; high-frequency info (stop bursts, fricative noise) is less crucial. Modeling this improves ASR.

\(\mathrm{mel}(f) = 1127\ln\left(1+\frac{f}{700}\right)\)

A bank of triangular filters spaced logarithmically along the mel scale (fine resolution at low, coarse at high frequencies), multiplied by the spectrum to get a mel spectrum (Davis & Mermelstein, 1980).

(1) Human response to signal level is logarithmic: less sensitive to small differences at high amplitudes. (2) Makes features less sensitive to input power variations, e.g. the mouth moving relative to the microphone.

Each scalar output of one filter: the log energy of one mel-spaced frequency band. A frame gives a vector of, say, 80 or 128 channels.

Scaled to between −1 and 1 with zero mean across the entire pretraining dataset.

Sample & quantize → window (e.g. 25 ms Hamming, 10 ms stride) → DFT/FFT → mel filter bank → log → normalize.

Mel frequency cepstral coefficients.

12 cepstral coefficients + 1 energy; 12 delta cepstral + 1 delta energy; 12 double-delta cepstral + 1 double-delta energy.

It separates source and filter. Glottal source characteristics (F0, pulse details) don't distinguish phones; the vocal tract filter (articulator positions) does.

Separate them so features show only the vocal tract filter.

The spectrum of the log of the spectrum.

High-frequency ripple: the harmonics of F0 (the source). Low-frequency components: the spectral envelope / formant structure (the filter).

Reversing the first four letters of spectrum.

Samples: taking the spectrum of the log spectrum leaves the frequency domain and returns to the time domain.

The peak corresponds to F0 (glottal pulse). Lower values represent the vocal tract filter.

Lower cepstral values for phones; higher cepstral values for pitch.

The first 12; they represent only the vocal tract filter, cleanly separated from the glottal source.

Their variances tend to be uncorrelated, unlike spectral coefficients at different frequency bands.

The inverse DFT of the log magnitude of the DFT of a signal.

Frame energy: \(\text{Energy} = \sum_{t=t_1}^{t_2} x^2[t]\), the sum over time of the samples' power.

It cues phone identity, e.g. vowels and sibilants have more energy than stops.

Delta (velocity): change between frames in each of the 13 features. Double delta (acceleration): change between frames in the deltas.

Changes over time — like formant slopes at transitions, or stop closure → burst — are useful cues for phone identity.

Simply by frame differences, but commonly by fitting a polynomial and taking its first and second derivatives.

Linguists of India, 800–150 B.C.: place and manner of articulation, the glottal mechanism of voicing, and assimilation. Europe caught up only in the late 19th century.

By Plato's Theaetetus and Cratylus, they distinguished vowels from consonants and stops from continuants. The Stoics developed the syllable and knew of phonotactic constraints.

Used the concept of the phoneme and proposed a phonemic writing system for Icelandic with diacritics for length and nasality (unpublished until 1818).

Henry Sweet, Handbook of Phonetics (1877): essentially proposed the phoneme, devised a transcription alphabet, distinguished broad vs narrow transcription, and made the first scientific recordings for phonetics.

Henry Higgins (George Bernard Shaw).

Polish scholar Baudouin de Courtenay (1894).

Spectrograph: Koenig et al. 1946. Source-filter: Fant 1960 (and Stevens et al.). F1×F2 space: Peterson & Barney 1952. Duration/intensity as stress cues: Fry 1955.

FFT: Cooley & Tukey 1965. Cepstral processing: Oppenheim et al. 1968. LPC for speech coding: Atal & Hanauer 1971.

[th r iy]

[s ih ng]

[ay z]

[s t ah d iy]

[dh ow]

[p l ae n ih ng]

[s l ay t]

dark [d aa r k]; suit [s uw t]; greasy [g r iy s iy]; wash [w aa sh]; water [w aa dx axr] (American)