Reading utterance endings from audio

February 2, 2020

A small Python 2 tool that finds pauses in a mono WAV file and measures how pitch moves before each one. A rising ending often makes a yes/no question audible, so a list of endings sorted by rise is a quick way to find them in generated speech.

Problem

A text-to-speech model can render a question with a rise, a flat line, or a fall. Checking that by ear across many clips and model versions is slow. The input here is only the audio: no transcript, no alignment, no timing metadata. The endings and their contours have to come from the signal.

A rise is question-like, not proof of a question. Some questions end low and some statements rise, and the language, dialect and delivery all matter. The output is a timestamp, a measured contour and a short clip to listen to, not a punctuation label.

Approach

Boundaries. Root-mean-square (RMS) energy is measured in 20 ms frames every 10 ms. A threshold set from the recording's own level separates speech from silence. Gaps up to 80 ms are filled so a stop consonant does not split a word, and speech regions closer than the minimum pause (0.2 s) are merged. The end of each region is a candidate ending (Figure 1). A pause inside a sentence has the same energy shape as a pause between sentences, so both are kept.

RMS energy 1 1.27 s 2 3.31 s 3 5.16 s 4 7.14 s
Figure 1. RMS energy of a clip with four sentences and three scripted 600 ms silences. The dashed line is the speech threshold and the bars mark the detected regions. Real tool output.

Pitch. The fundamental frequency (F0) is estimated in 60 ms windows every 10 ms, on audio averaged down to about 4 kHz. Each window is Hann-weighted, and its normalized autocorrelation is computed over lags for 70 to 450 Hz. Among the interior autocorrelation peaks with a strength of at least 0.55, the best-scoring one gives the period. The score is the strength minus a small penalty that favors shorter lags and a small penalty for large jumps from the previous frame, which reduces octave errors. Parabolic interpolation refines the chosen peak between integer lags. Frames without a strong peak are unvoiced and left blank.

Ending. The voiced frames in the last 400 ms before a boundary are split into thirds, and the median F0 of the first, middle and last third is compared in semitones:

terminal_change = 12 * log2(late_median_f0 / early_median_f0)

A change of at least +1.25 st is a rise, at most −1.25 st a fall, and anything between is level. A drop of at least 1 st from the first third to the middle followed by a rise of at least 1 st to the last third is a fall–rise. With fewer than 8 voiced frames, less than 120 ms of voiced span, or a last voiced frame more than 200 ms before the boundary, the shape is unknown. Only the part of a contour inside that window is summarized. The full track is kept for every ending, since one number cannot tell a steady rise from a dip followed by a rise.

Implementation

The script uses only the Python 2.7 standard library (wave, array, csv, json). Input is mono, 16-bit PCM WAV at any sample rate. The core of the pitch estimate is the normalized autocorrelation over the allowed lags, followed by the interpolation of each peak:

for lag in range(first_lag, last_lag + 1):
    dot = 0.0
    for i in range(count - lag):
        dot += windowed[i] * windowed[i + lag]
    left_energy = squares[count - lag]
    right_energy = squares[count] - squares[lag]
    strength = dot / math.sqrt(max(left_energy * right_energy, 1e-12))
    candidates.append((lag, strength))

curvature = before - 2.0 * strength + after
if curvature < 0:
    shift = 0.5 * (before - after) / curvature
    lag += shift

Without interpolation the F0 values fall on a grid of integer lags. At 4 kHz and 300 Hz, neighboring lags differ by about 1.2 st, which is the size of the rise and fall thresholds.

For each ending the tool writes a clip of the last 1.5 s plus 0.2 s after the boundary, an SVG of energy and F0, and a row in endings.csv with the start and end times, the following pause, the terminal change, the voiced fraction, and the shape. endings.json adds the full F0 track.

Results

Known contours. A tone with eight harmonics at 150 Hz carries a contour over its last 400 ms: level, a rise to +7 st, a fall to −7 st, and a dip to −4 st followed by a rise to +3 st. The tracked F0 stays within 0.22 st of the true F0 in every case (Figure 2). Table 1 compares the terminal change with the value the same summary gives on the true F0.

Level +0.0 st Rise +4.5 st Fall -4.5 st Fall–rise +1.9 st
Figure 2. Tracked F0 (solid) and true F0 (dashed) over the last second, in semitones from 150 Hz. The two overlap. Real tool output on synthetic audio.

Table 1. Terminal change on synthetic contours.

ContourTrue F0Tracked F0Shape
Level+0.00 st+0.00 stlevel
Rise+4.55 st+4.55 strise
Fall-4.55 st-4.54 stfall
Fall–rise+1.90 st+1.91 stfall-rise

Against Praat. Real speech has no true F0 to compare with, so the reference is Praat's pitch tracker (version 6.1.38 through the parselmouth bindings, same 70 to 450 Hz range, 10 ms step). The test set is 120 clips from the macOS say command: four sentence types, six sentences each, in five voices (Samantha, Daniel, Karen, Moira and Tessa). Praat's track is cut to the frames the tool can measure, and the same ending summary is applied to both.

-12 -12 0 0 +12 +12 Praat terminal change (st) Tool (st)
Figure 3. Terminal change from the tool against Praat for 117 clips. The dashed line is equality. Real tool output.

Table 2. Agreement with Praat on 120 clips.

MeasureValue
F0 per frame, median difference0.06 st
F0 per frame, 90th percentile0.22 st
Frames within 3 st of Praat99.8% of 9504
Terminal change, median difference0.19 st
Terminal change, 90th percentile0.66 st
Same shape label109 of 120

Frame by frame the two trackers agree closely. Of the 11 clips with a different shape label, 6 differ by less than 1 st and sit on either side of a label threshold. The other 5 are all in the Daniel voice. In the largest miss, F0 falls to 72 to 80 Hz in the last 90 ms. The tool reports a harmonic near 245 Hz there while Praat tracks the fundamental, which gives a terminal change of +15.0 st against −4.1 st. Lowering --pitch-floor to 50 Hz raises the number of matching labels from 109 to 113.

By sentence type. The check on the sentence types is whether the tool separates endings that should fall from endings that should rise. Statements and wh-questions end in a fall in nearly every clip, with no voice rising on a wh-question (Figure 4, Table 3). Yes/no questions and statements with a question mark rise in some voices and not in others.

Statement Wh-question Yes/no Rising statement -12 -6 0 +6 +12 Terminal change (st)
Figure 4. Terminal change of each clip by sentence type, one dot per clip, offset vertically by voice. The dashed lines are the ±1.25 st thresholds. Real tool output.

Table 3. Clips with a rising or fall–rise ending, by sentence type. Each type has 30 clips.

Sentence typeToolPraat
Statement1 of 300 of 30
Wh-question0 of 300 of 30
Yes/no question19 of 3020 of 30
Rising statement18 of 3017 of 30

The voices split cleanly (Table 4). Samantha and Moira render every yes/no question and every question-marked statement with a rise. Daniel, Karen and Tessa do so for only some of them. The tool and Praat count nearly the same clips in every cell, so the difference between voices is in the audio.

Table 4. Clips with a rising or fall–rise ending, by voice (tool / Praat). Six clips per cell.

VoiceYes/no questionRising statement
Samantha6 / 66 / 6
Daniel1 / 21 / 1
Karen2 / 23 / 2
Moira6 / 66 / 6
Tessa4 / 42 / 2

Examples. Four sentences in the default macOS voice show what the contours look like (Figure 5). Each clip is below to listen to.

Yes/no fall-rise Wh-question fall Statement fall Rising rise
Figure 5. Tracked F0 over the last second of each ending, in semitones from 234 Hz. Gaps are unvoiced frames. Real tool output.
Yes/no question: Are you coming to dinner tonight?
Wh-question: Where did you put the keys?
Statement: I put the keys on the table.
Rising statement: You put the keys on the table?

A longer clip joins four sentences with scripted 600 ms silences. The tool finds all four endings, and the silences appear as pauses of 0.70 to 0.76 s (Table 5). The first sentence repeats the statement above and returns the same change.

Table 5. Endings in the four-sentence clip.

EndingEnds atPause afterChangeShape
11.27 s0.71 s-6.8 stfall
23.31 s0.76 s+2.2 stfall-rise
35.16 s0.70 s-8.1 stfall
47.14 send of file-2.7 stfall
Four sentences: I put the keys on the table. Are you coming to dinner tonight? Where did you put the coat? It is on the chair, I think.

Run it

python2 endings.py speech.wav --output out/

The output directory must be empty unless --force is passed. --pitch-floor and --pitch-ceiling set the F0 search range in hertz and should fit the voice. --min-pause and --min-speech set the shortest pause and the shortest speech region in seconds. The script was run with Python 2.7.18 and also runs under Python 3.

Reading