From frames to audio features

KTS eats feature vectors. A video gave us one vector per image; audio has no images, so we build the vectors from the raw sound. This chapter shows, in phases, how a waveform becomes the (n, d) array KTS expects, and why the features we pick make speech, music, silence, and noise look obviously different.

Phase 0: what a waveform actually is

Recorded sound is air pressure measured many thousands of times per second. Each measurement is one number (a sample); the sample rate sr is how many we take per second (16,000 Hz is plenty for speech). A 25-second clip at 16 kHz is just an array of $25 \times 16000 = 400{,}000$ numbers.

amplitude
   +1 ┤      ╭╮        ╭─╮
    0 ┤─╮╭╮╭─╯╰╮╭──╮╭─╯ ╰╮╭─   ...  (400,000 of these)
   -1 ┤  ╰╯╰╯  ╰╯  ╰╯    ╰╯
      └────────────────── time →

That raw array is too long and too low-level to hand to KTS directly (400k "frames" of one number each). We compress it into a shorter sequence of richer descriptors.

Phase 1: framing, chop the stream into short frames

Sound changes slowly compared to its sample rate, so we analyse it in short frames, typically 20 to 50 ms, that overlap by a fixed hop so we do not miss events on a boundary. With a 50 ms window and a 25 ms hop, our 25-second clip becomes about 1000 frames.

def frame_signal(wave, sr, win_ms=50.0, hop_ms=25.0):
    win = int(sr * win_ms / 1000)        # 800 samples
    hop = int(sr * hop_ms / 1000)        # 400 samples
    n = 1 + (wave.size - win) // hop
    idx = np.arange(win)[None, :] + hop * np.arange(n)[:, None]
    return wave[idx], win, hop            # (n_frames, win)

Don't be confused: "frame" here vs. in the video book. In the video chapters a frame was one whole image. In audio a frame is one short time window of the waveform. Same word, same role for KTS (it is the unit we describe with a vector and segment between), different physical thing.

Phase 2: describe each frame with a few numbers

Each frame is still hundreds of raw samples. We summarise it with a handful of features that capture how it sounds. Five classic, model-free ones are enough to tell our four content types apart:

The frequency-domain features come from the FFT, a standard transform that tells you which pitches are present in a frame. We do not derive the FFT here; NumPy gives it to us as np.fft.rfft. The point is that energy, ZCR, and three spectral numbers form a 5-dimensional description of each frame:

spec   = np.abs(np.fft.rfft(frame * np.hanning(win)))   # magnitude spectrum
power  = spec ** 2
freqs  = np.fft.rfftfreq(win, d=1.0 / sr)
centroid = (freqs * power).sum() / power.sum()           # one of the five

So our audio is now exactly what KTS wants:

$$ x_1, x_2, \dots, x_n \in \mathbb{R}^5 \qquad (\text{one 5-vector per frame}). $$

Phase 3: do the features really separate the content?

This is the test that matters: if speech, music, silence, and noise produce visibly different feature vectors, KTS will find the boundaries between them. Averaging each feature over each known region of the lab clip (Chapter 15) gives:

per-region mean features:
region     log_energy        zcr   centroid  bandwidth   flatness
silence        -6.178      0.498   3978.086   2297.462      0.558
speech         -2.212      0.173   1006.251    878.143      0.127
silence        -6.172      0.504   3980.467   2290.049      0.557
music          -1.409      0.037    279.154    106.577      0.001
applause       -2.446      0.499   3973.502   2304.736      0.560
speech         -2.210      0.175   1053.582    892.559      0.126

Read it as a table of signatures. Every content type has its own:

• silence is by far the quietest (log_energy ≈ -6.2); its "spectrum" is just faint hiss, so flatness is high and the centroid is bright but meaningless.
• speech has moderate energy, low ZCR (0.17), and low flatness (0.13): voiced and tonal. This is the bite-worthy signature.
• music is loudest, with near-zero ZCR and flatness near 0 (a purely tonal chord).
• applause is about as loud as speech but has flatness ≈ 0.56 and ZCR ≈ 0.50, the clear noise signature.

Notice that the two speech regions (rows 2 and 6) have almost identical signatures even though one is pitched higher. That is what we want: same content type means same signature, so KTS keeps each one as a coherent segment.

Phase 4: one last step, standardize

The five features live on very different scales (energy near -6, centroid near 4000). A kernel would let the centroid drown out everything else. So before building the kernel we z-score each column (subtract its mean, divide by its standard deviation) so all five count equally:

def zscore(features):
    mu = features.mean(axis=0, keepdims=True)
    sd = features.std(axis=0, keepdims=True)
    return (features - mu) / np.maximum(sd, 1e-8)

Don't be confused: standardizing vs. the cosine kernel. The cosine kernel rescales each frame vector to unit length (direction only). Z-scoring rescales each feature column so no single feature dominates. They solve different problems; for these mixed-scale audio features, z-scoring first is what makes the kernel behave.

That is the whole front end: waveform, then frames, then five features, then standardize. The output is a clean (n, 5) array, and from here on it is just KTS. Next we survey the common ways people turn these segments into sound bites. 👉