Lab: extract sound bites with KTS
Time to build the thing. This lab takes a waveform and produces ranked sound
bites, reusing kts.py without changes: the same kernel and dynamic program from
the rest of the book. It runs in pure NumPy, with no audio files and no extra
libraries, so every number below is real output you can reproduce.
Run it any time with:
cd books/kts/code
python3 soundbites.py
We will go phase by phase, then list the whole file.
Phase 1: make a clip with known answers
To prove the pipeline works we need ground truth. synthesize_audio builds a
25-second waveform from six labelled regions (silence, speech, silence, music,
applause, speech), so we know exactly where the boundaries should be:
wave, sr, regions = synthesize_audio(sr=16000, seed=0)
clip: 25.0s @ 16000 Hz (400000 samples)
ground-truth regions:
silence 0:00.0 - 0:03.0
speech 0:03.0 - 0:09.0
silence 0:09.0 - 0:12.0
music 0:12.0 - 0:18.0
applause 0:18.0 - 0:21.0
speech 0:21.0 - 0:25.0
Each region is synthesised to sound like its label: speech is voiced harmonics with a 4 Hz syllable envelope, music is a steady three-note chord, applause is broadband noise bursts, and silence is faint hiss. The details are in the file at the end.
Why fake audio? The same reason the video demo uses synthetic data: with known boundaries we can check KTS is correct, not just that it runs. Swap
synthesize_audioforlibrosa.load("clip.wav")(Ch. 14) and the rest is identical.
Phase 2: turn the waveform into features
This is the front end from Chapter 12: frame the signal (50 ms window, 25 ms hop) and describe each frame with five features.
feats, hop_sec = frame_features(wave, sr, win_ms=50.0, hop_ms=25.0)
framed: 999 frames x 5 features (hop 25 ms)
That is 999 frames, each a 5-vector: our $x_1,\dots,x_n \in \mathbb{R}^5$.
Phase 3: segment with KTS (the reused engine)
Standardize the features, build a kernel, and let cpd_auto choose the cuts. This
is the point of the lab: the next three lines are imported from kts.py, so the
audio never touches new segmentation code.
from kts import build_kernel, cpd_auto # <-- the book's core, unchanged
fz = zscore(feats)
K = build_kernel(fz, kind="rbf")
cps, _ = cpd_auto(K, ncp_max=12, vmax=1.0, lmin=8)
KTS found 5 cut points at: 0:03.0, 0:09.0, 0:12.0, 0:18.0, 0:21.0
It nailed all five. Compare to Phase 1's ground truth (3.0, 9.0, 12.0, 18.0, 21.0): every boundary recovered exactly, and the count (5) was chosen
automatically by the penalty, not handed to it. The same cpd_auto that found
video shots just found audio scene changes.
Phase 4: score and rank the segments
Boundaries give six segments; now we rank them as sound bites with the simple, feature-only heuristic from Chapter 13: reward loud, voiced (low flatness), speech-band ZCR, and the right length.
segs = cuts_to_segments(cps, feats.shape[0])
ranked = score_bites(feats, segs, hop_sec)
6 segments, ranked as sound-bite candidates:
rank start end dur energy zcr flat score
------------------------------------------------------------
0 0:21.0 0:25.0 4.0 -2.21 0.18 0.13 0.81
1 0:03.0 0:09.0 6.0 -2.21 0.17 0.13 0.81
2 0:12.0 0:18.0 6.0 -1.41 0.04 0.00 0.80
3 0:18.0 0:21.0 3.0 -2.45 0.50 0.56 0.41
4 0:09.0 0:12.0 3.0 -6.21 0.51 0.56 0.10
5 0:00.0 0:03.0 3.0 -6.21 0.50 0.56 0.10
top bite: 0:21.0 - 0:25.0 (4.0s, score 0.81)
Read the ranking against what we know is in the clip:
- Ranks 0 and 1 are the two speech segments: loud, voiced (
flat ≈ 0.13), with speech-band ZCR. These are the bites a producer wants. - Rank 2 is the music: just as loud, but it is a chord, not a quote, so the speech-ZCR term keeps it a hair below the speech bites.
- Rank 3 is applause: loud-ish but noisy (
flat 0.56), correctly demoted. - Ranks 4 and 5 are the silences: quietest (
energy -6.21), correctly last.
The pipeline went from raw samples to "your best clip is 0:21 to 0:25" with no labels, no transcript, and no model: just features, KTS, and a scoring rule.
Phase 5: take it to real audio
To run this on an actual recording, change only Phase 1:
import librosa # see Chapter 14
wave, sr = librosa.load("interview.wav", sr=16000)
feats, hop_sec = frame_features(wave, sr) # Phase 2 unchanged
# ... Phases 3 and 4 unchanged ...
For longer or higher-quality results, swap layers in from Chapter 14 without touching the KTS core:
- Trim silence first with Silero VAD, so fewer frames keep the $O(n^2)$ cost down.
- Richer features: replace
frame_featureswithlibrosaMFCCs or wav2vec2 embeddings.build_kernelaccepts any(n, d)array. - Smarter ranking: feed the candidate timestamps plus a Whisper transcript to
claude-opus-4-8to pick the most quotable bites (Ch. 14, Layer 4). - Tune the cut count: raise
vmaxfor fewer, longer bites, lower it for more, or pin an exact number withcpd_nonlin(Ch. 5).
The complete script
"""
Sound bites with KTS — from scratch in NumPy.
A "sound bite" is a short, self-contained, salient excerpt of audio (a quotable
sentence, a chorus, a punch line). Finding them is the *audio* sibling of video
shot detection: instead of cutting where the *picture* changes, we cut where the
*sound* changes, then keep the segments that are most "bite-worthy".
The cut-finding is exactly KTS — the same kernel + dynamic-programming machinery
from this book — so this module imports the core straight from ``kts.py``:
build_kernel, calc_scatters, cpd_auto
Pipeline (see the "Lab: extract sound bites with KTS" chapter):
waveform -> frame features -> kernel -> KTS cut points
-> score each segment -> rank the top bites
Everything is pure NumPy and CPU-only. ``synthesize_audio`` fakes a clip with
known regions (silence / speech / music / applause) so the whole pipeline runs
and verifies with no audio files and no extra libraries.
"""
from __future__ import annotations
import numpy as np
from kts import build_kernel, calc_scatters, cpd_auto
# --------------------------------------------------------------------------- #
# 0 — a synthetic clip with known regions (stand-in for a real .wav)
# --------------------------------------------------------------------------- #
def synthesize_audio(sr=16000, seed=0):
"""
Build a 25-second mono waveform made of labelled regions:
0- 3s silence (near-zero, faint hiss)
3- 9s speech-like (voiced harmonics, syllable-rate modulation)
9-12s silence
12-18s music-like (steady tonal chord)
18-21s applause-like (broadband noise bursts)
21-25s speech-like
Returns (wave, sr, regions) where regions is a list of
(label, start_sec, end_sec).
"""
rng = np.random.default_rng(seed)
def t(seconds):
return np.arange(int(seconds * sr)) / sr
def silence(sec):
return 0.002 * rng.standard_normal(int(sec * sr))
def speech(sec, f0=120.0):
x = t(sec)
# a few harmonics = "voiced"; 4 Hz envelope = syllable rate
tone = sum((1.0 / k) * np.sin(2 * np.pi * f0 * k * x) for k in (1, 2, 3, 4))
env = 0.5 * (1 + np.sin(2 * np.pi * 4.0 * x)) ** 2
return 0.3 * env * tone + 0.01 * rng.standard_normal(x.size)
def music(sec):
x = t(sec)
chord = (np.sin(2 * np.pi * 220.0 * x)
+ np.sin(2 * np.pi * 277.18 * x)
+ np.sin(2 * np.pi * 329.63 * x))
return 0.2 * chord + 0.005 * rng.standard_normal(x.size)
def applause(sec):
x = silence(sec) * 0 + 0.25 * rng.standard_normal(int(sec * sr))
# crackle: random amplitude bursts
burst = (rng.random(x.size) < 0.02).astype(float)
return x * (0.3 + burst)
plan = [
("silence", 3, silence(3)),
("speech", 6, speech(6, f0=120.0)),
("silence", 3, silence(3)),
("music", 6, music(6)),
("applause", 3, applause(3)),
("speech", 4, speech(4, f0=160.0)),
]
wave = np.concatenate([seg for _, _, seg in plan])
regions, cursor = [], 0.0
for label, dur, _ in plan:
regions.append((label, cursor, cursor + dur))
cursor += dur
return wave.astype(np.float64), sr, regions
# --------------------------------------------------------------------------- #
# 1 — frame the waveform and describe each frame with a few features
# --------------------------------------------------------------------------- #
def frame_signal(wave, sr, win_ms=50.0, hop_ms=25.0):
"""Slice the waveform into overlapping frames -> (n_frames, win) array."""
win = int(sr * win_ms / 1000)
hop = int(sr * hop_ms / 1000)
if wave.size < win:
return np.empty((0, win)), win, hop
n = 1 + (wave.size - win) // hop
idx = np.arange(win)[None, :] + hop * np.arange(n)[:, None]
return wave[idx], win, hop
def frame_features(wave, sr, win_ms=50.0, hop_ms=25.0):
"""
Turn audio into KTS-ready feature vectors: one row per frame.
Five classic, model-free descriptors per frame:
log_energy loudness (log RMS)
zcr zero-crossing rate (noisiness / pitch proxy)
centroid spectral centroid in Hz (brightness)
bandwidth spectral spread in Hz
flatness spectral flatness (tonal ~0 vs noisy ~1)
Returns (features (n, 5), hop_sec).
"""
frames, win, hop = frame_signal(wave, sr, win_ms, hop_ms)
if frames.shape[0] == 0:
return np.empty((0, 5)), hop / sr
# window to reduce spectral leakage
window = np.hanning(win)
fw = frames * window
# time-domain features
rms = np.sqrt(np.mean(frames ** 2, axis=1) + 1e-12)
log_energy = np.log(rms + 1e-8)
signs = np.sign(frames)
signs[signs == 0] = 1
zcr = np.mean(np.abs(np.diff(signs, axis=1)) > 0, axis=1)
# frequency-domain features
spec = np.abs(np.fft.rfft(fw, axis=1)) + 1e-12
power = spec ** 2
freqs = np.fft.rfftfreq(win, d=1.0 / sr)
psum = np.sum(power, axis=1, keepdims=True)
centroid = np.sum(freqs[None, :] * power, axis=1) / psum[:, 0]
bandwidth = np.sqrt(
np.sum(((freqs[None, :] - centroid[:, None]) ** 2) * power, axis=1) / psum[:, 0]
)
geo = np.exp(np.mean(np.log(power), axis=1))
arith = np.mean(power, axis=1)
flatness = geo / arith
feats = np.stack([log_energy, zcr, centroid, bandwidth, flatness], axis=1)
return feats, hop / sr
def zscore(features):
"""Standardize each column to mean 0, std 1 (so no feature dominates)."""
mu = features.mean(axis=0, keepdims=True)
sd = features.std(axis=0, keepdims=True)
return (features - mu) / np.maximum(sd, 1e-8)
# --------------------------------------------------------------------------- #
# 2 — segment with KTS, then score each segment for "bite-worthiness"
# --------------------------------------------------------------------------- #
def segment(features, kind="rbf", ncp_max=12, vmax=1.0, lmin=4):
"""Run KTS on the features. Returns cut points (start frame of each seg)."""
K = build_kernel(features, kind=kind)
cps, scores = cpd_auto(K, ncp_max=ncp_max, vmax=vmax, lmin=lmin)
return cps, scores
def cuts_to_segments(cps, n):
"""Turn change points into [start, end) frame ranges covering 0..n."""
bounds = [0, *list(cps), n]
return [(bounds[i], bounds[i + 1]) for i in range(len(bounds) - 1)]
def score_bites(features, segments, hop_sec, want=(2.0, 12.0)):
"""
Rank segments by how good a *speech* sound bite each one is.
Heuristic score (all from the raw features):
+ loud : high mean energy
+ voiced : low spectral flatness (tonal, not hiss/applause)
+ speechy ZCR : mid-range zero-crossing rate
+ right length: duration inside the wanted window (seconds)
Returns a list of dicts sorted best-first.
"""
log_energy, zcr, _, _, flatness = features.T
e_lo, e_hi = np.percentile(log_energy, [5, 95])
out = []
for (a, b) in segments:
dur = (b - a) * hop_sec
e = log_energy[a:b].mean()
z = zcr[a:b].mean()
fl = flatness[a:b].mean()
loud = (e - e_lo) / max(e_hi - e_lo, 1e-8) # 0..1-ish
voiced = 1.0 - min(fl / 0.5, 1.0) # tonal -> ~1
speechy = float(0.05 <= z <= 0.25) # speech ZCR band
lo, hi = want
fit = 1.0 if lo <= dur <= hi else max(0.0, 1 - abs(dur - np.clip(dur, lo, hi)) / hi)
score = 0.45 * np.clip(loud, 0, 1) + 0.30 * voiced + 0.15 * speechy + 0.10 * fit
out.append({
"start_sec": a * hop_sec, "end_sec": b * hop_sec, "dur": dur,
"energy": e, "zcr": z, "flatness": fl, "score": float(score),
})
return sorted(out, key=lambda d: d["score"], reverse=True)
# --------------------------------------------------------------------------- #
# 3 — end-to-end demo / report
# --------------------------------------------------------------------------- #
def fmt_ts(sec):
m, s = divmod(sec, 60)
return f"{int(m):d}:{s:04.1f}"
def main():
wave, sr, regions = synthesize_audio(sr=16000, seed=0)
print(f"clip: {wave.size/sr:.1f}s @ {sr} Hz ({wave.size} samples)")
print("ground-truth regions:")
for label, s, e in regions:
print(f" {label:9s} {fmt_ts(s)} - {fmt_ts(e)}")
feats, hop_sec = frame_features(wave, sr, win_ms=50.0, hop_ms=25.0)
print(f"\nframed: {feats.shape[0]} frames x {feats.shape[1]} features "
f"(hop {hop_sec*1000:.0f} ms)")
fz = zscore(feats)
cps, _ = segment(fz, kind="rbf", ncp_max=12, vmax=1.0, lmin=8)
cut_times = [c * hop_sec for c in cps]
print(f"KTS found {len(cps)} cut points at: "
+ ", ".join(fmt_ts(t) for t in cut_times))
segs = cuts_to_segments(cps, feats.shape[0])
ranked = score_bites(feats, segs, hop_sec)
print(f"\n{len(segs)} segments, ranked as sound-bite candidates:")
print(" rank start end dur energy zcr flat score")
print(" " + "-" * 60)
for r, b in enumerate(ranked):
print(f" {r:>4d} {fmt_ts(b['start_sec']):>7s} {fmt_ts(b['end_sec']):>7s}"
f" {b['dur']:5.1f} {b['energy']:6.2f} {b['zcr']:5.2f}"
f" {b['flatness']:5.2f} {b['score']:5.2f}")
best = ranked[0]
print(f"\ntop bite: {fmt_ts(best['start_sec'])} - {fmt_ts(best['end_sec'])} "
f"({best['dur']:.1f}s, score {best['score']:.2f})")
if __name__ == "__main__":
main()
That is the entire sound-bite tool: synthesize, features, KTS, score, in one file that imports the book's core and runs top to bottom. You have now used the same Kernel Temporal Segmentation engine to cut both pictures and sound. 👉