What is a sound bite? (and why segmentation finds it)
So far this book has cut video into shots. The same machine cuts audio into pieces, and once you can cut audio you can pull out its most quotable moments: sound bites. This short section (Chapters 11 to 15) is the audio counterpart of everything you have already built.
The one-sentence idea
A sound bite is a short, self-contained excerpt of audio that stands on its own: a quotable sentence from an interview, the chorus of a song, the punch line of a podcast. Finding sound bites means two things. First, decide where the audio naturally breaks. Then decide which pieces are worth keeping.
An everyday analogy
You have listened to a 40-minute interview and a producer says: "give me the three best 8-second clips for social media." In your head you do two things. You mentally chop the recording where the sound changes (host talking, guest talking, laughter, a musical sting, silence), then you skim those chunks and keep the few that are loud, clear, and complete. A computer follows the same two steps, and step one is change-point detection, which is what KTS already does.
Why is it needed?
Raw audio is a long, undifferentiated stream of numbers. Almost anything useful you want to do with it first needs it sliced into meaningful pieces:
- Social and marketing clips. Auto-cut the ten most shareable seconds from a podcast or speech.
- Search and navigation. "Jump to the part where they talk about pricing" needs the audio indexed by segment.
- Captioning and transcription. Speech recognizers work far better on short, homogeneous chunks than on a 60-minute blob.
- Highlight reels. Sports commentary, lecture recaps, trailer audio.
- Data prep for ML. Just as KTS prepares video for summarization datasets, audio segmentation prepares clips for training and evaluation.
Don't be confused: sound bite vs. shot vs. utterance. A shot is a visual segment, which was KTS's original job. An utterance is one continuous bit of speech from one speaker. A sound bite is a curated excerpt, usually one utterance or a few, chosen because it stands on its own. All three are temporal segments. They differ only in what makes a boundary and what makes a piece worth keeping.
How does it work? (the two-step view)
raw waveform ranked sound bites
~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~
░░▒▓██▓▒░░▒▓██▓▒ ──▶ features ──▶ KTS cuts ──▶ score each ──▶ 🏆 0:21-0:25
(440k samples) (one vector (where the segment 🥈 0:03-0:09
per frame) sound for "bite- …
changes) worthiness")
- Where does the sound change? Turn the waveform into a sequence of feature vectors, one per short frame, then find the change points. This is the same thing KTS does on video. Only the feature vectors come from audio instead of pixels (Chapter 12).
- Which pieces are worth keeping? Score each resulting segment (loud enough? clear speech, not noise or silence? right length?) and rank (Chapter 15).
Why KTS is a natural fit
KTS never assumed its input was video. Re-read its contract: it takes a sequence of feature vectors $x_1,\dots,x_n \in \mathbb{R}^d$ and finds boundaries so that each segment is internally consistent (Introduction). It does not care whether each $x_t$ describes a video frame or a 50-millisecond slice of sound.
| In video KTS (this book) | In audio sound-bite detection |
|---|---|
| frame = one image (~1/sec) | frame = one ~50 ms slice of waveform |
| feature = color histogram or CNN embedding | feature = energy, pitch, spectral shape (Ch. 12) |
| kernel matrix $K$ over frames | the same kernel matrix, over audio frames |
| DP finds optimal cuts | the same cpd_auto finds optimal cuts |
| output = shots + keyframes | output = segments, then ranked sound bites |
Everything in the middle stays the same: the kernel
(Ch. 1 to 3), the scatter cost
(Ch. 4), and the dynamic program
(Ch. 5 to 6) are all reused without changes. In
Chapter 15 we literally import kts.py and feed
it audio.
A concrete teaser
Chapter 15's lab builds a 25-second clip with six known regions (silence, speech, silence, music, applause, speech) and asks KTS to find the boundaries blind. It recovers all five exactly, then ranks the speech segments as the best bites:
KTS found 5 cut points at: 0:03.0, 0:09.0, 0:12.0, 0:18.0, 0:21.0
top bite: 0:21.0 - 0:25.0 (4.0s, score 0.81)
That is the whole story in miniature: the same KTS you already understand, cutting sound instead of pictures, with a small scoring step on top.
Next: how a waveform becomes the feature vectors KTS expects. 👉