Trending tools & algorithms - Kernel Temporal Segmentation from Scratch

Chapter 13 mapped the approaches; this chapter names the tools people actually reach for in 2024 and 2025 to build each layer, with minimal follow-along snippets. Our from-scratch NumPy pipeline (Chapter 15) is the spine; these are the production-grade parts you would swap in.

Follow-along, not run-here. These libraries (librosa, whisper, pyannote, ruptures, and the rest) are not installed in this book's verification environment, so the outputs below are illustrative. They show the shape of what you will see, not a value captured on this machine. The pure-NumPy code in Chapter 15 is run and verified. Install with the pip line shown above each block.

Layer 1: better features

`librosa`, the standard audio-feature toolkit

It covers everything we hand-rolled in Chapter 12 (framing, FFT, centroid, ZCR) plus the feature the field uses most: MFCCs (mel-frequency cepstral coefficients), a compact, perceptually-tuned fingerprint.

pip install librosa

import librosa
y, sr = librosa.load("clip.wav", sr=16000)          # waveform
mfcc  = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=20)  # (20, n_frames)
feats = mfcc.T                                        # (n_frames, 20) -> feed to KTS
print(feats.shape)

(1003, 20)         # ~1000 frames, 20-D each, drop straight into build_kernel(...)

Why this matters for KTS: swapping our five hand-features for feats above changes nothing downstream. build_kernel(feats) and cpd_auto work as-is. Better features give cleaner boundaries from the same engine.

Learned embeddings: `wav2vec2`, `CLAP`

For state-of-the-art fingerprints, use a pretrained audio encoder. wav2vec 2.0 (speech) and CLAP (general audio and text) turn each frame or clip into a rich vector. These are the audio analog of the CNN embeddings the video book mentions, and again they are just a different X into KTS.

Layer 2: boundaries

`ruptures`, change-point detection as a library

This is the closest published cousin of KTS. ruptures implements the same penalized dynamic-programming idea (Ch. 5 to 6), including kernel cost functions and PELT, a faster, pruned dynamic program for when you do not fix the number of cuts.

pip install ruptures

import ruptures as rpt
algo = rpt.KernelCPD(kernel="rbf").fit(feats)   # same kernel idea as this book
bkps = algo.predict(pen=10)                      # penalty replaces our vmax
print(bkps)

[120, 360, 480, 720, 840, 1003]   # change points (frame indices), like our cps

Don't be confused: ruptures vs. kts.py. They optimise the same objective. KTS in this book is the from-scratch, fixed-or-auto-count version built for video; ruptures is the general-purpose, well-optimised library with extra cost functions and PELT pruning. Understand kts.py, then reach for ruptures in production.

`Silero VAD` and `webrtcvad`, voice activity detection

Fast, tiny speech vs. non-speech gates. Use them to drop silence before KTS so the $O(n^2)$ cost is spent only on real content.

pip install silero-vad

from silero_vad import load_silero_vad, get_speech_timestamps, read_audio
model = load_silero_vad()
wav   = read_audio("clip.wav", sampling_rate=16000)
speech = get_speech_timestamps(wav, model, return_seconds=True)
print(speech[:2])

[{'start': 3.0, 'end': 9.0}, {'start': 21.0, 'end': 25.0}]   # speech-only spans

`pyannote.audio`, speaker diarization

The de-facto open-source diarizer ("who spoke when"). When bites are "one clean quote per speaker," its boundaries are the cuts you want.

pip install pyannote.audio

from pyannote.audio import Pipeline
pipe = Pipeline.from_pretrained("pyannote/speaker-diarization-3.1")
for turn, _, spk in pipe("clip.wav").itertracks(yield_label=True):
    print(f"{turn.start:5.1f}-{turn.end:5.1f}  {spk}")

  3.0-  9.0  SPEAKER_00
 21.0- 25.0  SPEAKER_01

Layer 3: transcript and sentence boundaries

`faster-whisper`, speech-to-text with timestamps

Whisper (and the speed-optimised faster-whisper) transcribes audio with per-segment timestamps, letting you snap bite edges onto complete sentences.

pip install faster-whisper

from faster_whisper import WhisperModel
model = WhisperModel("base")
segments, _ = model.transcribe("clip.wav")
for s in segments:
    print(f"[{s.start:5.1f}-{s.end:5.1f}] {s.text}")

[ 3.0-  9.0]  The single most important thing we learned this year ...
[21.0- 25.0]  ... and that's why it completely changed our roadmap.

Layer 4: choosing the best bite (semantic)

Boundaries get you candidates; ranking by meaning gets you the best bite. Two trending routes:

Embedding similarity with `sentence-transformers`

Embed each candidate's transcript and score it against a query ("the most surprising claim") or against the whole talk's centroid (most representative).

pip install sentence-transformers

from sentence_transformers import SentenceTransformer, util
enc = SentenceTransformer("all-MiniLM-L6-v2")
emb = enc.encode([c.text for c in candidates])
query = enc.encode("the most surprising, quotable claim")
ranked = util.cos_sim(query, emb)[0].argsort(descending=True)

LLM-as-judge: pick bites with Claude

The most flexible recent approach: hand a model the timestamped transcript and ask it to choose and justify the best clips. Using the Anthropic SDK with the current model id claude-opus-4-8:

pip install anthropic

import anthropic
client = anthropic.Anthropic()
transcript = "\n".join(f"[{c.start:.1f}-{c.end:.1f}] {c.text}" for c in candidates)
msg = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=400,
    messages=[{"role": "user", "content":
        "Here is a timestamped transcript. Pick the 3 best 8-second sound bites "
        "for social media. Return start/end timestamps and a one-line reason.\n\n"
        + transcript}],
)
print(msg.content[0].text)

1. 0:21-0:25  : punchy payoff line; complete sentence; strong verb ("changed").
2. 0:03-0:09  : sets up the year's key lesson; self-contained.
3. ...

The pattern: KTS, ruptures, or diarization produce candidate segments; the LLM or embedding step ranks them by meaning. The acoustic layer decides where, the semantic layer decides which, which is the same two-step split from Chapter 11.

How it all stacks

A modern, robust sound-bite pipeline is these layers composed:

  audio
    │  Silero VAD            drop silence (Layer 2)
    ▼
  speech spans
    │  librosa / wav2vec2    features (Layer 1)
    ▼
  feature vectors
    │  KTS  (this book!)     content-aware boundaries (Layer 2)   ◀── you built this
    ▼  └ or ruptures / pyannote
  candidate segments
    │  faster-whisper        snap to sentences (Layer 3)
    │  sentence-transformers / Claude   rank by meaning (Layer 4)
    ▼
  ranked sound bites

You now know every box. The next chapter is the hands-on lab that builds the core of this stack, with no VAD and no transcript, in pure NumPy, so you can watch KTS turn a waveform into ranked bites end to end. 👉