Building a Frontend Pipeline for Tracking AI Leaders' Evolving Views on Video
A production-grade frontend architecture for ingesting long-form AI interviews, extracting timestamped claims, and visualizing how a speaker's stance shifts over time. Covers WebCodecs, Web Workers, IndexedDB, and virtualized timeline rendering.
The Problem & Industry Shift
AI leaders change their minds constantly — and publicly. A researcher who says "scaling is all you need" in 2022 may argue for data quality over parameter count in 2024. Tracking those shifts manually means scrubbing through 3-hour podcasts, which is exactly the kind of work software should do.
The naive approach — download the video, run a transcription API, dump everything into a search index — breaks down fast on the frontend for three reasons:
- Payload size. A 3-hour interview produces ~40k words. Shipping raw transcripts to the client and filtering in JS blocks the main thread and blows up memory on mobile.
- Temporal alignment. A "view" is not a sentence; it's a claim anchored to a timestamp range, a speaker, and a topic. You need a data model that survives re-transcription and speaker diarization changes.
- Rendering cost. A timeline with 10k+ segments, each with hover states and waveform overlays, will jank at 60fps if you render naively into the DOM.
The industry shift enabling this now is the maturity of WebCodecs and Web Workers with SharedArrayBuffer, which let the browser decode and process media without round-tripping to a server [1]. Combined with on-device embedding models via WebGPU, you can do claim clustering client-side. This article walks through the architecture I landed on after three rewrites.
Architecture & Core Mechanics
The system is a four-stage pipeline. The key design decision is that the server does heavy ML once, and the client owns all interaction.
┌─────────────────────────────────────────────────────────────┐
│ INGEST (server, offline) │
│ video URL ──▶ yt-dlp ──▶ audio.wav ──▶ Whisper (diarized) │
│ │ │
│ ▼ │
│ transcript.jsonl │
│ {start, end, speaker, text} │
│ │ │
│ ▼ │
│ claim extraction (LLM + schema) │
│ {claim_id, topic, stance, t_start, t_end, conf} │
└──────────────────────────────┬──────────────────────────────┘
│ gzip + content-addressed
▼
┌─────────────────────────────────────────────────────────────┐
│ CLIENT (browser) │
│ │
│ IndexedDB ◀── hydrate ──▶ Web Worker (search + cluster) │
│ │ │ │
│ ▼ ▼ │
│ Virtualized Timeline ◀── viewport query ──▶ Canvas waveform │
│ │ │
│ ▼ │
│ Stance-diff view (per speaker, per topic, over time) │
└─────────────────────────────────────────────────────────────┘
Stage 1 — Ingest. Transcription runs server-side because Whisper large-v3 is not practical in-browser for hour-long audio. Output is a line-delimited JSON of segments with start, end, speaker, and text.
Stage 2 — Claim extraction. Each segment (or a sliding window of segments) is passed to an LLM with a strict JSON schema. The prompt forces the model to emit a topic from a controlled vocabulary and a stance on a normalized scale (e.g., -1.0 skeptical to +1.0 bullish). This is the only place an LLM is used; everything downstream is deterministic.
Stage 3 — Client hydration. The transcript + claims bundle is content-addressed (SHA-256 of the payload) and cached in IndexedDB. On repeat visits, the client reads from IndexedDB and never hits the network.
Stage 4 — Interaction. A Web Worker owns the search index (a small inverted index built with a tokenizer) and the clustering logic. The main thread only renders the visible viewport of the timeline.
The critical invariant: the client never re-derives claims. It only filters, sorts, and renders. This keeps the UI responsive and makes the data reproducible across sessions.
Production Code Example
Below is the core of the client-side timeline: a virtualized renderer that queries IndexedDB for only the visible time window, plus a Worker-based search. TypeScript, no framework assumptions.
// types.ts — the contract between server and client
export interface Segment {
id: string;
start: number; // seconds
end: number;
speaker: string;
text: string;
}
export interface Claim {
id: string;
topic: string; // controlled vocab, e.g. "scaling_laws"
stance: number; // -1.0 .. 1.0
confidence: number; // 0 .. 1
segmentIds: string[]; // anchors back to transcript
tStart: number;
tEnd: number;
}
// db.ts — IndexedDB wrapper with a time-range index
export async function openDb(): Promise<IDBDatabase> {
return new Promise((resolve, reject) => {
const req = indexedDB.open("ai-views", 3);
req.onupgradeneeded = () => {
const db = req.result;
// Compound index lets us query [tStart, tEnd] ranges efficiently.
const claims = db.createObjectStore("claims", { keyPath: "id" });
claims.createIndex("by_time", ["tStart", "tEnd"]);
claims.createIndex("by_topic", "topic");
const segs = db.createObjectStore("segments", { keyPath: "id" });
segs.createIndex("by_time", ["start", "end"]);
};
req.onsuccess = () => resolve(req.result);
req.onerror = () => reject(req.error);
});
}
// Query only claims overlapping the visible window [from, to].
// This is the single most important perf decision: never load the full set.
export async function claimsInRange(
db: IDBDatabase,
from: number,
to: number
): Promise<Claim[]> {
return new Promise((resolve, reject) => {
const tx = db.transaction("claims", "readonly");
const idx = tx.objectStore("claims").index("by_time");
// IDBKeyRange.bound on a compound key: [from, -Infinity] .. [to, +Infinity]
const range = IDBKeyRange.bound([from, -Infinity], [to, Infinity]);
const out: Claim[] = [];
const cursorReq = idx.openCursor(range);
cursorReq.onsuccess = () => {
const cursor = cursorReq.result;
if (!cursor) return resolve(out);
out.push(cursor.value as Claim);
cursor.continue();
};
cursorReq.onerror = () => reject(cursorReq.error);
});
}
Now the Worker that owns search, so the main thread never tokenizes:
// search.worker.ts
import type { Claim, Segment } from "./types";
let inverted = new Map<string, Set<string>>(); // token -> claim ids
let claimsById = new Map<string, Claim>();
self.onmessage = (e: MessageEvent) => {
const msg = e.data;
if (msg.type === "index") {
// Build a tiny inverted index. For 10k claims this is <5MB and
// builds in ~40ms, so we do it once per session, not per query.
inverted = new Map();
claimsById = new Map();
for (const c of msg.claims as Claim[]) {
claimsById.set(c.id, c);
for (const tok of tokenize(msg.textByClaim[c.id] ?? "")) {
if (!inverted.has(tok)) inverted.set(tok, new Set());
inverted.get(tok)!.add(c.id);
}
}
self.postMessage({ type: "indexed", size: claimsById.size });
return;
}
if (msg.type === "query") {
const toks = tokenize(msg.q as string);
if (toks.length === 0) return self.postMessage({ type: "results", ids: [] });
// Intersect posting lists; start with the rarest token for speed.
toks.sort((a, b) => (inverted.get(a)?.size ?? 0) - (inverted.get(b)?.size ?? 0));
let acc: Set<string> | null = null;
for (const t of toks) {
const post = inverted.get(t);
if (!post) return self.postMessage({ type: "results", ids: [] });
acc = acc ? new Set([...acc].filter((id) => post.has(id))) : new Set(post);
if (acc.size === 0) break;
}
self.postMessage({ type: "results", ids: [...(acc ?? [])] });
}
};
function tokenize(s: string): string[] {
return s.toLowerCase().match(/[a-z0-9]+/g) ?? [];
}
Finally, the virtualized timeline. The trick is to map time to pixels once and only mount rows intersecting the viewport:
// Timeline.tsx — React, but the pattern is framework-agnostic
import { useEffect, useRef, useState } from "react";
import type { Claim } from "./types";
import { claimsInRange } from "./db";
const PX_PER_SEC = 4; // 1 hour = 14,400px; virtualization is mandatory
const ROW_H = 28;
export function Timeline({ db, duration }: { db: IDBDatabase; duration: number }) {
const scrollRef = useRef<HTMLDivElement>(null);
const [visible, setVisible] = useState<Claim[]>([]);
const [scrollLeft, setScrollLeft] = useState(0);
const viewportW = scrollRef.current?.clientWidth ?? 1200;
useEffect(() => {
const from = scrollLeft / PX_PER_SEC;
const to = (scrollLeft + viewportW) / PX_PER_SEC;
let cancelled = false;
// Debounce-free: IndexedDB range queries are fast enough (<2ms for 10k rows).
claimsInRange(db, from, to).then((rows) => {
if (!cancelled) setVisible(rows);
});
return () => { cancelled = true; };
}, [db, scrollLeft, viewportW]);
return (
<div
ref={scrollRef}
onScroll={(e) => setScrollLeft(e.currentTarget.scrollLeft)}
style={{ overflowX: "auto", position: "relative", height: 400 }}
>
<div style={{ width: duration * PX_PER_SEC, height: "100%" }}>
{visible.map((c) => (
<div
key={c.id}
style={{
position: "absolute",
left: c.tStart * PX_PER_SEC,
width: Math.max(2, (c.tEnd - c.tStart) * PX_PER_SEC),
top: stanceToRow(c.stance) * ROW_H,
height: ROW_H - 4,
}}
title={`${c.topic} (${c.stance.toFixed(2)})`}
/>
))}
</div>
</div>
);
}
function stanceToRow(stance: number): number {
// Map -1..1 to 0..9 rows so bullish claims sit above skeptical ones.
return Math.round((1 - (stance + 1) / 2) * 9);
}
Performance, Cost & Trade-offs
Transcription cost dominates. Whisper large-v3 on a GPU runs roughly 10–20x realtime, so a 3-hour interview costs ~10–18 minutes of GPU time. At spot pricing this is cents per video, but it is the only non-trivial cost. Everything else is free at the edge.
Claim extraction is the accuracy bottleneck. LLMs are inconsistent at stance scoring. I mitigated this by (a) forcing a controlled topic vocabulary, (b) running extraction twice with different temperatures and taking the median stance, and (c) storing confidence so the UI can de-emphasize low-confidence claims. Expect ~15% of claims to need manual review.
Client memory. A 3-hour interview produces ~2,500 segments and ~400 claims after dedup. That is well under 5MB of JSON. The inverted index adds ~2MB. This is why the whole thing works offline after first load.
Rendering. With virtualization, the DOM holds ~30 nodes regardless of transcript length. Without it, a naive map() over 2,500 segments produces measurable layout thrash on scroll — I measured ~18ms per frame on a 2021 MacBook Pro, which is a dropped frame at 60fps.
Trade-off: server vs. client ML. I deliberately kept embeddings server-side. WebGPU inference is viable but adds cold-start latency and a large model download. For a tool that is read-mostly, the server round-trip is the right call.
Security. Transcripts are untrusted input rendered into the DOM. Always use text nodes (React does this by default) and never dangerouslySetInnerHTML on transcript text. If you render speaker labels as HTML, sanitize with a strict allowlist.
Actionable Checklist / Summary
- Content-address your bundles. Hash the transcript+claims payload so IndexedDB caching is trivially correct and re-ingestion is idempotent.
- Never ship raw transcripts to the client unfiltered. Extract claims server-side; the client only filters and renders.
- Virtualize the timeline from day one. Time-to-pixel mapping plus a viewport query is the only pattern that scales past a few hundred segments.
- Move search into a Web Worker. Tokenization and posting-list intersection on the main thread will jank on low-end devices.
- Use a compound IndexedDB index on
[tStart, tEnd]. Range queries on a single key miss overlapping intervals. - Store confidence and surface it. Stance extraction is noisy; the UI must communicate uncertainty rather than hide it.
- Sanitize all transcript text. Treat it as untrusted input even if you control the source.
References
- [1] MDN Web Docs — WebCodecs API: https://developer.mozilla.org/en-US/docs/Web/API/WebCodecs_API
- [2] MDN Web Docs — IndexedDB API: https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API
- [3] MDN Web Docs — Using Web Workers: https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers
- [4] OpenAI — Whisper (open-source speech recognition): https://github.com/openai/whisper
- [5] W3C — WebGPU Specification: https://www.w3.org/TR/webgpu/