Architecture#

vid2context turns a video (a URL, a playlist, a podcast episode, a local file) into context an AI agent can read and cite: metadata, chapters and a timed transcript, plus optional on-screen notes, comments and cited briefs across many videos. One TypeScript core runs behind every surface: a CLI, an MCP server, an HTTP API and, later, a web app and a browser extension.

Layout#

packages/
  core/        types, provider contracts, rung chain, pipeline, formatters, storage contract (runtime-neutral)
  registry/    per-runtime provider registries, generated from providers/* (node, workers, browser)
  cli/         the vid2context command
  mcp/         the MCP server (stdio now, Streamable HTTP later)
  server/      the HTTP API on Hono
providers/
  youtube/                 source + captions (innertube, keyless)
  media-file/              local files (node only)
  stt-openai-compatible/   speech-to-text rungs
  llm-openai-compatible/   chat rungs
scripts/gen-registry.mjs   scans providers/*/package.json, writes packages/registry/src/generated/
tests/fixtures/            shared real fixtures (public-domain audio, real VideoContext JSON)
docs/contracts/            the contracts parallel work builds against

Every package exports its TypeScript source directly ("exports": { ".": "./src/index.ts" }) and imports with explicit .ts extensions. tsx runs it, vitest tests it, Biome lints it. Bundles for npx (CLI, MCP) come later with tsup.

Data model#

input string ──resolve──▶ VideoRef { platform, kind, id = "platform:nativeId", url }
                              │
                              ▼
VideoContext {
  metadata:   VideoMetadata  (title, channel, description, duration, chapters[], tags, language)
  transcript: Transcript     (language, origin, producer, model, segments[{startSec, endSec, text}])
  visual?:    VisualNote[]   (slide text, code, charts seen on screen)
  comments?:  Comment[]
  attempts:   Attempt[]      (every provider and rung tried, with outcome and reason)
}

TranscriptOrigin says how trustworthy the text is: creator-captions, auto-captions, native-transcript, stt, video-model.

Design rule: raw transcripts are always kept. Summaries, briefs, extractions and translations are layers stored next to them, never replacements.

The pipeline#

getVideoContext(registry, input, opts, ctx) in packages/core/src/pipeline.ts:

resolve input ─▶ storage hit? ─▶ metadata ─▶ transcript ─▶ comments? ─▶ save
                                              │
            mode auto:     captions (creator, auto, translated) ─▶ audio + speech-to-text chain
            mode captions: captions only, never download audio
            mode stt:      always transcribe the audio
            mode video:    a multimodal video rung reads the whole video (transcript + visual notes)

• Source resolution: the registry asks each source's synchronous match(); platform-specific sources go before catch-alls (yt-dlp, web page discovery).
• Metadata failures that mean "this video cannot be had" (not-found, unavailable, blocked) stop the pipeline; any other metadata failure falls back to minimal metadata and keeps going.
• Audio can come from any source whose canProvideAudio(ref) is true, so yt-dlp can supply audio for YouTube refs.
• Speech-to-text rungs whose maxBytes is below the audio size are skipped before the chain runs (long audio is split by the longform orchestrator, item 04).
• Every step appends to attempts. Errors carry the attempts too, so every surface can show why something failed.

expandInput turns playlist, channel, feed and page refs into video refs. transcribe and chat expose the speech-to-text and LLM chains directly.

The rung chain#

packages/core/src/chain.ts. A rung is one provider + one model + one credential (Groq and Cerebras meter quota per model, so each model is its own rung; each pooled key is its own rung too).

• A rung is present only when its credential is set; keyless and local rungs need none.
• Default policy free orders by tier: local (this machine, private), free (hosted free tier, your key), keyless (hosted, no account), paid (only when its key is set). Then by priority.
• local-only keeps only local rungs. quality orders by quality score, tier breaking ties.
• runChain tries rungs in order, falls through on any error except caller input errors, and records every attempt.
• Per-process cooldowns: rate-limited for Retry-After or 60 s, auth failures 10 min, a retired model 30 min, 5xx 30 s.
• Env controls: VID2CONTEXT_POLICY, VID2CONTEXT_ALLOW_PAID=0, VID2CONTEXT_SKIP=svc1,svc2, VID2CONTEXT_KEYLESS=0.

The same chain serves every rung kind: stt, llm, embed, video, visual. Providers build rungs from env (rungs(env)), so model overrides and key pools need no code change.

Providers and the registry#

Providers are separate workspace packages under providers/<name>/. Each declares "vid2context": { "runtimes": [...] } in its package.json and default-exports one provider or an array. scripts/gen-registry.mjs (run on install and before typecheck and test) writes one registry per runtime:

packages/registry/src/generated/node.ts      every provider with "node"
packages/registry/src/generated/workers.ts   ... "workers"
packages/registry/src/generated/browser.ts   ... "browser"

So node-only providers (child processes, filesystem) never reach a Workers or browser bundle, and adding a provider never means editing a shared file: fifty agents can each add one at once. The generated files are gitignored.

Providers never read process.env or call the global fetch. Everything comes through the Ctx passed to each call (env, fetch, log, signal, runtime), which is what lets one provider run on Node, Workers, the browser and in tests with recorded responses. See contracts/providers.md.

Formats#

packages/core/src/format/index.ts holds a formatters map (text, json today). Every surface lists formats from that map and renders through render(), so a formatter added there appears in the CLI, MCP, HTTP API and web app at once.

Storage#

The Storage interface in packages/core/src/storage.ts covers saved contexts, collections and search. MemoryStorage is the zero-dependency default. Storage providers (SQLite, Cloudflare D1, Postgres) register like any provider; the enabled one with the lowest priority wins. The pipeline reads through storage as a cache and saves every result.

Surfaces#

All of them log to stderr. stdout belongs to program output and to the MCP stdio protocol.

Who synthesizes#

By default the calling agent does. The MCP server returns transcripts, sections and search hits, and offers a "brief" prompt, so there is no extra API key or cost. Server-side map-reduce on a cheap free model is optional, for collections too large for the agent's context (item 32).

Size guide: speech is about 12k tokens per hour, so a 15-episode series of 30-minute videos is about 90k tokens. That is why outlines, sections, chapters and search exist.

Runtimes and deployment#

YouTube blocks many datacenter IPs ("sign in to confirm you're not a bot"). Running locally, or behind a Cloudflare Tunnel to a home network, is the robust path; the YouTube provider reports a blocked error with that hint.

Decisions and why#