Provider contract#

How to write a provider package. The types are the source of truth: packages/core/src/provider.ts and packages/core/src/types.ts. This page explains the rules around them. If this page and the types disagree, the types win; fix this page in the same commit.

Package layout#

providers/<name>/
  package.json        name "@vid2context/provider-<name>", "vid2context": { "runtimes": [...] }
  src/index.ts        default export: one Provider or Provider[]
  test/*.test.ts      unit tests on recorded fixtures, plus LIVE=1 tests
  test/fixtures/      real recorded responses, with a README saying where each came from

package.json essentials:

{
  "name": "@vid2context/provider-<name>",
  "type": "module",
  "exports": { ".": "./src/index.ts" },
  "vid2context": { "runtimes": ["node", "workers", "browser"] },
  "dependencies": { "@vid2context/core": "*" }
}

• runtimes lists only what the code really supports. Anything that imports node:* modules, spawns processes or reads files is ["node"]. scripts/gen-registry.mjs keeps such providers out of the Workers and browser registries.
• Run npm install (or npm run gen) after creating the package. Never edit packages/registry/src/generated/.
• Add dependencies only to your own package.json. Prefer plain fetch over vendor SDKs.

Rules for every provider#

1. Use ctx, never globals. Read config from ctx.env, make requests with ctx.fetch (or fetchWithTimeout(ctx, url, init, ms) from core), log with ctx.log, honour ctx.signal. Never touch process.env or the global fetch inside a provider. This is what makes one provider work on Node, Workers, the browser and in tests.
2. stdout is not yours. Log only through ctx.log (stderr). A stray console.log corrupts the MCP stdio protocol.
3. Throw Vid2ContextError. Every error a user can see has a stable code and, where you can say how to recover, a hint. Use errorFromResponse(res, label) for non-2xx responses: it maps 429 to rate-limited (with Retry-After), 401/402/403 to auth, 404/410 to not-found, 413 to too-large, other statuses to upstream.
4. Input errors stop the chain. Throw bad-request or unsupported-input only when the caller's input is wrong and no other rung could do better. Everything else falls through to the next rung.
5. No placeholders. No mock data, no TODO stubs, no fake responses. If something cannot be built yet, leave it out and say so in the docs.

Source providers (kind: 'source')#

VideoRef rules:

• id is ${platform}:${nativeId} and is the storage and cache key. Lists use ${platform}:${kind}:${nativeId}, for example youtube:playlist:PL....
• platform is a short slug: youtube, vimeo, file, podcast, web.
• url is canonical when one exists. timestampUrl in core turns it into deep links.

Captions providers (kind: 'captions')#

• supports(ref) is synchronous.
• getCaptions(ref, opts, ctx) returns the best Transcript for opts, or null when the video has no usable track. Throw only on real failures (network, blocking).
• Respect opts.language, opts.allowAuto (default true) and opts.allowTranslated (default true). A machine-translated result sets translated: true and keeps the source track's origin.
• origin is creator-captions, auto-captions or native-transcript.

Rung providers (stt, llm, embed, video, visual)#

A rung is one provider + one model + one credential. Providers build rungs from env on every call:

rungs(env: Env): SttRung[]   // and LlmRung[], EmbedRung[], VideoRung[], VisualRung[]

rungs(env) must be pure and fast (no network). Return every rung the provider knows, enabled or not, so /api/v1/health can show what a key would unlock.

Rung fields#

Capability vocabulary (add new words here when you need them):

Kind-specific members: SttRung.maxBytes (largest upload; the pipeline skips rungs whose limit is below the audio size), LlmRung.contextTokens (usable window on that tier), EmbedRung.dimensions.

Env var conventions#

Every new env var goes into .env.example (commented, optional) and docs/providers.md in the same commit.

Keyless lanes#

Keyless rungs send data to a third party without an account. They are on by default (the zero-config path) and VID2CONTEXT_KEYLESS=0 turns them all off. Never send an Authorization header to a keyless endpoint: OVH and Pollinations reject bogus ones.

Model ids#

Free model ids are retired without notice. Pin ids only where the provider keeps them stable, make every id overridable through env, and never pin an OpenRouter :free id (pick from the live list instead).

Streaming#

An LLM rung streams by also implementing StreamingLlmRung from packages/core/src/llm-stream.ts: an LlmRung plus stream(req, ctx): AsyncIterable<ChatStreamEvent>. canStream(rung) tells them apart, and listModels adds the stream capability to such rungs (do not list stream in capabilities yourself). Rungs without stream still work behind a streaming UI: streamFromRung calls chat() and emits the whole answer as one delta.

Events a rung yields, in order:

Errors are thrown, never yielded: a non-2xx before the stream goes through errorFromResponse, an error object inside the stream becomes a Vid2ContextError, a body that ends with neither [DONE] nor a finish_reason is an upstream error, and an empty answer is an upstream error (as in chat()).

Fallback rule (chatStream in packages/core/src/llm/stream.ts): rungs are ranked exactly as chat() ranks them. A rung that fails before its first delta is skipped and the next one is tried, with the chain's cooldowns (opening a stream runs through runChain, so a 429 cools the rung down for chat() too). After the first delta the stream never switches rungs: a failure ends it with an error that carries attempts. chatStream adds a leading { type: 'meta', rung, model, contextTokens? } event, sent again with restart: true only when a rung failed after reasoning but before any answer text, and its done also carries rung and attempts.

How a provider adds streaming:

• OpenAI-compatible APIs: call streamOpenAiChat({ url, model, key?, label?, headers?, body? }, req, ctx) from @vid2context/core. Pass body: (req) => <your non-streaming body builder> so both paths send the same request; it adds stream: true and stream_options: { include_usage: true }, and retries once without stream_options when a service answers 400 naming it (set noStreamOptions: true to skip it from the start).
• Other wire formats: parse the body with parseSSE(res.body, signal) from @vid2context/core and yield the events above.
• Type the rung object as StreamingLlmRung where you build it, then return it in the LlmRung[].
• Test with real recorded stream bodies (raw bytes) replayed through replayFetch; never hand-write an SSE stream.

Platform bindings (ctx.bindings)#

ctx.bindings is an optional Record<string, unknown> of platform objects that are not strings: on Cloudflare Workers, createWorkersContext(env) puts every non-string env value there (the AI, D1, R2, KV and Vectorize bindings) and keeps strings in ctx.env. On Node and in the browser it is usually absent.

• Treat it as optional. rungs(env) and isEnabled(env) see only env, so a provider that needs a binding is switched on by a string env var naming the binding (for example CLOUDFLARE_AI_BINDING=AI, VID2CONTEXT_D1=DB) and reads ctx.bindings[name] at call time.
• When the named binding is missing or has the wrong shape, throw a Vid2ContextError whose hint names the binding and the wrangler.jsonc entry that creates it.
• Never copy bindings into logs, attempts or error messages.

Telemetry (ctx.telemetry)#

ctx.telemetry is an optional Telemetry from packages/core/src/telemetry/types.ts: spans, counters, histograms, events and an optional usage ledger. The chain and the pipeline already record a span per stage and per rung attempt, so providers need do nothing.

• Treat it as optional. When it is absent, do nothing and allocate nothing.
• A provider may add detail with ctx.telemetry?.event(name, attrs) inside its call. Attribute values are ids, counts and flags only: never transcript text, prompts, URLs with query strings, keys or any env value.
• Report usage on the result, not through telemetry: LLM rungs fill ChatResult.usage; the chain turns it into token counters and paid cost.

Storage providers (kind: 'storage')#

isEnabled(env) decides from env alone; the enabled backend with the lowest priority wins. create(ctx) returns an object implementing Storage from packages/core/src/storage.ts. Unknown ids throw Vid2ContextError('not-found'). Raw transcripts are always stored; summaries and briefs are stored next to them, never instead of them.

Testing#

Unit tests replay real recorded responses; they never reach the network and never use invented payloads.

import { replayFetch, testContext } from '@vid2context/core/testing';

const fetch = replayFetch([
  { method: 'POST', url: 'https://api.example.com/v1/audio/transcriptions', response: { body: fixture('recorded.json') } },
  { url: /timedtext\?v=jNQXAC9IVRw/, response: { status: 429, headers: { 'retry-after': '31' }, body: fixture('429.html') } },
]);
const ctx = testContext({ fetch, env: { GROQ_API_KEY: 'gsk_test' } });

• replayFetch throws on any request without a recorded response and keeps every call in .calls (URL, method, headers, body) for assertions.
• testContext never reads process.env; pass the env the test needs.
• Live tests: describe.skipIf(!process.env.LIVE)(...) with createContext({ runtime: 'node', env: process.env }). Run them with npm run test:live.
• Record fixtures from the real service, then trim anything personal before committing: stream URLs that embed your IP, emails, phone numbers, keys. Grep the fixture for your public IP. Document each file in test/fixtures/README.md (source, date, what was trimmed).
• Shared real fixtures live in tests/fixtures/ (a public-domain mp3 and real VideoContext JSON). Read them; do not edit them.
• When a credential is missing, test against recorded fixtures and the keyless or local lane, and say in your final report what could not be verified live.

Reference implementations#