# Collections and watch lists

A collection groups videos on one topic and can follow sources: YouTube playlists and channels, podcast feeds, pages, anything a source provider can list. A refresh expands every source and ingests only what the collection has not seen yet, through the normal free-first pipeline (captions first, speech-to-text only when there are none). A watch list refreshes on a schedule, and a webhook hears about every new video.

Code: [packages/library](https://github.com/nirholas/vid2context/tree/main/packages/library/). HTTP routes: the collection rows in [contracts/http-api.md](https://vid2context.com/docs/contracts/http-api.md). MCP tools: `list_collections`, `create_collection`, `add_to_collection`, `get_collection` in [contracts/mcp-tools.md](https://vid2context.com/docs/contracts/mcp-tools.md).

## Walkthrough: a G-Wagon research collection

Over HTTP (`npm run dev:server`, port 8787):

```sh
API=http://127.0.0.1:8787/api/v1

# 1. Create the collection and follow the series playlist.
curl -s -X POST $API/collections -H 'content-type: application/json' \
  -d '{"name": "G-Wagon", "sources": ["https://www.youtube.com/playlist?list=PLHeawKfarQSPKIjpe86at3Ttc2sz7uD_b"]}'
# -> 201 {"id": "g-wagon-1f661f13", "sources": [...], "videoIds": [], ...}

# 2. Ingest every video in the playlist.
curl -s -X POST $API/collections/g-wagon-1f661f13/refresh
# -> 202 Job (kind "refresh") when the server runs the jobs queue: follow GET /jobs/:id/events.
#    Without a queue: {"collection": ..., "summary": {"added": [...], "skipped": 0, "failed": [...], ...}}

# 3. Check every day at 06:00 UTC for new episodes.
curl -s -X PATCH $API/collections/g-wagon-1f661f13 -H 'content-type: application/json' \
  -d '{"watch": {"every": "daily", "at": "06:00"}}'

# 4. List what is in it.
curl -s $API/collections/g-wagon-1f661f13
# -> the collection plus "videos": [{ "id": "youtube:cCPsdFdlshc", "title": "Best & Worst G-Wagon Years: ...", "origin": "auto-captions", ... }]
```

From an agent over MCP: `create_collection { name: "G-Wagon", sources: [...] }`, then `add_to_collection { collection, urls: [<playlist URL>] }`, then `get_collection`. Search and cite with `search_transcript { query, collection }`.

From TypeScript:

```ts
import { MemoryStorage } from '@vid2context/core';
import { createNodeContext, createRegistry } from '@vid2context/registry/node';
import { createLibrary } from '@vid2context/library';

const registry = createRegistry();
const ctx = createNodeContext();
const lib = createLibrary({ registry, ctx, storage: (await registry.openStorage(ctx)) ?? new MemoryStorage() });

const c = await lib.createCollection({ name: 'G-Wagon' });
await lib.addSource(c.id, 'https://www.youtube.com/playlist?list=PLHeawKfarQSPKIjpe86at3Ttc2sz7uD_b');
const summary = await lib.refresh(c.id, { onProgress: (p) => console.error(`${p.done}/${p.total} ${p.message}`) });
```

What the live run on 2026-11-14 from a GCP VM returned: the playlist expanded to its 9 videos, and YouTube answered every caption download with HTTP 429 (its block on datacenter IPs), so all 9 came back in `failed` with `code: "no-transcript"` and `cause: "youtube-captions blocked: ... HTTP 429 ..."`. Nothing was added, and the next refresh retries them. From a home connection, or with the yt-dlp provider and a speech-to-text rung installed, they ingest. See [youtube.md](https://vid2context.com/docs/guides/youtube.md) for blocks.

## Sources

`addSource(id, input)` (or `sources` on create and PATCH) accepts only lists: the input must resolve to a playlist, channel, feed or page, and its source provider must be able to list it. A single video is refused with `bad-request` (add it as an item instead), and an input no provider understands with `unsupported-input`. Different spellings of the same list are stored once. `removeSource` stops following a list; videos already ingested stay.

## Items

`POST /collections/:id/items { inputs }`, MCP `add_to_collection`, or `lib.addVideos(id, inputs)`: each input is a video or a list (lists are expanded). Videos already in the collection are skipped. `DELETE /collections/:id/items { videoIds }` removes videos and remembers them, so a later refresh does not bring them back; adding one explicitly does. Saved transcripts stay in the library either way.

## Refresh

`refresh(id, { limit?, concurrency?, delayMs?, onProgress? })`:

1. Expands every source (plus inputs queued by an import).
2. Keeps the items that are neither in the collection nor in its seen set.
3. Fetches them with at most `concurrency` (default 3) at a time and a pause of `delayMs` (default 300) between items on each worker, adding each success to the collection as it lands.
4. Records each failure (`input`, `code`, `message`, `hint`, `cause`) and keeps going. Failures are retried on the next refresh.

It returns `{ collectionId, added: VideoSummary[], skipped, failed, remaining, checkedAt }` and reports progress as `{ done, total, message }`, the Job progress shape. Running it twice adds nothing the second time. One process never runs two refreshes of the same collection at once: a second call joins the first.

Limits: `VID2CONTEXT_MAX_PLAYLIST_ITEMS` (default 500, a core limit) caps the items read from each source and the items ingested per run. `limit` lowers the per-run cap; items beyond it are counted in `remaining` and picked up next time.

## Watch lists

A schedule per collection, stored under `library:watch:<id>` in the KvStore (the storage backend when it implements one, else process memory):

```json
{ "every": "hourly" | "daily" | "weekly" | { "minutes": 30 }, "at": "06:00", "enabled": true }
```

`at` is a UTC time of day (hourly schedules use only its minutes). Without `at`, a new watch list is due at once and then every period after each check. With `at`, it runs at the next matching slot, then at the same time the next day (or week). `minutes` is 5 or more. State lives under `library:state:<id>`: `{ seenIds, lastCheckedAt, lastError, nextDueAt }`.

Set it with `PATCH /collections/:id { "watch": {...} }` (`null` removes it), `POST /collections { ..., "watch": {...} }`, or `lib.setWatch(id, schedule)`.

Something has to call `lib.runDue(now)`, which refreshes every enabled collection whose `nextDueAt` has passed, one at a time:

- Node, in a long-running process: `startScheduler(lib, { intervalMs: 60_000 })` returns `{ stop() }`. The timer is unref'd, and `stop()` waits for a run in progress.
- Cloudflare Workers: `export default { scheduled: scheduledHandler({ library: (env) => createLibrary(...) }) }` with a cron trigger. It runs at the cron's scheduled time and keeps the Worker alive until webhooks are sent.

Calling `runDue` from several processes at once is safe: each collection is taken under a lease (`library:lease:<id>`, one hour TTL, so a crashed run frees it), and a collection someone else holds is reported as `leased` and left alone. On a KvStore without atomic writes (Cloudflare KV is eventually consistent) the lease is best effort; refreshes are idempotent, so a rare double run only costs requests.

## Webhooks

Set `VID2CONTEXT_WEBHOOK_URL` and `VID2CONTEXT_WEBHOOK_SECRET`. After a refresh that adds videos, vid2context POSTs:

```json
{ "collection": { "id": "g-wagon-1f661f13", "name": "G-Wagon" }, "added": [ { "id": "youtube:cCPsdFdlshc", "title": "...", "ref": { "url": "https://www.youtube.com/watch?v=cCPsdFdlshc" }, "origin": "auto-captions", "language": "en", "savedAt": "..." } ], "checkedAt": "2026-11-14T06:00:12.000Z" }
```

with headers `X-Vid2context-Signature: sha256=<hex HMAC-SHA256 of the raw body, keyed with the secret>` and `X-Vid2context-Timestamp: <unix seconds>`. 5xx, 429 and network errors are retried with exponential backoff (4 attempts, 1 s base); 4xx is not retried. A failed delivery is logged and never fails or slows the refresh result.

Verify the signature over the raw body, before parsing it:

```js
import { createHmac, timingSafeEqual } from 'node:crypto';

function verify(rawBody, header, secret) {
  const expected = Buffer.from(`sha256=${createHmac('sha256', secret).update(rawBody).digest('hex')}`);
  const got = Buffer.from(header ?? '');
  return got.length === expected.length && timingSafeEqual(got, expected);
}
```

```python
import hashlib, hmac

def verify(raw_body: bytes, header: str, secret: str) -> bool:
    expected = "sha256=" + hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, header or "")
```

The signature covers the body only. To refuse replays, also reject requests whose timestamp header is more than a few minutes old, and ignore `added` ids you have already handled. In JS, `verifySignature(body, secret, header)` from `@vid2context/library` does the comparison.

These two variables are a simple per-deployment hook for new collection items. For per-user subscriptions to many event types, see the webhooks package (`/webhooks` routes).

## Import and export

`GET /collections/:id/export` (or `lib.exportCollection(id)`) returns a portable definition, without transcripts:

```json
{
  "format": "vid2context.collection",
  "version": 1,
  "name": "G-Wagon",
  "description": "Buying research",
  "sources": ["https://www.youtube.com/playlist?list=PLHeawKfarQSPKIjpe86at3Ttc2sz7uD_b"],
  "videos": ["https://www.youtube.com/watch?v=jNQXAC9IVRw"],
  "schedule": { "every": "daily", "at": "06:00", "enabled": true }
}
```

`POST /collections/import` (or `lib.importCollection(json)`) validates everything first (`bad-request` with a hint, nothing created), then creates the collection, sets its schedule, and queues `videos` for the next refresh. Add `?refresh=1` (`{ refresh: true }`) to ingest at once. Videos whose transcript is not saved in the library are left out of an export, because only a saved context knows its URL.