simpleplayer-helper

A small long-running service beside simpleplayer-svc. The server knows every video in the library by its S3 key and etag, but not what the file is. The helper finds out: for each media file it reads just enough of the file with ffprobe (over a presigned Range-capable URL, never a full download) to get the container, duration and resolution, and reports that back. It does this once per file version: the server stores each result against the object's etag and lists only media whose result is absent or stale.

The contract with the server is docs/HELPER_API.md in simpleplayer-svc: base path /api/helper/v1, three endpoints (GET /media/unprobed, GET /media/:id/download-url, PUT /media/:id/probe). There is no authentication yet (TODO(auth) on the server).

Status: complete for version 1 of the contract. It runs until stopped, probing new and changed files as scans find them.

Configuration

Everything comes from environment variables. The helper refuses to start, listing every problem, when a required variable is missing or a value is malformed.

Variable	Default	Notes
SERVER_URL	required	The svc base URL, e.g. http://localhost:8080. A path prefix is kept (https://host/simpleplayer)
POLL_INTERVAL_SECONDS	60	How long to wait before asking again once nothing is left to probe (1 to 86400)
FFPROBE	ffprobe	A bare name is looked up on PATH; a path must be an executable file. Checked at startup
PROBE_TIMEOUT_SECONDS	120	How long one ffprobe run may take before it is killed and retried later (1 to 3600)
LOG_LEVEL	info	trace, debug, info, notice, warning, error or critical

The helper needs no storage of its own and no S3 credentials: the server hands it a presigned URL per file. Those URLs point at the server's S3_PUBLIC_ENDPOINT, so the helper must be able to reach that host.

Probing

For each file the helper runs

ffprobe -v error -print_format json -show_format -show_streams <presigned URL>

and takes container from format.format_name, durationSeconds from format.duration (else the video stream's), and width, height, videoCodec, frameRate (r_frame_rate evaluated) and hdr (colour transfer PQ or HLG; false for any other, untagged or unknown transfer) from the first video stream.

ffprobe fetches over HTTP with Range requests and seeks where the container needs it, so it reads only the headers, plus the moov atom at the end of an MP4 that has it there. No -probesize or -analyzeduration is set because none is needed. Measured with ffprobe 8.1's own read counter (-v debug, "Statistics: … bytes read") on the svc dev seed clips: 64 to 200 KB of files of 3 to 36 MB (0.6 to 2 %), and 230 KB of the 36 MB clip rewritten with its moov at the end (two seeks: to the tail and back). Over HTTP the server sends somewhat more than that, because it fills socket buffers before ffprobe hangs up: a Python range server logged 1.1 to 1.4 MB per clip, and the test suite's server (16 KB send buffer) sees about 0.56 MB of a 26 MB moov-at-the-end clip.

Every run ends one of three ways:

Outcome	When	What happens
Identified	ffprobe exits 0 and the JSON has a video stream, a container, a duration and a size	Submitted
Permanent	ffprobe says "Invalid data found when processing input" (not media, corrupt, moov atom not found), or exits 0 with no video stream, no duration (a still image, random bytes), no size, or JSON that does not parse	Submitted with error set, so this file version is not offered again
Transient	stderr shows an HTTP error (403 expired URL, 404, 5xx) or a connection problem; the run timed out or was killed; or it failed with a message not known to be permanent	Not submitted; retried later

A network error wins over "Invalid data": if the seek to a trailing moov atom gets a 403 because the URL expired, ffprobe reports both, and the file is not to blame. Error messages are ffprobe's stderr lines, shortened and with the URL removed (it carries a signature).

How it runs

On start and then every POLL_INTERVAL_SECONDS the helper makes a pass: it fetches GET /media/unprobed until the list has nothing new, and for each file gets a download URL, runs ffprobe on it and PUTs the result, one file at a time. A probed file drops out of the server's list, so a pass ends when only files that are backing off are left, and a second pass over an unchanged library does nothing.

• Success or permanent failure: submitted; the server will not offer that file version again.
• Transient failure (the table above, or a 5xx from the server for that file): logged at warning with the attempt count and retried on a later pass after 1 min, 2, 4 … capped at 1 h, per media id and etag. The backoff is held in memory; a restart simply retries. A file that never stops failing shows up once an hour as a warning, and in each pass summary as deferred with mostAttempts. If the PUT was what failed, the probe is kept and resent, not run again.
• 409 (the file changed while it was probed) and 404 (it is gone): dropped; the new version, if any, is listed again and probed.
• 400/422 (a bug on one side): logged at error and left on the same backoff.
• Server unreachable: logged once at warning, retried with the same backoff, and Server reachable again once it answers. The helper never exits because of the network.

It exits non-zero only for a bad configuration at startup. SIGTERM or SIGINT kills a running ffprobe (that file is retried next time) and exits 0 within a moment.

Log lines, at info: Probed per file (key, container, duration, resolution, hdr), Not identifiable; stored as failed per permanent failure (at notice), and Pass finished with probed, failedPermanently, deferred (and dropped, rejected, mostAttempts when non-zero). A pass that found nothing to do logs its summary at debug only, so an idle helper is quiet.

Running

Run exactly one helper per server, as a single replica next to it. Nothing stops a second one, but both would probe the same files; the contract expects one at a time.

Needs Swift 6 and ffmpeg (for ffprobe). Against the svc dev stack, in simpleplayer-svc:

docker compose -f dev/compose.yaml up -d    # SeaweedFS on :8333
dev/seed.sh                                 # upload the test clips
dev/run-server.sh                           # the server on :8080 (keep it running)
curl -X POST http://localhost:8080/api/admin/v1/library/scan

then here:

SERVER_URL=http://localhost:8080 swift run simpleplayer-helper
SERVER_URL=http://localhost:8080 LOG_LEVEL=debug swift run simpleplayer-helper

and check the result with curl http://localhost:8080/api/admin/v1/library: every media has a probe whose etag is its own.

From the Docker image (see below), the server and the presigned URLs must both be reached through the host, so start the server with S3_PUBLIC_ENDPOINT=http://host.docker.internal:8333 dev/run-server.sh instead, then:

docker build -t simpleplayer-helper .
docker run --rm -e SERVER_URL=http://host.docker.internal:8080 simpleplayer-helper

(host.docker.internal is Docker Desktop's and OrbStack's name for the Mac; on Linux add --add-host=host.docker.internal:host-gateway.) docker stop sends SIGTERM.

Tests

swift test
swift test --filter HelperAPITests     # one suite

The API client is tested against an in-process stub (Tests/HelperCoreTests/Support/StubServer.swift, a URLProtocol with its own host name per test). The runner is tested against Support/FakeLibrary.swift, which serves the three endpoints from state on such a stub (so files drop out of the list as they are submitted), with a scripted FakeProber and a ManualClock for the backoff. Nothing in the tests touches the network or the dev stack.

The prober tests need ffmpeg and ffprobe on PATH; without them that suite is skipped with a message saying so. They generate a few tiny clips (720p, 2160p, .mov, one tagged HDR, a text file and a 26 MB moov-at-the-end clip) once into $TMPDIR/simpleplayer-helper-clips-v1 and reuse them on later runs; delete that directory to regenerate. One clip is probed by URL from an in-process HTTP server that supports Range (Support/RangeServer.swift, on 127.0.0.1 and a free port), which checks that ffprobe fetched under a fifth of it.

Docker image

docker build -t simpleplayer-helper .

Two stages: Swift 6.4 on Debian trixie builds the binary with the Swift runtime linked in statically, and a debian:trixie-slim runtime gets it together with the ffmpeg package. It runs as the non-root user 10001, needs no volume and listens on nothing. The ffmpeg package makes the image large (about 850 MB).

To run it against the dev stack, see Running.

Layout

Sources/simpleplayer-helper/   main.swift: configuration, logging, the runner, signals
Sources/HelperCore/            everything else (see CLAUDE.md)
Tests/HelperCoreTests/         tests, the stub server, the range server and the clip fixtures
Dockerfile