What it is

A second playback engine, opt-in per tile, that delivers ~500–800ms glass-to-glass latency instead of HLS's 2–6s. The use case is the public viewer (/p/<token>) — the surface most likely to drive a kiosk display or a remote operator watching a single feed, where HLS's segment-based delay is the wrong default.

It is not a frame decoder. It does remux only: ingest upstream MPEG-TS (H.264 + AAC), transmux to fragmented MP4 in the browser, feed Media Source Extensions to a <video> element. The browser handles decode; the wasm only re-shapes the container.

While it remuxes, it also taps the KLV PID (private MPEG-TS PID carrying MISB ST 0601 telemetry) and surfaces parsed fields to the HUD overlay.

Why "custom-chrome wrapper" (not Vidstack stable)

The original plan used @vidstack/react. Mid-design we discovered that Vidstack stable's React adapter is React-18-only. The dashboard is on React 19 (see Next.js + React). Rather than block on Vidstack's React-19 release, we wrote a thin custom chrome wrapper that:

• Hosts the <video> element in a container the chrome can fullscreen (so the HUD button, snapshot button, snapshot-history button, and KLV overlay all survive the fullscreen API — native <video controls> strips sibling DOM during fullscreen).
• Owns the play/pause/mute/scrub buttons with a uniform look across HLS-mode and low-latency-mode tiles.
• Is engine-agnostic — the same chrome wraps either an hls.js instance or the wasm transmux engine.

Engines, side by side

Architecture

flowchart LR
  subgraph Upstream
    R[Restreamer datarhei/core]
  end
  subgraph Browser["Browser tab"]
    F[fetch streaming .ts]
    W[@ffmpeg/ffmpeg wasm]
    K[KLV demux tap]
    H[HUD overlay state]
    MSE[MediaSource buffer]
    V[<video> element]
    CC[Custom chrome wrapper]
  end
  R -->|.ts MPEG-TS, all PIDs| F
  F --> W
  W -->|fragmented MP4| MSE
  W -.->|KLV PID| K
  K --> H
  MSE --> V
  CC -.->|controls + fullscreen| V
  H -.->|overlay render| CC

Capacity cap

Some hosts can run multiple concurrent wasm transmux contexts; cheaper kiosks can run one. The cap is counted, not binary:

• Default 1 active tile per browser tab.
• Override at panel level up to gridCols × gridRows.
• URL query ?hud=<slotI[,slotI…]> enumerates which slots get the low-latency engine on a kiosk URL.
• Per-route persistence in localStorage['r2d2.player.hudSlots'] (array of slot indices).

When the cap is hit, additional tiles stay on HLS until another tile yields its slot (unmount, navigate away, page hidden).

HUD controls

The chrome surfaces three controls related to HUD/low-latency mode:

The suppress flag persists across panel swaps — it's a property of the feed (via /api/restreaming/feed-meta), not the tile.

KLV demux + overlay

The wasm engine reads any KLV PID present on the MPEG-TS stream and parses MISB ST 0601 fields. The HUD overlay reads from a per-tile zustand slice keyed by feed ID:

// see src/lib/playerHud.ts and src/lib/hudVariants.ts
type HudState = {
  feedId: string;
  klv: {
    timestampMicros?: bigint;
    platformHeading?: number;
    platformPitch?: number;
    sensorLat?: number;
    sensorLon?: number;
    sensorAltMeters?: number;
    // ...other ST 0601 fields as needed
  };
  latencyMs?: number;
  firstFrameMs?: number;
};

hudVariants.ts selects which fields render (and how) based on the feed's platform — a drone gets attitude + sensor pose, a fixed camera gets only timestamp + status.

Restreamer KLV passthrough — required upstream change

Restreamer's current .ts output strips the KLV PID. Until the KLV-passthrough mod ships, the wasm engine reads video only and the KLV overlay shows — for telemetry fields. The mod preserves every PID including KLV without re-encoding.

This is tracked as a separate piece of upstream work; the player engine is forward- compatible — once the mod ships, KLV fields populate automatically.

Source protocol decision

Single path, via Restreamer. No client-side gateway or sidecar. The browser fetches http://r2d2-restreamer.innovationhub.vpn:8080/<feed-id>.ts directly, streaming.

Considered and rejected:

• WebRTC — would require a SFU and signaling. Adds an operational surface; the benefit (extra ~200ms) isn't worth it for kiosk video.
• MPEG-DASH — same segment-based latency profile as HLS. Doesn't solve the problem.
• HLS-LL with #EXT-X-PART — depends on Restreamer emitting partial segments, which datarhei/core doesn't do today.

Threading: single-threaded first

@ffmpeg/ffmpeg v0.12+ has both single-threaded and multi-threaded (SharedArrayBuffer) builds. Multi-threaded needs COOP/COEP response headers; we'll scope those strictly via middleware when/if we adopt it. The current plan is single-threaded first, then measure whether it's enough for 1080p30 transmux. If it isn't, multi-threaded with narrow COOP/COEP gets enabled for the player route only.

Fallback chain (planned)

The opt-in engine is allowed to fail. The chrome wrapper enforces a watchdog:

stateDiagram-v2
  [*] --> Init
  Init --> Loading: user enables HUD
  Loading --> Playing: first frame ≤ 3s
  Loading --> FallbackHLS: first frame > 3s
  Playing --> Loading: error / stall
  FallbackHLS --> Init: tile remount

3s first-frame watchdog → silently fall back to HLS for that tile, mark the engine "unavailable" for the session. The HUD toggle stays visible but disabled, with a tooltip explaining the fallback.

Where it lives in src/ (current)

See also

• hls.js — the default engine
• Restreaming — the plane this engine serves
• Architecture — where Restreamer sits in the cluster