Operator QA Runbook
Field checks for media ingest, network reachability, ports, and stream validation with FFmpeg, GStreamer, ping, nc, and nmap.
Use this page when a camera, HoloVid tile, Magic URL, or projector panel looks wrong. The goal is to prove the path in layers before changing dashboard config: host reachability, open ports, protocol handshake, media decode, Restreamer process, then R2-D2 rendering.
Keep scans narrow and authorized. Use these commands only against fleet networks, cameras, Restreamer hosts, and services you are responsible for operating.
Variables
Set the target once so every command is repeatable in the incident notes.
export CAM_HOST=100.85.44.70
export RTSP_PORT=8554
export RTSP_PATH=/sancar
export RTSP_URL="rtsp://${CAM_HOST}:${RTSP_PORT}${RTSP_PATH}"
export RESTREAMER_HOST=r2d2-restreamer.innovationhub.vpn
export RESTREAMER_HTTP=8080
export RESTREAMER_API_URL="http://${RESTREAMER_HOST}:${RESTREAMER_HTTP}"
export RTMP_PORT=1935
export SRT_PORT=6000Check that the QA tools are installed before you start chasing the stream.
for tool in ffmpeg ffprobe gst-launch-1.0 gst-inspect-1.0 nmap nc; do
command -v "$tool" >/dev/null || echo "missing: $tool"
doneLayer 1: Host Reachability
Use ping to answer one question: can this host see the other host at all?
ping -c 4 "$CAM_HOST"
ping -c 4 "$RESTREAMER_HOST"If ICMP is blocked, a failed ping is not proof that the camera is down. Continue to port checks. If ping has packet loss or wildly unstable latency, note it before testing media; jitter often shows up later as decoder stalls.
On macOS and most Linux workstations, traceroute is useful when a site-to-site path
changed:
traceroute "$CAM_HOST"
traceroute "$RESTREAMER_HOST"Layer 2: Port Checks
Use nc for quick TCP checks. This does not prove the stream is valid; it proves the
socket is reachable.
nc -vz "$CAM_HOST" "$RTSP_PORT"
nc -vz "$RESTREAMER_HOST" "$RESTREAMER_HTTP"
nc -vz "$RESTREAMER_HOST" "$RTMP_PORT"UDP checks are weaker because UDP has no handshake, but they can still catch routing or firewall mistakes:
nc -uvz "$RESTREAMER_HOST" "$SRT_PORT"Use nmap when you need reason codes, service guesses, or a compact port inventory.
Keep the port list explicit.
nmap -Pn --reason -p 554,8554 "$CAM_HOST"
nmap -Pn --reason -sV -p "$RTSP_PORT" "$CAM_HOST"
nmap -Pn --reason -sT -p 8080,1935 "$RESTREAMER_HOST"
sudo nmap -Pn --reason -sU -p "$SRT_PORT" "$RESTREAMER_HOST"Good signs: open for the expected port, stable latency, and a service guess that
matches the role. Bad signs: filtered, host down, the wrong port open, or a
different service answering where RTSP/RTMP/SRT should live.
Layer 3: FFmpeg / FFprobe
Start with ffprobe over RTSP/TCP. TCP is the calmer diagnostic default because packet
loss does not masquerade as random frame damage.
ffprobe -hide_banner -v warning \
-rtsp_transport tcp \
-timeout 5000000 \
-show_streams \
-select_streams v:0 \
"$RTSP_URL"Compare with RTSP/UDP only after TCP works:
ffprobe -hide_banner -v warning \
-rtsp_transport udp \
-timeout 5000000 \
"$RTSP_URL"Decode ten seconds without writing a file. This is the cleanest "can this machine actually consume the camera?" test.
ffmpeg -hide_banner -loglevel info \
-rtsp_transport tcp \
-timeout 5000000 \
-i "$RTSP_URL" \
-t 10 \
-map 0:v:0 \
-f null -Grab a single frame when an operator needs a visual proof point:
ffmpeg -hide_banner -y \
-rtsp_transport tcp \
-timeout 5000000 \
-i "$RTSP_URL" \
-frames:v 1 \
/tmp/r2d2-camera-check.jpgValidate a Restreamer HLS output directly when the camera probes cleanly but the dashboard tile still looks wrong:
ffprobe -hide_banner -v warning \
"${RESTREAMER_API_URL}/memfs/<feed-id>.m3u8"Good signs: codec, width, height, frame rate, and steady frame counts. Bad signs:
401 Unauthorized, 404 Not Found, Connection timed out, repeated SPS/PPS errors,
or frame counters that stop moving.
Layer 4: GStreamer
Check the plugins first, especially on fresh operator laptops.
gst-inspect-1.0 rtspsrc
gst-inspect-1.0 rtph264depay
gst-inspect-1.0 rtph265depay
gst-inspect-1.0 srtsrcFor H.264 cameras:
gst-launch-1.0 -v \
rtspsrc location="$RTSP_URL" protocols=tcp latency=250 \
! rtph264depay \
! h264parse \
! fakesink sync=falseFor H.265 cameras:
gst-launch-1.0 -v \
rtspsrc location="$RTSP_URL" protocols=tcp latency=250 \
! rtph265depay \
! h265parse \
! fakesink sync=falseUse a local preview only on a workstation with a display:
gst-launch-1.0 -v \
rtspsrc location="$RTSP_URL" protocols=tcp latency=250 \
! decodebin \
! autovideosink sync=falseFor SRT receive checks, match the Restreamer mode, stream ID, and passphrase used by the feed. This example assumes caller mode with no passphrase:
gst-launch-1.0 -v \
srtsrc uri="srt://${RESTREAMER_HOST}:${SRT_PORT}?mode=caller" \
! tsdemux \
! h264parse \
! fakesink sync=falseKnown-Good Test Signals
When cameras are suspect, publish a generated signal. If the test signal works and the camera does not, the dashboard is probably fine.
Push a color-bar test pattern to RTMP with FFmpeg:
ffmpeg -re \
-f lavfi -i testsrc2=size=1280x720:rate=30 \
-f lavfi -i sine=frequency=1000:sample_rate=48000 \
-c:v libx264 -preset veryfast -tune zerolatency -b:v 2500k \
-g 60 -pix_fmt yuv420p \
-c:a aac -b:a 128k \
-f flv \
"rtmp://${RESTREAMER_HOST}:${RTMP_PORT}/rtmp-app/r2d2-qa.stream"Push a test pattern to SRT with FFmpeg. Add passphrase= or a feed-specific streamid
when Restreamer requires it.
ffmpeg -re \
-f lavfi -i testsrc2=size=1280x720:rate=30 \
-c:v libx264 -preset veryfast -tune zerolatency \
-f mpegts \
"srt://${RESTREAMER_HOST}:${SRT_PORT}?mode=caller&transtype=live&latency=20000&streamid=r2d2-qa,mode:publish"Push a color-bar test pattern to RTMP with GStreamer:
gst-launch-1.0 -v \
videotestsrc is-live=true pattern=smpte \
! video/x-raw,width=1280,height=720,framerate=30/1 \
! x264enc tune=zerolatency speed-preset=veryfast bitrate=2500 key-int-max=60 \
! flvmux streamable=true \
! rtmpsink location="rtmp://${RESTREAMER_HOST}:${RTMP_PORT}/rtmp-app/r2d2-gst-qa.stream live=1"R2-D2 / Restreamer Checks
These calls prove whether the dashboard can see what Restreamer sees. Run them from the dev host or an operator terminal with the right network path.
curl -fsS "http://localhost:3210/api/restreaming/status" | jq .
curl -fsS "http://localhost:3210/api/restreaming/cameras" \
| jq '.status, (.feeds[]? | {id, label, exec, inputAddress, lastLogline})'
curl -fsS "${RESTREAMER_API_URL}/api/v3/process" | jq '.[]? | {id, state, order}'If these fail but FFmpeg/GStreamer succeed from the same host, inspect
RESTREAMER_API_URL, authentication, and the dashboard server logs before changing
camera config.
Decision Table
| Symptom | Likely layer | Next command |
|---|---|---|
| Hostname does not resolve | DNS / operator network | nslookup "$RESTREAMER_HOST" |
| Ping works, RTSP port is closed | Camera service or firewall | nmap -Pn --reason -p "$RTSP_PORT" "$CAM_HOST" |
| RTSP TCP works, RTSP UDP fails | Firewall, NAT, or packet loss | Keep Restreamer on TCP or fix the path before enabling UDP |
| FFprobe sees codec but FFmpeg stalls | Decoder or packet timing | Run the ten-second ffmpeg -f null - test with -loglevel verbose |
| Test pattern renders, camera does not | Camera source | Capture ffprobe output and compare RTSP URL, credentials, codec, and transport |
| HLS probe works, browser tile is blank | Dashboard/browser path | Check /api/restreaming/cameras, browser console, and Magic URL token status |
| Restreamer API fails, direct media works | Control-plane path | Verify RESTREAMER_API_URL, service DNS, auth, and container logs |
Incident Notes
Record the exact command, timestamp, host, and result. A good note says which layer was proven healthy and which layer failed. That keeps the next operator from retesting the same thing and gives developers the right artifact when the issue is in R2-D2 itself.