Technology — Streaming Observability
Metrics, logs, and lifecycle events are all live streams on the same per-VM control socket. A caller subscribes once and gets pushed updates as they happen — no file to re-poll, no dashboard silently going stale.
flowchart LR Agent["Guest-agent
reports every 3s
(CPU/mem/swap/cache)"] --> Broadcast Host["Host-measured
disk/quota/NIC counters
(at frame arrival)"] --> Broadcast Broadcast["tokio::broadcast
channel"] --> SSE1["/metrics/events"] Broadcast --> SSE2["/logs/events"] Broadcast --> SSE3["/events (lifecycle)"] Broadcast --> SSE4["/guest/messages"] SSE1 --> Consumer["flash watch / flash_watch /
flash_watch_fleet"] SSE2 --> Consumer SSE3 --> Consumer SSE4 --> Consumer Broadcast -.slow consumer.-> Lagged["event: lagged
{missed: n}"]
lagged event with a count — never silently dropped data.Guest-reported figures (CPU, memory, swap, cache) arrive every 3 seconds over the control channel. Disk, quota, and NIC counters are measured host-side at frame arrival — no guest cooperation needed for the numbers that matter most for capacity planning. Both blend into one unified live view rather than two separate things an operator has to reconcile.
Each VM keeps two log streams (run — the workload's own stdout; console — raw kernel serial), each capped at 256 KiB with a head+tail scheme: the first 64 KiB frozen forever, the most recent 192 KiB rolling. The gap between them carries an explicit --- N bytes dropped --- marker and a truncated: bool flag in the API response — a reader can always tell "quiet" apart from "truncated."
Log cursors (?since=) are absolute line counts over the stream's entire history, not offsets into the current buffer — so a poller walking forward through a long-running VM's log never stalls once the buffer reaches steady state and starts rolling. This is a deliberate design choice, not an incidental property: an earlier flat-buffer approach couldn't make this distinction at all.
Because every stream lives on the same per-VM socket already used for control, watching hundreds of VMs is hundreds of independent SSE subscriptions, not a separate monitoring subsystem to stand up and keep in sync with the fleet. flash_watch_fleet in flash-mcp does exactly this — one real OS thread per VM, results prefixed by VM id, concurrency-capped specifically because the same SSE-connection-exhaustion pattern once froze an earlier fleet dashboard and that lesson is now enforced in code, not just remembered.