Fleet controller and live migration

The FlashVMM fleet controller, flash controller, is a placement and migration service that manages many FlashVMM hosts through each host's public flashhostd API, reached locally, over SSH or over mutual TLS, placing FlashVMs by best fit on predicted free room, merging every host's events and metering, and moving running VMs between hosts, over a strict resource-CRUD REST API (OpenAPI 3), the flash controller CLI and the flash_controller MCP tool.

The controller is the same flash binary in another role and needs no virtualization of its own, so it runs anywhere. It holds no private channel into a host: everything it does is a call any operator could make to that host's host API, and each host's own admission has the final word.

Key facts

How it works

Hosts and transports

POST /v1/hosts {name, addr} adds a host, reaching it once on the spot and returning the reason in one sentence when it cannot. Three transports carry the same host API:

• local: this machine's flashhostd socket or pipe.
• ssh://user@host[:port]: an SSH login with keys, batch mode and a shared control connection; on the far side flash host relay hands each request to that machine's flashhostd. Nothing else to set up.
• host:port: the host's mutual-TLS listener. flash host ca create makes the CA, flash host cert issue issues each host's certificate and, with --client, the controller's; the controller presents its certificate (PUT /v1/controller) and the host accepts only certificates its CA signed.

A poller per host reads its summary every second and its events with backoff, so the controller always knows each host's capacity, free room, pressure, platform and guest bundle.

Best-fit placement

POST /v1/flashvms takes the same body as a host's POST /v1/host/flashvms. Candidates are connected hosts, not under memory pressure, whose free memory covers the ask; memory is the hard limit and CPU, being time-shared, only ranks. Each candidate scores the dot product of the ask and its free room, both normalized by its capacity, and ties go to the host with fewer VMs. A host's figures are believed only within its own capacity. The controller then creates the VM on the best host; if that host's admission refuses, it moves to the next. When none accepts, the reply is 409 fleet_full with each host's reason.

Merged events and usage

GET /v1/events merges every host's events, each tagged with its host, plus the controller's own host_added, host_removed, host_connected, host_disconnected and placed, as a cursor-based long poll. GET /v1/usage runs the same metering query on every host, tags each VM with its host, sums totals across hosts (bytes exactly), counts a host added under two names once, and lists any host that did not answer in unreachable.

Moving a running VM

POST /v1/moves {flashvm, to?} moves a running or sleeping VM to another connected host of the same platform and guest bundle, named or chosen by best fit. The move runs in the background:

1. sleeping: the VM sleeps on the source host, its memory and device state written to files, and the source copy is marked with the move id so nothing else wakes or moves it;
2. copying: the target begins an incoming VM with the record, platform and the bundle's SHA-256 fingerprint, reserving room, and the controller copies every file in 4 MiB chunks, reporting copied_bytes of total_bytes;
3. waking: the target installs the files as a sleeping VM and wakes it under the same name, carrying on from the instant it slept;
4. done: the source copy is removed.

A failure at any step wakes the VM back on the source. Sleep and wake are the same mechanisms flash sleep and flash wake use on one host; see snapshots.

Interfaces

CLI

flash controller serve --state-dir /var/lib/flash-controller
flash controller hosts add --name local --addr local
flash controller hosts add --name rack2 --addr ssh://ops@10.0.0.12
flash controller hosts add --name edge1 --addr 10.0.1.5:7443
flash host cert issue --ca ./ca --name controller --client --out ./certs
flash controller vms create --vcpus 2 --mem-mib 1024 --entry "python3 job.py"
flash controller vms
flash controller move job1 --to rack2
flash controller moves
flash controller events
flash controller usage

MCP

flash_controller(method: "POST", path: "/v1/hosts", body: {"name": "rack2", "addr": "ssh://ops@10.0.0.12"})
flash_controller(method: "POST", path: "/v1/flashvms", body: {"shape": {"vcpus": 2, "mem_mib": 1024}, "entry": "python3 job.py"})
flash_controller(method: "POST", path: "/v1/moves", body: {"flashvm": "job1", "to": "rack2"})

REST

POST /v1/hosts {"name": "edge1", "addr": "10.0.1.5:7443"}
PUT  /v1/controller {"cert": "/etc/flash/controller.pem", "key": "/etc/flash/controller.key", "ca": "/etc/flash/ca.pem", "placement": "best_fit"}
POST /v1/flashvms {"shape": {"vcpus": 2, "mem_mib": 1024}, "entry": "python3 job.py"}
GET  /v1/events?since=0&wait_s=30
GET  /v1/usage?from=1761955200
POST /v1/moves {"flashvm": "job1", "to": "rack2"}
GET  /v1/moves/{move_id}

Contract an agent must know: placement with no host able to take the VM returns 409 fleet_full with hosts mapping each host to its reason; a name already used on the chosen host returns 409 flashvm_exists; a VM id on several hosts returns 409 ambiguous_flashvm unless host picks one; POST /v1/moves returns 202 with the move, or 409 not_movable, same_host, unknown_host, arch_mismatch, no_target, move_in_progress or host_unreachable; adding a host that cannot be reached returns 502 host_unreachable with the reason.

Guarantees

• Built on the public API. Every action is a call to a host's own host API, which keeps the final word on admission.
• Best fit on predicted room. Memory decides eligibility, ask-times-free decides rank, and a refusal moves on to the next host.
• Three transports, one contract. Local, SSH keys or mutual TLS, chosen per host.
• One fleet view. Events and metering from every host, merged and tagged with the host.
• Moves return home on failure. A VM that cannot complete its move wakes back where it was.