GPU pools

What it is

One model on the host GPU, shared by many VMs at once.

A GPU pool loads a model on the host GPU once: Metal on macOS, Vulkan on Windows and Linux. Granted VMs call it over their network card with their own token. No VM holds a GPU device, so snapshots and warm pools work exactly as before.

Why it matters

Models run fastest on a GPU, and one GPU usually means one VM. A GPU pool turns a single GPU into a shared service for every sandbox on the host.

• Load once, serve many. One model in memory, parallel requests from many VMs.
• Per-VM access. Each VM gets its own grant, revocable at once.
• Every platform. Metal and Vulkan, the same commands everywhere.

Runs on

• Linux x86_64
• Linux arm64
• macOS
• Windows x64
• Windows arm64

How it works

Create a pool from a model file, then start VMs with --gpu-pool.

CLI

flash gpu info
flash gpu pool create --id llm --model /models/model.gguf --slots 4 --wait-ready 600
sudo flashnet --mode nat --vm-id vm-gpu-1 -- flash run --bundle BUNDLE --id vm-gpu-1 --vcpus 1 --mem-mib 512 --nic '{NIC}' --gpu-pool llm --detach
flash gpu grant ls --gpu-pool llm

MCP

flash_gpu(method: "POST", path: "/v1/gpu-pools", body: {id: "llm", model: "/models/model.gguf", slots: 4})
flash_gpu(method: "POST", path: "/v1/gpu-grants", body: {vm_id: "vm-gpu-1", gpu_pool: "llm"})

API

POST /v1/gpu-pools {"id": "llm", "model": "/models/model.gguf", "slots": 4}
POST /v1/gpu-grants {"vm_id": "vm-gpu-1", "gpu_pool": "llm"}
GET  /v1/gpu-grants?gpu_pool=llm

The result

Three VMs ask one model three different questions at the same time, and each gets its answer.

One GPU, one loaded model, many sandboxes served in parallel.