Inference & model serving¶

SilentSwarm can serve the models it trains for interactive use: pick a trained model, load it onto a GPU node, chat with streaming tokens, and call it from an OpenAI-compatible API. This document describes the runtime; the staged rollout is tracked in the project's internal planning notes.

Design principle: the leader stays torch-free¶

The leader is the control plane and never imports torch (keeps its image small and the public tunnel healthy). All inference compute runs on a fellow (GPU node). The leader is a gateway: it owns the model catalog, decides which fellow serves a model, and proxies/streams requests — exactly like training dispatch (signal_hub WS for control + outbound-only HTTP for data, so it works behind NAT/Cloudflare).

Components¶

Layer	Module	Role
Serving core	`runtime/torch/serving.py`	reconstruct a servable model → `.generate()` + streaming
Servable layout	(same)	`<model>/servable.json` + `shards/stage-*/weights.pt` + `tokenizer/`
Registry	`leader/serving/model_registry.py`	discover servable models (torch-free)
Serving subprocess	`runtime/torch/inference_server.py`	one torch process per loaded model (JSONL protocol)
Fellow manager	`fellow/inference_manager.py`	load/unload, capacity cap, shard fetch, idle auto-unload
Fellow dispatch	`fellow/inference_dispatch.py`	drive the subprocess, per-request metrics
Broker	`leader/serving/inference_broker.py`	NAT-safe request/response fan-out
Tokens	`leader/accounts/api_tokens.py`	hashed bearer API tokens
Pipeline (E7)	`runtime/torch/pipeline_serving.py`	distributed generation with a compression boundary
UI	`leader/templates/inference.*`	`/ops/inference` chat + dropdown + API panel

Request flow (chat)¶

browser / curl ──POST /v1/chat/completions──► leader gateway
                                               │ pick serving fellow (heartbeat loaded_models)
                                               ▼
                       broker.submit(fellow_id, request)  ── fellow long-polls
                                                              GET /api/v1/infer/requests/{fellow_id}
                       fellow runs generate(stream) ──► POST /api/v1/infer/responses/{id} (token/done)
 browser ◄── SSE deltas ◄── broker.responses(id) ◄────────────┘

Client disconnect drops the response queue; the fellow's next frame POST returns 410, which aborts forwarding (soft-cancel; hard subprocess interrupt is future work).

HTTP API¶

GET /api/v1/models · GET /api/v1/models/{id} · GET /api/v1/models/{id}/archive
POST /api/v1/models/{id}/load · POST /api/v1/models/{id}/unload · GET /api/v1/models/{id}/status
POST /api/v1/chat/completions and POST /v1/chat/completions (OpenAI shape, stream supported)
POST/GET/DELETE /api/v1/api-tokens
Fellow transport: GET /api/v1/infer/requests/{fellow_id}, POST /api/v1/infer/responses/{request_id}

CLI¶

swarm models                       # list servable models
swarm model-load <id> --wait       # load onto a fellow, wait until ready
swarm infer <id> "your prompt"     # one-shot completion
swarm model-unload <id>            # free VRAM

Authentication¶

Bearer tokens (Authorization: Bearer sk-swarm-…) authorize the chat endpoints. Tokens are minted server-side, shown once, and stored only as SHA-256 hashes. Auth is enforced when any token exists or SWARM_REQUIRE_API_TOKEN=1.

Distributed (pipelined) inference — E7¶

pipeline_serving.PipelineGenerator runs autoregressive generation across pipeline stages on different devices, with the trained compression module on the boundary, so the activation crossing the boundary is the narrow code. It reports per-token wire bytes; no-compression sends hidden_size × 2 bytes/token, a learned bottleneck materially less. Greedy pipeline output matches single-node output. Cross-node transport reuses the training ZMQ/relay channels.

Hardening & observability — E8¶

Capacity: one model per fellow by default (SWARM_MAX_LOADED_MODELS), oldest evicted on overflow; generation is serialized per model.
Idle auto-unload: a loaded model is stopped after SWARM_MODEL_IDLE_UNLOAD_S seconds without a request (default 900), and its VRAM goes back to the driver. The next request pays the reload. A generation in flight is never reaped, however long it streams. Set the variable to 0 on a dedicated inference node where reload latency matters more than the memory.
Metrics: per-model tokens_per_s, last_ttft_s, requests, idle_s, vram_bytes surface in the heartbeat loaded_models and GET .../status.

VRAM lifecycle¶

VRAM is only ever borrowed, and every path that ends a fellow's involvement with a model gives it back:

Event	What releases the memory
`UNLOAD_MODEL` signal, or `swarm model-unload`	the serving subprocess is asked to exit, then killed after 10 s
No request for `SWARM_MODEL_IDLE_UNLOAD_S`	the fellow's idle reaper stops the subprocess
A second model loaded past `SWARM_MAX_LOADED_MODELS`	the oldest model is evicted before the new one starts
`STOP` / `PURGE` signal, fellow shutdown	every serving subprocess is stopped
`UPDATE` signal	subprocesses are stopped before the `execv` re-exec

The last row is the one that is easy to get wrong: a serving subprocess is a child of the fellow but not in its process group, so an exit that skips the shutdown leaves it running and holding a GPU that the leader now believes is free. Nothing short of killing it by hand gets that memory back.

Training is not on this list because it never needed to be: a torch job runs in its own subprocess that exits when the job ends, which releases the CUDA context with it.

Note that a loaded model's reported vram_free_gb is a high-water mark, not a live reading - torch's caching allocator holds freed KV-cache blocks rather than returning them to the driver. That is why the leader budgets the cache arithmetically (telemetry/vram_budget.py) instead of trusting free VRAM.

Environment variables¶

Var	Default	Meaning
`SWARM_SERVABLE_MODELS_ROOT`	`.swarm/servable_models`	leader: where servable models live
`SWARM_FELLOW_MODEL_CACHE`	`~/.silent-swarm/models`	fellow: downloaded model cache
`SWARM_MAX_LOADED_MODELS`	`1`	fellow: concurrent loaded models
`SWARM_MODEL_IDLE_UNLOAD_S`	`900`	fellow: idle TTL auto-unload (`0` disables)
`SWARM_REQUIRE_API_TOKEN`	unset	leader: force bearer auth
`SWARM_API_TOKENS_FILE`	next to state DB	leader: token store path