Inference & model serving

SilentSwarm can serve the models it trains for interactive use: pick a trained model, load it onto a GPU node, chat with streaming tokens, and call it from an OpenAI-compatible API. This document describes the runtime; the staged rollout is tracked in the project's internal planning notes.

Design principle: the leader stays torch-free

The leader is the control plane and never imports torch (keeps its image small and the public tunnel healthy). All inference compute runs on a fellow (GPU node). The leader is a gateway: it owns the model catalog, decides which fellow serves a model, and proxies/streams requests — exactly like training dispatch (signal_hub WS for control + outbound-only HTTP for data, so it works behind NAT/Cloudflare).

Components

Layer	Module	Role
Serving core	runtime/torch/serving.py	reconstruct a servable model → .generate() + streaming
Servable layout	(same)	<model>/servable.json + shards/stage-*/weights.pt + tokenizer/
Registry	leader/serving/model_registry.py	discover servable models (torch-free)
Serving subprocess	runtime/torch/inference_server.py	one torch process per loaded model (JSONL protocol)
Fellow manager	fellow/inference_manager.py	load/unload, capacity cap, shard fetch, idle auto-unload
Fellow dispatch	fellow/inference_dispatch.py	drive the subprocess, per-request metrics
Broker	leader/serving/inference_broker.py	NAT-safe request/response fan-out
Tokens	leader/accounts/api_tokens.py	hashed bearer API tokens
Pipeline (E7)	runtime/torch/pipeline_serving.py	distributed generation with a compression boundary
UI	leader/templates/inference.*	/pools/{owner}/{pool}/inference chat + dropdown + API panel

Request flow (chat)

browser / curl ──POST /v1/chat/completions──► leader gateway
                                               │ pick serving fellow (heartbeat loaded_models)
                                               ▼
                       broker.submit(fellow_id, request)  ── fellow long-polls
                                                              GET /api/v1/infer/requests/{fellow_id}
                       fellow runs generate(stream) ──► POST /api/v1/infer/responses/{id} (token/done)
 browser ◄── SSE deltas ◄── broker.responses(id) ◄────────────┘

Client disconnect drops the response queue; the fellow's next frame POST returns 410, which aborts forwarding (soft-cancel; hard subprocess interrupt is future work).

HTTP API

• GET /api/v1/models · GET /api/v1/models/{id} · GET /api/v1/models/{id}/archive
• POST /api/v1/models/{id}/load · POST /api/v1/models/{id}/unload · GET /api/v1/models/{id}/status
• POST /api/v1/chat/completions and POST /v1/chat/completions (OpenAI shape, stream supported). Beyond the OpenAI fields: chat_template_kwargs (only enable_thinking is passed on - false turns a Qwen3 model's reasoning off), and the reply (the last chunk when streaming) carries swarm.timing when the serving fellows measured where the time per token went: stage0_ms, link_ms, stage1_ms, prefill_ms, bytes_per_token, route. finish_reason is length when max_tokens cut the reply off, else stop.
• Fellow → leader: POST /api/v1/infer/responses/{request_id} with token frames - batched by fellow/token_batcher.py (texts lists the tokens, text joins them for older leaders) - then one done frame with completion_tokens, prompt_tokens, finish_reason and timing, or an error frame with an optional code.
• GET /api/v1/models/{id}/performance - throughput, stages with their GPUs, relay links and the time split per token over the last 15 minutes (the Inference page's Performance view; MCP models_performance)
• POST/GET/DELETE /api/v1/api-tokens - POST takes {"label", "scopes"}; chat needs the operate scope, listing models needs read (see API token scopes)
• Fellow transport: GET /api/v1/infer/requests/{fellow_id}, POST /api/v1/infer/responses/{request_id}

CLI

swarm models                       # list servable models
swarm model-load <id> --wait       # load onto a fellow, wait until ready
swarm infer <id> "your prompt"     # one-shot completion
swarm model-unload <id>            # free VRAM

Authentication

Bearer tokens (Authorization: Bearer sk-swarm-…) authorize the chat endpoints. Tokens are minted server-side, shown once, and stored only as SHA-256 hashes. Auth is enforced when any token exists or SWARM_REQUIRE_API_TOKEN=1. An expired but unrevoked token still counts: the hour-long access tokens an MCP connector gets through OAuth expire while the connection can still be refreshed, and enforcement must not lapse each time a connector sits idle. Revoking every token (or letting the leader prune an expired one, 31 days after it expired) returns chat to the open first-run mode; set SWARM_REQUIRE_API_TOKEN=1 to rule that out.

Distributed (pipelined) inference — E7

pipeline_serving.PipelineGenerator runs autoregressive generation across pipeline stages on different devices, with the trained compression module on the boundary, so the activation crossing the boundary is the narrow code. It reports per-token wire bytes; no-compression sends hidden_size × 2 bytes/token, a learned bottleneck materially less. Greedy pipeline output matches single-node output. Cross-node transport reuses the training ZMQ/relay channels.

Across fellows (runtime/torch/distributed_inference_server.py) stage 0 holds the tokenizer and stage 1 samples. Stage 0 stops on the model's end-of-turn ids (tokenizer and generation config) and ends the session; it times every token (its compute, the push, the wait for the token) and reports the split with the result. On wire 2 its first frame names the request's sampling (temperature, top_p, top_k, min_p; the leader has filled in the model's defaults, leader/serving/sampling.py) for stage 1, which samples with runtime/torch/sampling.py like every other serving path. Stage 1 returns its compute time with each token (compute_ms) - header extras an older peer ignores, so mixed versions keep working, without the new numbers. The leader's relay counts bytes and frames per lane (leader/relay/stats.py).

More than two stages (XL5). A model trained over more than two stages serves over as many by default (the num_stages its servable records), or over whatever num_stages the load asks for. The stages form a ring. The hidden state goes 0 -> 1 -> ... -> last and the sampled token goes from the last stage straight back to stage 0, so a token costs one hop per stage, not a round trip through every middle stage. On the relay, stage 0 sends on act, middle stage i on act<i>, and the last stage sends the token on tok, which stage 0 receives (ring_channels). For two stages that is the act / tok pair they always used. A middle stage (StageInferenceRuntime.run_middle) has no embeddings and no head. It decodes what comes in with the receiver half of the boundary before it, runs its blocks with its own KV cache, and encodes what goes out with the sender half of the boundary after it. It passes on the request's sampling (in the first frame, for the last stage) and the end of the session. Direct links become a ring too: every stage listens on its own serving port and dials the next stage's (the last stage dials stage 0's), each as a DuplexLink (pipe/link/duplex.py). A middle stage tags what it passes on with the session id it received, and the relay stays the fallback. Fellows declare serve-n-stage; the leader refuses a ring that includes a fellow without it and names that fellow.

Which shards a serving stage reads. A model records where training cut it (boundaries in servable.json, from XL2 on). Served with as many stages as it trained with, it splits at those boundaries. Each serving stage then reads exactly one trained shard, which the fellow that trained it already has, and every learned bottleneck sits where it learned. Served with a different number of stages, each serving stage reads every trained shard its layers overlap: the leader sends them as shards and the fellow fetches the archive when it lacks one. A boundary that training did not have is served uncompressed. Before this, a serving stage read only its own stage's shard. A split that differed from training's ran some layers with base weights instead of trained ones. A model that records no split (exported before XL2) serves only over the number of stages it was exported with; any other count is refused.

Only the stage's slice, quantized when asked (XL6). A serving stage used to rebuild the whole base model in its compute dtype and lay its trained shard over it, which is 240 GB per stage of a 120B model. Now stage_loader.load_serving_stage loads only the stage's layers of the base checkpoint, downloading only the files that hold them, as training does (XL3). It then lays over them the trained shards' weights that fall in those layers. With a weight quantization, both steps quantize as they load (NF4 or int8, bit-identical to a from_pretrained quantized load of the trained model). LoRA adapters shipped beside a shard (export: adapters) are merged into a full-precision stage as before. Over a quantized stage they stay separate, so a QLoRA run serves as it trained: a 4-bit base plus 16-bit adapters, without a second rounding.

A model serves with the quantization it trained with by default (the servable manifest's quantization, recorded at export). A load can ask for another with quantization ({"method": "nf4"}, {"bits": 8}, or {"method": "none"} for full precision). The leader prices quantized weights at 0.55 (4-bit) or 1.05 (8-bit) bytes per parameter of the base checkpoint, read from its safetensors headers. It places the load only on fellows that declare serve-quantized; with too few of them, the model serves in full precision rather than overrunning memory the estimate promised. Single-node serving takes the same path (serving.load_servable_model).

Faster 4-bit decode (XL7). bitsandbytes NF4 dequantizes a whole weight for every matmul. That is fine for training's batches and slow for decoding one token at a time. {"method": "int4-fast"} (or "backend": "torchao") serves 4-bit weights through torchao's int4 weight-only kernels instead, which multiply the packed weight directly. It is for serving only (training refuses it) and opt-in. torchao is an optional install, not in requirements-runtime.txt: a fellow that finds it, and whose GPUs are all Ampere or newer (the kernel's floor), declares serve-int4-fast, and the leader sends the fast kernels only to such fellows. With too few of them the load stays 4-bit on bitsandbytes. torchao quantizes on the GPU, so a stage is quantized block by block as each block moves to its card (fast_kernels.place_and_quantize), and the stage's 16-bit copy never sits in VRAM. Trained adapters are merged first, since the int4 kernels take none. Where the kernels cannot run after all (no CUDA card, say), the stage reloads as bitsandbytes NF4 rather than failing. Which backend is faster on which card is what the XL0 spike measures.

Several requests in flight (XL8). With one request, a ring of N stages has one position in it at a time, so N - 1 stages wait on every token. When every stage's fellow declares serve-concurrent and the stages speak wire 2, the leader sets the model's max_in_flight and budgets a KV cache per request in flight when it admits the load. It aims for two per stage (SWARM_SERVE_IN_FLIGHT otherwise) and lowers that until every stage's cards hold that many caches; only a stage that cannot hold one refuses the load. One per stage, the old default, kept every stage busy only while no frame was on the link. Over the internet relay a frame's trip was a third of a token's time, and a third and further user queued at the leader for whole replies before seeing a first token. Every stage then serves interleaved sessions on one persistent link per neighbour (runtime/torch/concurrent_serving.py). Each frame names its session, and every stage keeps a KV cache per session, dropped on that session's end frame. Stage 0 admits requests up to the limit and sends the next position of whichever session's token just came back, so while the last stage samples one request, stage 0 already runs another's. A single reply is no faster; the pipeline serves up to N times the tokens. On the fellow, every request's events go to their own queue and the stage process says in its ready event how many it takes. The intake loop serves requests as tasks (SWARM_SERVE_MAX_IN_FLIGHT bounds how many it holds). Stage 0 uses the direct link only after one throwaway position has gone all the way round it (probe_ring), which proves the last stage's hop back too, and the relay otherwise; it does not switch per request in this mode. A frame that fails on a later stage ends only its own request, at once. Each reply still carries its timing (stage 0's compute, the last stage's, the link). In this mode the link share also holds the time a position waited behind other requests' positions.

Admission by tokens. A KV cache sized for the longest conversation on every request in flight wastes most of it: on two home PCs, 4,096 tokens a request left room for only 4 at once, while chats use a few hundred. When every stage's fellow declares serve-kv-budget, the leader sizes the in-flight count from a typical chat instead (SWARM_SERVE_TYPICAL_TOKENS, 1,024; four per stage at most) and plans a token budget per stage, transport.kv_tokens. It is in_flight x 1,024, never below one SWARM_SERVE_CONTEXT_TOKENS conversation, and grows into spare VRAM up to a full context per request in flight. Stage 0 holds every session to it (run_stage0_concurrent): a session reserves its prompt plus its max_new_tokens from admission to its end, the most its cache grows to, and the next request in line waits until it fits beside the others. A session that finishes or fails gives its tokens back. A request that needs more than the whole budget is refused before it starts, with code: "context_too_long" and the numbers ("this conversation needs 4,312 tokens (...); the pool serving this model holds 4,096"). The leader answers that with a 413 (context_length_exceeded in a stream), and its serving metrics count it as too_long, not as a failure. A batched decode step pads every session's cache to the longest one for one layer at a time, so beside a long conversation a batch stops growing where that copy would pass the budget (_fits_padded); the later stages get the budget too and do the same. The serial and one-stage loops refuse what cannot fit the same way. A fellow from before admits by count alone, so with one in the pipeline the leader plans a full context per request, as before.

The leader's plan rests on the free VRAM a node reported before the load and an estimate of the weights, and on stage 0 of the 14B benchmark (float32 on two 4 GB cards) that was optimistic: a 4,000-token conversation ran a card out of memory in its prompt. So each stage process checks the plan once it has loaded and warmed up (measured_kv_tokens): per card, 85 % of what is free (the driver's free plus torch's cached, unused blocks) less 300 MiB, divided by what one token costs there - its layers' keys and values plus one prompt piece's attention scores against it. The smallest card sets the stage's budget, which it reports as kv_tokens in its ready event and status (with kv_tokens_planned) and logs when it is lower than planned. Stage 0 admits by its own measured budget; a later stage uses its own to cap its batches.

Batches stay whole across the link. A batched step sends one frame per session, back to back, and they arrive a few milliseconds apart. A stage that started on whatever had arrived split the batch, and on weight-bound cards every part costs nearly a whole forward: at 8 requests in flight the two PCs served 5.8 tokens/s, less than at 4. Batched frames and the tokens answering them now carry batch (how many were sent together; older peers ignore it), and the receiving stage waits up to 50 ms (COALESCE_S) for the rest before it runs them.

One group per stage. Whole, one batch of every session still left each stage idle while the other ran it: stage 0 on two GTX 1050 Ti computes in float32, and its batch of 8 took 468 ms against 112 ms for one session, so bigger batches buy it nothing while stage 1 waits. Stage 0 now splits the sessions one token on into groups of ceil(in flight / stages) (_group_size, _decode_batches): while stage 1 runs one group, stage 0 runs the next. The size comes from all sessions in flight, not the ones ready at that moment, so a group that comes back together is not split again. SWARM_SERVE_MAX_BATCH still caps a batch, and what is beyond the cap now batches too instead of going one session at a time.

Small batches on older cards. The per-stage stats lines ([stage0 stats 30s] batch_size=... batch_ms=... wait_ms=..., one per 30 s of traffic in the fellow log) showed the rest: in groups of 4, stage 1 (two GTX 1660 SUPER, NF4) took 671 ms per batch against about 120 ms for one session, and was busy the whole time while stage 0 waited. Batches of 2 cost little more than one. So the leader caps the decode batch for the whole pipeline (transport.max_batch, serving_max_batch): at 2 when any of its GPUs is older than Ampere (compute capability below 8.0), with no cap otherwise, and as SWARM_SERVE_MAX_BATCH on the leader says when that is set. On the two PCs, 8 users went from 5.7 to 10.5 tokens/s in total. Batched decode. A decode step reads every weight of a stage for a single token, and NF4 dequantizes each one for every matmul. So when several sessions each need their next position, a stage now runs them together in one forward (StageInferenceRuntime.first_positions, middle_positions, last_positions) and reads the weights once. Each session keeps its own KV cache. For one forward, _BatchedCache hands each attention layer the sessions' keys and values, left-padded to the longest with the padding masked, and gives each row its own position. Afterwards it appends each session's new position to that session's cache, so sessions join and leave a batch freely. Stage 0 batches the sessions whose tokens came back, taking every reply already waiting before it computes. The later stages take every frame already waiting and batch the decode positions among them; a session's other frames (its end, say) keep their order behind it. Prompts, and pieces of prompts, still go one at a time. Batching needs attention-only blocks under sdpa or eager attention: a hybrid model's linear-attention layers keep a recurrent state per session and stay one at a time. SWARM_SERVE_MAX_BATCH caps a batch (1 turns batching off). Every token is what the session would get alone, and a test holds that.

The relay, pipelined and bundled. Through the leader's relay, each frame was one HTTP request each way. A stage pushed and popped on the thread it computed on, so every position paid two round trips to the relay on top of its compute, whatever was in flight. Over a 21 ms internet path that capped Qwen3-14B across two PCs at about 4.5 tokens/s; exp/spikes/serving/relay_sim.py reproduces the cap with sleeping stages and a delaying proxy, and shows it follows 1 / (compute + 2 x round trip). A concurrent ring now uses PipelinedRelayLink (pipe/link/relay_link.py): - A sender thread pushes in order, and a receiver thread always holds a pop open, so a stage computes while its frames travel. - Frames that queue up travel together. A push may carry several (X-Relay-Bundle: 1), and a pop takes up to 32 at once (bundle=32), so a batch's tokens arrive as a batch rather than one per round trip. - The leader says X-Relay-Bundles: 1 on every relay answer. A client bundles only after seeing it, and a pop answered without bundle headers is read as a single frame, so old and new ends interoperate. - A failed push is not retried (a duplicate frame would run a position twice). It breaks the link at once, so the peer's request fails then rather than after its timeout.

Long prompts in pieces. A prompt used to cross every stage as one position. Attention's scores grow with the square of its length, so 1,500 tokens of Qwen3-14B ran a 4 GB stage-0 card out of memory, and the last stage put every prompt position through the head (about 450 MB of float16 logits for 1,500 tokens) to sample from one. Stage 0 now sends a prompt in pieces of SWARM_SERVE_PREFILL_CHUNK tokens (256; 0 sends it whole). Each piece's keys and values land in every stage's cache and the next piece attends to them, so the reply is the same token for token. The last stage samples after every piece, as it does after every position; stage 0 receives those answers without reporting them, and only the prompt's last piece starts the reply. The head now reads only the last position everywhere. Nothing changes on the wire, so a stage on an older fellow serves a prompt in pieces as well. A reply's prefill_ms is the whole prompt's: wall time from its first piece to the first token.

Stage 0 does not wait for each piece's answer before it sends the next one: up to one piece per stage is in the ring at once (prefill_ahead), so stage 0 computes piece i + 1 while the next stage runs piece i. Waiting for each answer left every stage idle while the others worked: on two PCs a 1,526-token prompt took the sum of both stages, 60 s (2026-10-26). The cap keeps a long prompt from queueing many pieces in front of other requests' tokens on a slower stage. This is stage 0's scheduling only. The frames and the answers are the ones an older fellow sends and expects.

One stage on one PC. A model too big for any one card may still fit one PC's cards: Qwen3-14B in 4 bits is 8.3 GB, against two 6 GB cards. Loaded with num_stages: 1 (swarm model-load --num-stages 1, optionally with fellow_id), the model serves as one stage, its layers spread over that fellow's GPUs. Stage 0 then also holds the head, generates locally (StageInferenceRuntime.run_local) with the prompt in pieces, and nothing crosses a link per token. A cut that training made inside the stage is applied where it was trained (_inner_boundaries): a learned bottleneck's two halves, or a fixed quantizer. The PC serves the model that was trained, not its adapters without the glue they learned around. The leader sends such a load only to fellows that declare serve-one-stage; an older fellow would wait for a second stage and answer every request empty. A model that records no training split cannot be served this way (its shards do not say which layers they hold). One stage serves one request at a time. The old single-node path (fellow_id without num_stages) still puts the whole model on one card.

A serving stage uses every GPU of its fellow unless the fellow pins a device: its blocks are spread like training's (plan_balanced_gpu_layers, each card to the same share of its free VRAM, the embeddings counted on the first and the head on the last), with the boundary compressor's half next to the block it borders. On a card that computes in float32 only (a GTX 10xx), a shard that was exported in half precision - a LoRA fine-tune over a frozen backbone stored in float16 - keeps its weights in that precision and upcasts them per matmul (pipe/split/torch/half_storage.py): nothing is lost, and the weights take half the VRAM. Before both, every block went to cuda:0 in float32, and the first stage of Qwen3.5-4B (13 layers and a 248k-token embedding, 8.3 GB that way) could not load on a fellow with two 4 GB cards.

Hardening & observability — E8

• Capacity: one model per fellow by default (SWARM_MAX_LOADED_MODELS), oldest evicted on overflow; generation is serialized per model.
• Idle auto-unload: a loaded model is stopped after SWARM_MODEL_IDLE_UNLOAD_S seconds without a request (default 900), and its VRAM goes back to the driver. The next request pays the reload. A generation in flight is never reaped, however long it streams. Set the variable to 0 on a dedicated inference node where reload latency matters more than the memory.
• Metrics: per-model tokens_per_s, last_ttft_s, requests, idle_s, vram_bytes surface in the heartbeat loaded_models and GET .../status.
• Speculative decoding (serving-speed TS1-TS3): a stage holding the whole model, or stage 0 of a ring, can let each decode pass carry up to k tokens drafted by prompt lookup (runtime/drafting.py: the latest earlier occurrence of the reply's last 4, 3 or 2 tokens, in the prompt or the reply, and what followed it). The pass runs [last, *drafts] as k + 1 positions, every one through the head, and runtime/torch/speculative.verify_drafts keeps the drafts the model agrees with plus one token of its own: greedy, a draft stands while it is the argmax; sampled, it stands with the probability the request's temperature and top-p give it, and a rejected draft's position is drawn from that distribution without it - exact speculative sampling for a point-mass draft. The KV cache is then cropped (DynamicCache.crop) to the positions that stand. A session whose drafts keep failing stops drafting for 16 passes (DraftGuard), and the token budget keeps k positions back for the pass in flight. Hybrid models (linear attention) cannot roll back and decode without drafts. Off by default: SWARM_SERVE_SPECULATIVE=k (at most 8) on the leader sets transport.speculative for one-stage loads, or on a fellow for its own. Measured offline on Qwen3-14B replies (TS0): with k = 2, code edits gain 2.1 tokens per pass and chat 1.1; on a GTX 1050 Ti every extra position of a pass costs about 58 %, so k above 2 only pays on cards where it is cheap. Each reply's timing adds spec_k, spec_passes, spec_tokens, spec_drafted and spec_accepted.

Across a ring (TS3) stage 0 drafts, and the pass travels the ring as k + 1 positions. Every hidden frame of a drafting session carries pos, where its positions start: a stage whose cache for that session is longer (it still holds drafts of a pass the last stage partly rejected) crops it there first, so no stage but the first and the last ever needs to know what was kept. The frame that carries drafts also carries draft, the drafted ids; the last stage checks them in that pass and answers with every token gained (send_tokens: the token reply's body holds them all), and stage 0 crops its own cache to the positions that stood. Passes without drafts go through the stages in decode batches as before, pos included. Stage 0 drafts only while at most one request per stage is in flight (SPEC_MAX_ACTIVE_PER_STAGE): in a fuller ring every stage already has work queued and a drafted position would cost another request its turn. Only the concurrent loop drafts, and the leader sets transport.speculative for a ring only when every stage's fellow declares serve-speculative - a stage from before would keep rejected drafts in its cache.

A draft model (TS9). Prompt lookup only guesses what a reply repeats. With SWARM_SERVE_DRAFT_MODEL on the leader the last stage of a drafting ring loads a small model of the same family (auto: Qwen3-0.6B for a Qwen3 model; or a model id) and guesses everything: replayed offline against the Qwen3-14B fine-tune's replies, Qwen3-0.6B agreed with 71 % of chat tokens, 2.2 tokens a pass at k = 2 where prompt lookup made 1.1 (exp/spikes/serving/draft_acceptance.py). The last stage produces every token anyway, so after a pass is checked it drafts the next k tokens (runtime/torch/draft_model.py) and they go back to stage 0 with the reply (draft on the token frame); the next pass sends them round instead of prompt lookup's. The draft model needs the conversation as tokens, so every prompt piece carries its ids (ids, with more while pieces follow), and stage 0 asks for drafts per pass (draft_k) only while the ring has room. A sampled draft is checked against the distribution it was drawn from - kept with probability min(1, p/q), a fallen one redrawn from max(0, p - q) - so sampled replies stay exact too. The draft model loads on the last stage's card with the most free memory, only with a vocabulary identical to the served model's, and its KV cache counts in the stage's token budget (on Qwen3-0.6B, 114 KB a token - more than a 23-layer stage of Qwen3-14B's own). Every stage must declare serve-draft-model.

Measured on two PCs, transformers' own decode took Qwen3-0.6B about 95 ms a drafted token on a GTX 1660 SUPER - reading its weights takes about 4 - so a Llama/Qwen-style draft model runs through a decoder of fixed shape instead (runtime/torch/draft_graph.py): its own modules over key/value buffers allocated at load (one slot of 4,096 positions per stage, about 470 MB each for Qwen3-0.6B, counted by the budget as memory already taken), one token a step, the step captured as a CUDA graph per slot and replayed. A history longer than a few tokens is read in one eager pass; a session without a slot takes the least recently used one and re-reads its history. The load logs both step times ([stageN draft] ... fixed-shape step X ms eager, Y ms replayed); where capture fails a slot steps eagerly. Every captured slot must reproduce an eager step's logits at load, or it steps eagerly too (a capture on another card than the draft model's left an empty graph once). Replayed, the step still took 63 ms of GPU time on a GTX 1660 SUPER in float16, so a float16 draft model on CUDA first times one row through an MLP-sized weight in float16, float32, and float16 upcast to float32 per matmul (draft_graph.matmul_ms); where the upcast is clearly faster (30 %), the draft model computes in float32 over its float16 weights and cache (half_storage), at no extra memory. The load log shows the three times and the choice. Prompt lookup still drafts first when it found a run of 3 or more tokens and a full set of drafts (LOOKUP_FIRST_RUN): those cost nothing, and while a reply copying its prompt keeps them all, stage 0 does not ask the draft model at all. - 4-bit matmuls of a few rows: bitsandbytes picks its 4-bit kernel by architecture and row count, and on a GTX 1660 SUPER (Turing without tensor cores) the one it takes from 4 rows on was 4.5 times slower than the one for 3. A whole stage took 147 ms for 3 positions and 659 ms for 4, which is why decode batches stay at 2 on pre-Ampere cards. So each stage times its own 4-bit layers for 1 to 8 rows at load, per card, and splits a call of more rows than the cheapest limit into calls of at most that many (pipe/split/torch/fourbit_rows.py). The fellow log says what it found: [stageN 4bit-rows] cuda:0 one block ms M=1:… M=8:…; calls of more rows split into 3. Calls of more than 8 rows (prompt pieces) are not split; each row's result is the same either way. SWARM_SERVE_4BIT_ROWS on a fellow: auto (default), off, or a fixed row limit. Measured on two PCs (2026-10-18): stage 1 took 188 ms for 4 positions instead of 659, a decode batch of 4 took 211 ms instead of 685, and 8 users got 11.6 tokens/s in all with batches of 4 against 9.9 with batches of 2. A fellow that splits declares serve-4bit-rows, and the leader lifts the pre-Ampere batch cap from 2 to 4 when every stage's fellow does. - 4-bit matmuls in float32 (TS12): the same kernel choice made prompts slow. bitsandbytes 0.50 sends float16 calls of up to a few hundred rows on sm_75 to its tensor-core kernel, tuned on a T4. On a GTX 1660 (no tensor cores) a 256-token prompt piece took 6.1-7.7 s on stage 1, against 2.0 s on stage 0's GTX 1050 Ti in float32 (2026-10-26). Float32 rows take another path: a SIMT kernel under 8 rows, above that the weight is dequantized and multiplied by cuBLAS. So a stage that does not compute in float32 already also times its 4-bit layers in float32 at load, at 1 to 8 rows and a prompt piece's width (SWARM_SERVE_PREFILL_CHUNK). It then computes in float32 the row counts where that saves at least 15 %: the input is cast up and the result back, and weights, adapters and the KV cache stay as they were. The fellow log line gains the float32 curve and the verdict, for example ...; float32 M=1:… M=256:…; calls of more rows split into 3; float32 from 4 rows up. A card that computes prompt pieces in float32 holds one of its weights dequantized in float32 while a piece runs (about 356 MB for Qwen3-14B's MLP), and the KV budget keeps that much free. That room replaces the 300 MB it keeps for other passing buffers rather than adding to it, since a stage runs one forward at a time. A card computes prompt pieces in float32 only with twice that buffer plus 300 MB free after loading, because the free memory is fragmented. The second 1660 of a Qwen3-14B stage, which also holds the head and the draft model, had about 0.8 GB free. Its first float32 prompt piece found no 340 MiB in one block, so that card keeps prompt pieces in float16 and the log line says no room for float32 prompt pieces. Turn it off with SWARM_SERVE_4BIT_FLOAT32=off on the fellow. - A cheaper decode step (serving-speed M3): - Lean LoRA (TS7). Adapters served unmerged over a 4-bit base go through pipe/split/torch/finetune/lean.py instead of PEFT's layer: the same arithmetic (the scale folded into B, the base output not cloned, the add fused into one addmm when the dtypes allow), without PEFT's per-call Python. PEFT's layers cost stage 1 of the two-PC benchmark 45-50 ms of a 170 ms token. On by default; SWARM_SERVE_LORA on a fellow: lean, lean16 (adapters in the stage's compute dtype: two casts fewer per layer, the adapter's product rounded in half precision) or peft. DoRA and other variants stay with PEFT. - A KV cache that grows in place (TS5). Attention-only stages keep each layer's keys and values in buffers grown 64 positions at a time (pipe/split/torch/kv_cache.py) rather than concatenated every token, which copied the whole past. Every rollback goes through crop_to, which passes crop a negative count: transformers changed what a positive one means in 5.18. - Grouped attention (TS5). Each key/value head serves its query heads in one matmul (pipe/split/torch/grouped_attention.py) where sdpa with a mask - a decode batch, a verify pass - or float32 on a card without flash attention copies the keys and values once per query head (5 times for Qwen3-14B). Same results to rounding. Measured on the GTX 1050 Tis (float32): one position at a 2,048-token context 168 ms -> 113 ms, a verify pass of 3 284 -> 240 ms; on the GTX 1660s (float16) no difference. SWARM_SERVE_ATTENTION on a fellow: auto (default: grouped on float32 stages), grouped or sdpa. - Step costs at load: before a stage reports ready it measures, on its own cards, what one serving step costs (runtime/torch/step_costs.py), and writes it to the fellow log (GET /api/v1/workers/<id>/logs): [stageN verify-cost] - one request with M = 1 to 8 positions at once, on a KV cache of 256 and 2,048 positions (what checking M - 1 drafted tokens would cost); [stageN batch-cost] - batched decode of 2 and 4 requests; [stageN step-profile] - GPU time, kernel launches and the busiest CPU ops of a one-token step, only with SWARM_SERVE_STEP_COSTS=profile (on the two-PC cluster a stage measured with the profiler served about 35 ms a token slower than without, so measurement loads leave it out); [stageN lora-cost] - the step with unmerged LoRA adapters switched off - with lean adapters, peft_ms, lean_ms, lean16_ms and off_ms side by side; [stageN attn-cost] - sdpa against grouped attention, one request at the longest context with 1 and 3 positions, and a batch of 4; [stageN graph-spike] - one block's MLP replayed as a CUDA graph against run eagerly, and the saving that would make over all blocks (TS6). The cache is filled with random keys and values instead of a prompt, and every position a measurement adds is rolled back, so the stage serves exactly as before. After ready, stage 0 times ten connections to the stage it dials ([stage0 link-probe]): one round trip of the frames' path (the mesh and its tunnels) without ZeroMQ or compute. Both are off by default: on a GTX 1050 Ti or 1660 they add 35-50 s to a load, and a model unloaded when idle pays its reload on the next request. SWARM_SERVE_STEP_COSTS=1 on the leader asks every stage of the next load for them (transport.step_costs); on a fellow it turns them on for that fellow's loads.

VRAM lifecycle

VRAM is only ever borrowed, and every path that ends a fellow's involvement with a model gives it back:

Event	What releases the memory
UNLOAD_MODEL signal, or swarm model-unload	the serving subprocess is asked to exit, then killed after 10 s
No request for SWARM_MODEL_IDLE_UNLOAD_S	the fellow's idle reaper stops the subprocess
A second model loaded past SWARM_MAX_LOADED_MODELS	the oldest model is evicted before the new one starts
STOP / PURGE signal, fellow shutdown	every serving subprocess is stopped
UPDATE signal	subprocesses are stopped before the execv re-exec

The last row is the one that is easy to get wrong: a serving subprocess is a child of the fellow but not in its process group, so an exit that skips the shutdown leaves it running and holding a GPU that the leader now believes is free. Nothing short of killing it by hand gets that memory back.

Training is not on this list because it never needed to be: a torch job runs in its own subprocess that exits when the job ends, which releases the CUDA context with it.

Note that a loaded model's reported vram_free_gb is a high-water mark, not a live reading - torch's caching allocator holds freed KV-cache blocks rather than returning them to the driver. That is why the leader budgets the cache arithmetically (telemetry/vram_budget.py) instead of trusting free VRAM. A model shipped as LoRA adapters (training.lora.export: adapters - every run over a quantized base) is sized by its base, not its shard files, which hold a few MB while each stage loads the whole base under them: the base checkpoint's parameter count from its headers, else 12 x layers x hidden^2 from the model's own shape. Sized by the files, every card would have looked big enough.

Environment variables

Var	Default	Meaning
SWARM_SERVABLE_MODELS_ROOT	$SWARM_DATA_DIR/servable_models	leader: where servable models live
SWARM_FELLOW_MODEL_CACHE	~/.silent-swarm/models (/state/models in the fellow image)	fellow: downloaded model cache
SWARM_MAX_LOADED_MODELS	1	fellow: concurrent loaded models
SWARM_MODEL_IDLE_UNLOAD_S	900	fellow: idle TTL auto-unload (0 disables)
SWARM_SERVE_IN_FLIGHT	4 x stages (2 x stages with a fellow that predates serve-kv-budget)	leader: requests a served pipeline keeps in its ring at once (XL8; 1 serves one at a time), lowered to what every stage's KV cache fits
SWARM_SERVE_TYPICAL_TOKENS	1024	leader: tokens of a typical chat; the in-flight count is sized by it when the stages admit by tokens
SWARM_SERVE_CONTEXT_TOKENS	4096	leader: the longest conversation a served model must take; the token budget never goes below it
SWARM_SERVE_MAX_BATCH	0	fellow: most requests one forward takes in batched decode (0: all in flight, 1: one at a time); overridden by the leader's
SWARM_SERVE_MAX_BATCH (leader)	unset: with any GPU older than Ampere 4 when every fellow declares serve-4bit-rows, else 2; with newer GPUs none	leader: the decode-batch cap for every stage of a served pipeline (transport.max_batch); 0 is no cap
SWARM_SERVE_SPECULATIVE	0	leader: tokens drafted per decode pass by prompt lookup (transport.speculative, at most 8; 0 off) - on one-stage loads, and on rings whose fellows all declare serve-speculative; fellow: the same for its own loads
SWARM_SERVE_LORA	lean	fellow: unmerged LoRA adapters served lean, lean16 (in the compute dtype) or through peft
SWARM_SERVE_ATTENTION	auto	fellow: grouped shares each key/value head with its query heads instead of copying it per head; auto does so on stages that compute in float32; sdpa never
SWARM_SERVE_DRAFT_MODEL	unset	leader: the small model a drafting ring's last stage drafts with - auto (Qwen3-0.6B for Qwen3 models) or a model id; needs SWARM_SERVE_SPECULATIVE and a ring whose fellows all declare serve-draft-model
SWARM_SERVE_4BIT_ROWS	auto	fellow: split 4-bit matmuls of up to 8 rows into calls of the row count this card runs fastest (auto: measured at load; off; or a fixed limit)
SWARM_SERVE_4BIT_FLOAT32	auto	fellow: compute 4-bit matmuls in float32 for the row counts where this card is clearly faster that way (auto: measured at load against the stage's own dtype; off)
SWARM_SERVE_STEP_COSTS	off	leader: every stage of a load measures its step costs and the link, into its fellow log (transport.step_costs); fellow: the same for its own loads
SWARM_SERVE_PREFILL_CHUNK	256	fellow: prompt tokens per piece sent through the stages (0: the whole prompt at once)
SWARM_SERVE_MAX_IN_FLIGHT	8	fellow: requests it holds at once, served or waiting their turn
SWARM_REQUIRE_API_TOKEN	unset	leader: force bearer auth
SWARM_API_TOKENS_FILE	next to state DB	leader: token store path