Protocol and API reference

This page is the compact reference for the control-plane API and data-plane transport. Use Runtime model first if you want the conceptual flow.

Control plane

Purpose	Route or command
Health	GET /api/v1/health (liveness: the process answers), GET /api/v1/ready (readiness: 503 with reasons while starting or shutting down, when the state DB takes no write, or below SWARM_MIN_FREE_DISK_GB free)
Jobs	POST /api/v1/jobs, GET /api/v1/jobs, GET /api/v1/jobs/{job_id}
Job plan preview	POST /api/v1/jobs/plan (same body as a submission; nothing is stored)
Job status	POST /api/v1/jobs/{job_id}/status
Job metrics	GET /api/v1/jobs/{job_id}/metrics/stream
Cancel job	DELETE /api/v1/jobs/{job_id}
Checkpoint	POST /api/v1/jobs/{job_id}/checkpoint
Workers	GET /api/v1/workers, DELETE /api/v1/workers/{fellow_id}
Worker heartbeat	POST /api/v1/workers/register, POST /api/v1/workers/heartbeat
Worker management	POST /api/v1/workers/release, POST /api/v1/workers/update
Worker logs	GET /api/v1/workers/{fellow_id}/logs/stream
Cluster view	GET /api/v1/cluster/nodes[?pool=], GET /api/v1/cluster/devices
Signals	WS /ws/signals/{fellow_id} (node credential, see auth)
Bootstrap	GET /api/v1/bootstrap/manifest.json, GET /api/v1/bootstrap/fellow.sh, POST /api/v1/bootstrap/join-sessions
Community numbers	GET /api/v1/public/stats (public, aggregate, cached)
Your contribution	GET /api/v1/me/contribution (session; measured totals, no currency)
Signup attribution	GET /api/v1/admin/users/signup-intents (admin; which landing door converted)
Public pages	GET / (en), GET /de (de), GET /en (301 to /), /robots.txt, /sitemap.xml, /llms.txt
Pools	GET/POST /api/v1/pools, GET/PATCH/DELETE /api/v1/pools/{id}
Pool health	GET /api/v1/pools/{id}/metrics, …/metrics/stream (SSE), …/metrics/history?metric=
Pool timeline	GET /api/v1/pools/{id}/events
Node roles	POST /api/v1/pools/{id}/nodes/{fellow_id}/role
Buckets	GET/POST /api/v1/buckets, GET/PATCH /api/v1/buckets/{id}, …/datasets
Model catalog	GET /api/v1/models, GET /api/v1/models/{id}, …/status, GET /api/v1/serving
Model serving	POST /api/v1/models/{id}/load, …/unload
Inference	POST /api/v1/chat/completions (and /v1/chat/completions, OpenAI-compatible)
Inference transport	GET /api/v1/infer/requests/{fellow_id}, POST /api/v1/infer/responses/{request_id}
Web UI	GET /dashboard, /pools/{owner}/{pool} (and /training, /inference, its tabs), /buckets, /buckets/{owner}/{bucket}, /orgs, /orgs/{org}, /training, /inference, /profile - see Organizations and addresses

Every route is classified in leader/route_policy.py; anything not listed there is denied by default, so a new endpoint is unreachable until it is classified.

Control signal values

ControlSignal.signal supports:

• START
• PAUSE
• CHECKPOINT
• STOP
• RECONFIGURE
• UPDATE
• PURGE
• PING — a control-plane round-trip probe. Only sent to fellows that advertised the ping capability at registration: an older fellow treats an unknown signal as a protocol error and reconnects, so an unnegotiated ping would knock a running cluster into a reconnect loop.

What a START carries

A START tells one fellow its part of a training job. Its payload.assignment is a StageAssignment (shared/contracts.py):

{
  "job_id": "3f2c...",
  "stage": {
    "stage_index": 0,
    "split": {"num_stages": 2, "split_layer": 6, "compression": {"method": "fixed_int8", ...},
              "lora": {"enabled": true, "r": 8}, "freeze_backbone": false,
              "dtype": "float16", "quantization": {"bits": 4}, "weight_dtype": null},
    "placement": {"device": "cpu", "block_devices": null, ...}
  },
  "route": {"mode": "tunnel", "job_id": "3f2c...", "act_channel": "1-2-act", "grad_channel": "2-1-grad"}
}

stage is the StageSpec the worker's build_stage runs; placement is left at its default because the fellow places blocks on its own GPUs. route is the link to the peer stage without the parts the fellow supplies itself (the relay URL of the leader it is connected to, and its node credential). The leader keeps every assignment in the job's config.stage_assignments. One function, shared/assignment.build_assignment, builds them, and a fellow that receives a START from an older leader without assignment builds the same one from the legacy keys (pipeline, peers, transport_mode, config), which are still sent for older fellows.

The fellow turns the assignment into a WorkerSpec (runtime/protocol.py) for its torch worker subprocess, which validates it before loading anything and answers with progress, result and error events.

Registration and heartbeat

sequenceDiagram
    participant Fellow
    participant Leader
    Fellow->>Leader: POST /api/v1/workers/register
    Leader-->>Fellow: RegisterResponse (node_token)
    Fellow->>Leader: WS /ws/signals/{fellow_id}<br/>Authorization: Bearer node_token
    loop every heartbeat interval
        Fellow->>Leader: POST /api/v1/workers/heartbeat<br/>status, step, loss, bytes, live hardware
        Leader-->>Fellow: HeartbeatResponse
    end

Heartbeats include live GPU hardware telemetry when available. The leader uses that payload to refresh per-GPU free VRAM in /api/v1/cluster/nodes and /api/v1/cluster/devices, so the Ops UI shows current VRAM utilization instead of only the registration-time inventory.

/api/v1/cluster/nodes reports each node's pool_id and accepts ?pool=<id> to narrow the list to one pool — the same filter /api/v1/ops/join-graph takes, so the ops console's table and graph always describe the same set of nodes.

Heartbeats may also carry a telemetry block (NodeTelemetry): per-GPU utilization, temperature and power, the round trip the fellow observes talking to the leader, round trips to its pipeline peers, and monotonic OOM / error / restart counters. Every field is optional — a fellow built before the block still registers and heartbeats, and consumers render its missing signals as unknown rather than zero.

Server-rendered navigation

Pages are full server renders, so anything the browser has to fill in is rebuilt on every navigation. The sidebar used to be exactly that: a skeleton, an identity fetch, then a swap — visible on each click.

leader/web/nav_render.py renders it server-side instead. The page builders stay free of request state (the same reason the CSP nonce is substituted late): the template carries tokens — __NAV_USER__, __NAV_POOL_NAME__, __NAV_STATE_<key>__, __NAV_ADMIN_HIDDEN__, __USER_NAME__ — and the HTTP middleware substitutes them once per response, where the session cookie is available. The active pool comes from the swarm_pool cookie, so the server renders the same chip the user selected. Every value that originates with a user (display name, pool name) is HTML-escaped there.

shared/chrome.js - the first ES module on every page - does not fetch identity or pools on load; it only wires the menus and the drawer, and loads the pool list lazily when the switcher is opened.

Pool telemetry

PoolTelemetry (leader/telemetry/pool_telemetry.py) keeps a bounded in-memory window of health signals per fellow and rolls them up per pool. Nothing polls a fellow; every input is something the leader already receives:

flowchart LR
    HB[heartbeat handler] -->|availability, step rate, node telemetry| PT[PoolTelemetry]
    WS[signal WS hub] -->|ping/pong RTT, disconnects| PT
    REG[register route] -->|flap counter| PT
    CHAT[chat completions + broker] --> SM[ServingMetrics]
    SM -->|tokens/s, TTFT, queue wait, errors| PT
    PT --> API["/api/v1/pools/{id}/metrics (+ SSE)"]

Design rules that the implementation holds to:

• The request path never blocks on telemetry. Recording a generation is a few dict writes and perf_counter() reads into a bounded deque; a telemetry failure can never fail a generation (the broker's dispatch hook swallows its own exceptions).
• One clock. Every duration is a delta of leader-side perf_counter() reads, or a delta the fellow measured about itself. No metric depends on leader and fellow clocks agreeing.
• Bounded memory. Each node keeps at most ~1024 samples (≈1 h at the 5 s heartbeat interval); released nodes are forgotten so they stop skewing pool roll-ups.
• Unknown is not zero. A signal that was never reported is null, so a node that simply cannot measure something is not penalised for it.

Availability is received heartbeats ÷ expected heartbeats over the window, so a node cannot hide an outage by reconnecting; it stays null until a node has been observed for at least two intervals.

The rest of leader/telemetry/ builds on that snapshot:

Module	Turns the snapshot into
grades.py	one explainable 0-100 score and an A-F band per node, with hysteresis so a band does not churn
pool_metrics.py	stored history (minute resolution for 7 days, hourly for 90) and the pool event timeline
vram_budget.py	what a serving stage needs — weights plus the KV cache its layers will allocate — which gates model loads
replacement.py	the decision to quarantine a failing node and promote a standby, guarded by dwell times and a cooldown

notify/alerts.py turns the resulting events into email/webhook notifications.

leader/resilience/ledger.py keeps the placement ledger: which node holds which shard (layer range) of which training job or served model, in which placement epoch, with an append-only log of every state change. The places that send work write it (training dispatch, model load); the sweep loop reconciles it every few seconds against the fellows' heartbeats, so a node past its heartbeat timeout has its shards marked lost within one pass. It is the record that recovery from a lost node will act on; today it is read-only.

leader/resilience/snapshots.py holds the stage snapshots that recovery will resume from. Each stage of a pipeline job saves its trainable state, optimizer, schedule, step and RNG every every_steps steps (runtime/torch/stage_state.py: a safetensors file, parameters named by their global layer so a different split can find them), the fellow uploads it in chunks (fellow/snapshot_upload.py), and the leader checks size, SHA-256 and the header - read without torch - before it counts. A step is committed once every stage delivered it: the pipeline schedules finish a step on every stage before any starts the next, so those files are one consistent state. The leader keeps the newest two committed steps per running job.

leader/resilience/resume.py resumes a stopped pipeline run from its newest committed step (POST /api/v1/jobs/{id}/resume, robustness R3). It checks that every stage's node is connected, idle, still schedulable in the job's pool and declares resume, drops the uncommitted steps the stopped run left behind, turns the job RECOVERING and sends every stage START again on the same node with a ResumePoint (the step, and its own file's SHA-256) in its StageAssignment. The fellow takes its local copy when it is that file and otherwise fetches it from .../snapshots/{step}/stages/{i}/node; the worker (WorkerSpec.resume) restores the state, skips the first stage's data cursor by that many batches and opens its links with Session(start_step=k). The first progress report past the step turns the job RUNNING. The job states are PENDING -> RUNNING -> COMPLETED | FAILED | CANCELLED, plus RUNNING | FAILED | CANCELLED -> RECOVERING -> RUNNING | ..., which only a resume enters - a fellow's status report can never reopen a finished job. The operator-facing description of all of this — grading rules, the replacement triggers and their guards, the VRAM arithmetic — is in the Pools guide.

leader/resilience/recovery.py is the Recovery Coordinator (R4). When the replacement policy finds a node gone that holds a stage of a running pipeline job, it opens an incident - a durable saga, one row in incidents - instead of failing the job: fence (RECOVERING at the newest committed step, the next placement epoch, PAUSE to every stage), pick (the best standby that fits the stage, or a re-split over the nodes left via plan_job), move (roles and placement plan), resume (START with a ResumePoint per stage - after a re-split its sources, the old stages' files holding its layers, merged by global parameter name in runtime/torch/stage_state.merge_stage_snapshots), verify (the first step past the snapshot). Each step is saved before the next, so the sweep loop picks a saga up again after a restart. The placement epoch (config.placement_epoch, StageAssignment.epoch) fences the runs of one job from each other: relay channel names carry it from epoch 2 on and the relay refuses older ones, Session(epoch=) drops link frames of another epoch, and the leader ignores status reports and refuses snapshot uploads of an older one.

leader/resilience/gatekeeper.py is the Gatekeeper (R5): it judges nodes and writes their roles, and asks the coordinator to move a job with an urgency - evict_now (fence now), migrate and drain (the incident's first step, boundary, sends STAGE_CONTROL stop_at=k to every stage; the worker's StageSnapshotter reads the stage's control.json, saves step k and raises StageStop, so the run resumes from k with no step lost). It runs admission probation (SELFTEST: the leader draws a seed, the fellow builds the network with shared/selftest.probe_arrays and computes it on its GPU in runtime/torch/selftest.py, the leader checks it against numpy), persists quarantine history, and handles host checkout and availability windows (eligible_workers skips nodes about to be taken back). The same probe with nothing to allocate is the speed benchmark (Gatekeeper.benchmark): float32 and float16 matmul TFLOPS and memory bandwidth per GPU, and bandwidth to the leader, kept in node_benchmarks and ranked per pool by leader/telemetry/speed.py. A watch node halves its job's snapshot cadence through the same control file.

leader/resilience/serving_failover.py keeps a served model up when a stage's node dies (R6). The model's stage payloads are kept (serving_plans); the sweep loop opens a serve incident for a lost stage, moves the ring to its next epoch with REWIRE (the stage process gets {"cmd": "rewire"} on stdin: it restarts its serving loops - never its model - on relay channels named after the epoch, closing every session of the old ring, so stage 0 fails what was in flight with code: "rewired"), loads the stage elsewhere from its own files (.../shards/{i}/weights), and the InferenceBroker replays the held requests with continue_text once the model is ready.

leader/resilience/covers.py plans covers (R7): per pool, the riskiest stages of the served models and the running job each get a standby - hot (a serving stage loaded with --cover, out of the ring until a REWIRE with activate switches it in) when its VRAM allows, else warm (a serving stage's files, or a training stage's base checkpoint files via runtime/torch/prefetch.py and stage_loader.prefetch_stage_files). Covers sit outside the fellow's model slots and idle reaper, are reported in the heartbeat, and are what both failovers try first.

Data plane

Two adjacent stages talk over one link (silent_swarm.pipe.link), which moves a message - a list of byte parts - in both directions: activations forward, the (loss, gradient) reply back. A message is two parts, a JSON header with shape and dtype and the raw tensor bytes. There are two kinds of link:

flowchart LR
    subgraph Direct["p2p: one ZeroMQ PAIR socket"]
        U1[Stage 0] -- activation --> D1["Stage 1 (binds :act_port)"]
        D1 -- loss + gradient --> U1
    end
    subgraph Relayed["tunnel: through the leader"]
        U2[Stage 0] -- "push {up}-{down}-act" --> L[Leader relay]
        L -- pop --> D2[Stage 1]
        D2 -- "push {down}-{up}-grad" --> L
        L -- pop --> U2
    end

On a direct link the downstream fellow binds its activation port and the upstream one connects; each part is one ZeroMQ frame. Through the relay, each message is one HTTP body in a length-prefixed multipart encoding (!I part count, then !I length and bytes per part), on a lane named by the job id:

Relay route	Purpose
POST /api/v1/relay/{job_id}/{channel}/push	Queue one frame on a channel. 503 when the channel stays full (SWARM_RELAY_QUEUE_MAX, SWARM_RELAY_PUT_TIMEOUT_S), 413 above SWARM_RELAY_MAX_FRAME_MB.
GET /api/v1/relay/{job_id}/{channel}/pop?timeout=	Long-poll for the next frame: 200 with the frame, 204 if none arrived.

Distributed inference uses the same routes with the served model's id as the lane and two channels, act (hidden states forward) and tok (sampled tokens back). Wire 1 sends one raw payload per body (a float32 .npy, a packed (token, stop), a \x00 to end). Wire 2 sends multipart tensor frames tagged hidden / token / end, keeping the model's dtype - half the bytes for bf16.

Wire versions and capabilities

Some changes alter what one stage must agree on with its peer, so the leader only uses them when every fellow involved declares support at registration (RegisterRequest.capabilities). A fellow from before a feature simply does not declare it, and the job runs exactly as it always did.

Capability	What the leader then does
stage-assignment	Sends a weighted job's VRAM-weighted boundary in assignment.stage.split.split_layer. Otherwise the boundary is a pinned layer_index or halfway.
wire-v2	Sets assignment.wire (training) and transport.wire (serving) to 2: quantized boundaries cross as N-bit codec frames, inference as tensor frames.
serve-codec	Sets transport.boundary_codec (serving, wire 2 only): a quantized boundary is served as N-bit codes. Without it, stage 0 quantizes in the graph and sends floats.
ping	Sends control-plane round-trip probes.
stage-snapshot	Declared by fellows whose training stage saves its state every assignment.snapshot.every_steps steps and uploads it (robustness R2). The leader sets assignment.snapshot only when every fellow in the job declares it - a step is only useful when every stage saved it.
resume	Declared by fellows whose training stage can start from a committed snapshot: it fetches its stage's file, restores it, skips the data it trained on and opens its links at that step (robustness R3). The leader resumes a job only when every fellow in it declares this.
recovery	Declared by fellows that take part in an automatic recovery (robustness R4): they fence links, progress reports and snapshot uploads to the run's placement epoch and restore a re-split stage from several old stages' files. The Recovery Coordinator moves a job only when every fellow it would run on declares this.
selftest	Declared by fellows that run the admission self-test (robustness R5): a node joining a pool with it is on probation until it passes. Older fellows skip probation.
stage-control	Declared by fellows whose running training stage takes STAGE_CONTROL (robustness R5): stop after a named step at a snapshot boundary, or a new snapshot cadence. Migrate and drain lose no step only when every stage declares it.
serve-rewire	Declared by fellows whose served stage takes REWIRE (robustness R6): a new ring epoch and peer without a reload, the old ring's sessions ended at once, and continue_text on stage 0. Serving failover acts only when every stage's fellow declares it.
cover	Declared by fellows that hold covers for other nodes' stages (robustness R7, COVER): a hot serving stage out of the ring, or a stage's files fetched ahead, in slots outside SWARM_MAX_LOADED_MODELS and the idle reaper. Only such standbys are given covers.
calibrated-bottleneck	Declared by fellows with the linear, calibrated learned bottleneck: before step 1 the stage before a learned boundary sends the next one the principal components it calibrated from, and serving reads 8-bit codes with a scale per token (tokint8). The leader starts a job with a learned bottleneck, and loads a served model with one, only on fellows that declare it; the retired bottleneck could not read either.
serve-one-stage	Declared by fellows that serve a model as one stage across their own GPUs: the whole model, generated locally, a cut that training made applied in place, the prompt in pieces. The leader sends a num_stages: 1 load only to such a fellow; an older one waits for a second stage and answers every request empty.
microbatch	Declared by fellows whose training stage runs micro-batched schedules (XL4). train.micro_batches above 1 applies only when every fellow in the job declares it; otherwise the batch trains whole and the plan says why.
serve-concurrent	Declared by fellows whose serving stage keeps several requests in the ring at once (XL8). The leader sets a model's max_in_flight above 1 only when every stage's fellow declares it and the stages speak wire 2.
serve-int4-fast	Declared by fellows that have torchao installed and only Ampere-or-newer GPUs (XL7). A load asking for int4-fast uses torchao's int4 kernels only on fellows that declare it; otherwise it serves 4-bit on bitsandbytes.
serve-quantized	Declared by fellows that serve weights quantized as they load (XL6). A quantized load goes only to fellows that declare it; with too few of them the model serves in full precision.
serve-n-stage	Declared by fellows that can serve any stage of a ring of more than two (XL5). Serving over more than two stages is refused unless every placed fellow declares it.
n-stage	Declared by fellows that can run a middle stage (XL2). Plans a training pipeline of more than two stages (XL1): use_all_workers takes up to 16 fellows instead of 2, and min_workers above 2 can start. Each stage is sent every boundary (assignment.stage.split.boundaries) and a route to each neighbour (prev_route, next_route). A job that asks for more than two stages while one of its fellows lacks it stays pending and names that fellow.

Capabilities live with the leader's in-memory worker record; after a leader restart a fellow counts as capability-less until it registers again, which falls back to the old behaviour rather than ahead of it.

Where a job splits

The leader decides the boundary once and shows exactly that one per worker: a pinned compression_boundaries[0].layer_index wins; a partition: weighted job splits by its nodes' free VRAM when the leader knows the model's layer count and every fellow declares stage-assignment; anything else splits halfway (num_layers // 2, as each worker computes it). The layer count comes from the job's model.num_layers, else from the model's config.json on the Hugging Face Hub (HF_ENDPOINT and HF_TOKEN are honoured; SWARM_LEADER_HF_LOOKUP=0 turns the lookup off). The decision is kept in the job's config.split_plan, with a note when a requested weighted split could not be applied.

A pipeline of more than two stages (XL1) gets every boundary decided on the leader and sent to each stage in split.boundaries: an even split (source: even, the first stages take the remainder), a VRAM-weighted one for a partition: weighted job, or the pins when every boundary has its own compression_boundaries[i].layer_index. A single layer_index pins only a two-stage job. No stage is ever left without a layer, and a model with fewer layers than stages is held with that reason. When the leader does not know the layer count, each worker splits evenly by stage_layer_ranges (the remainder goes to the last stage). Compression is configured per boundary: entry i of compression_boundaries is boundary i (between stage i and i + 1), a single entry applies to every boundary, and a boundary without an entry is uncompressed. split_plan.boundaries lists them all; the training console and swarm jobs show print "after layers 8, 16 of 24".

To see it: swarm jobs show <job_id> prints a started job's stages - fellow, GPUs, layer range - with the boundary and the rule that chose it, the dtype, the wire version and the transport (from config.placement_plan, split_plan and stage_assignments). Before submitting, swarm train --config ... --plan asks POST /api/v1/jobs/plan the same question against the pool's ready nodes now. It runs the dispatcher's own planner (training.dispatch.plan_job), so the preview cannot disagree with what a start would do, and says why a job could not start (too few ready nodes, GPUs below the VRAM floor or ruled out by dtype). Only the transport is left open: choosing it means probing every node's ports, which happens at start. swarm train --dry-run stays an offline estimate that never contacts the leader.

Transport selection is controlled with SWARM_PIPELINE_TRANSPORT=auto|tunnel|p2p|mesh. auto picks a direct link when every fellow's pipeline ports answer the leader's probe or the fellow is on the mesh, and the relay otherwise. A fellow holds both of its ports from start-up so the probe finds them; SWARM_ZMQ_BIND_HOST restricts which interface they listen on (mesh for the overlay address).

The job metrics stream reports act_mb_s and grad_mb_s per fellow: the activation and gradient payload its link moved, from the act_bytes / grad_bytes counters in its heartbeat. The heartbeat's bytes_sent / bytes_recv are the fellow's control traffic with the leader, a different number.

Authentication notes

• The swarm CLI can send Authorization: Bearer <token> via --token or config.
• Bootstrap downloads can require a pool/invite token when SWARM_BOOTSTRAP_REQUIRE_TOKEN=1 — via X-Bootstrap-Token header or ?token= query param. The script is HTTPS-only and the manifest publishes source_archive_sha256 / script_sha256 for verification (see bootstrap).
• Production deployments should enforce broader authentication at the leader or ingress layer.