Protocol and API reference¶

This page is the compact reference for the control-plane API and data-plane transport. Use Runtime model first if you want the conceptual flow.

Control plane¶

Purpose	Route or command
Health	`GET /api/v1/health`, `GET /api/v1/ready`
Jobs	`POST /api/v1/jobs`, `GET /api/v1/jobs`, `GET /api/v1/jobs/{job_id}`
Job status	`POST /api/v1/jobs/{job_id}/status`
Job metrics	`GET /api/v1/jobs/{job_id}/metrics/stream`
Cancel job	`DELETE /api/v1/jobs/{job_id}`
Checkpoint	`POST /api/v1/jobs/{job_id}/checkpoint`
Workers	`GET /api/v1/workers`, `DELETE /api/v1/workers/{fellow_id}`
Worker heartbeat	`POST /api/v1/workers/register`, `POST /api/v1/workers/heartbeat`
Worker management	`POST /api/v1/workers/release`, `POST /api/v1/workers/update`
Worker logs	`GET /api/v1/workers/{fellow_id}/logs/stream`
Cluster view	`GET /api/v1/cluster/nodes[?pool=]`, `GET /api/v1/cluster/devices`
Signals	`WS /ws/signals/{fellow_id}` (node credential, see auth)
Bootstrap	`GET /api/v1/bootstrap/manifest.json`, `GET /api/v1/bootstrap/fellow.sh`, `POST /api/v1/bootstrap/join-sessions`
Community numbers	`GET /api/v1/public/stats` (public, aggregate, cached)
Your contribution	`GET /api/v1/me/contribution` (session; measured totals, no currency)
Signup attribution	`GET /api/v1/admin/users/signup-intents` (admin; which landing door converted)
Public pages	`GET /` (en), `GET /de` (de), `GET /en` (301 to `/`), `/robots.txt`, `/sitemap.xml`, `/llms.txt`
Pools	`GET/POST /api/v1/pools`, `GET/PATCH/DELETE /api/v1/pools/{id}`
Pool health	`GET /api/v1/pools/{id}/metrics`, `…/metrics/stream` (SSE), `…/metrics/history?metric=`
Pool timeline	`GET /api/v1/pools/{id}/events`
Node roles	`POST /api/v1/pools/{id}/nodes/{fellow_id}/role`
Buckets	`GET/POST /api/v1/buckets`, `GET/PATCH /api/v1/buckets/{id}`, `…/datasets`
Model catalog	`GET /api/v1/models`, `GET /api/v1/models/{id}`, `…/status`, `GET /api/v1/serving`
Model serving	`POST /api/v1/models/{id}/load`, `…/unload`
Inference	`POST /api/v1/chat/completions` (and `/v1/chat/completions`, OpenAI-compatible)
Inference transport	`GET /api/v1/infer/requests/{fellow_id}`, `POST /api/v1/infer/responses/{request_id}`
Web UI	`GET /dashboard`, `/pools/{id}`, `/buckets`, `/ops/join-v2`, `/ops/inference`, `/profile`

Every route is classified in leader/route_policy.py; anything not listed there is denied by default, so a new endpoint is unreachable until it is classified.

Control signal values¶

ControlSignal.signal supports:

START
PAUSE
CHECKPOINT
STOP
RECONFIGURE
UPDATE
PURGE
PING — a control-plane round-trip probe. Only sent to fellows that advertised the ping capability at registration: an older fellow treats an unknown signal as a protocol error and reconnects, so an unnegotiated ping would knock a running cluster into a reconnect loop.

Registration and heartbeat¶

sequenceDiagram
    participant Fellow
    participant Leader
    Fellow->>Leader: POST /api/v1/workers/register
    Leader-->>Fellow: RegisterResponse (node_token)
    Fellow->>Leader: WS /ws/signals/{fellow_id}<br/>Authorization: Bearer node_token
    loop every heartbeat interval
        Fellow->>Leader: POST /api/v1/workers/heartbeat<br/>status, step, loss, bytes, live hardware
        Leader-->>Fellow: HeartbeatResponse
    end

Heartbeats include live GPU hardware telemetry when available. The leader uses that payload to refresh per-GPU free VRAM in /api/v1/cluster/nodes and /api/v1/cluster/devices, so the Ops UI shows current VRAM utilization instead of only the registration-time inventory.

/api/v1/cluster/nodes reports each node's pool_id and accepts ?pool=<id> to narrow the list to one pool — the same filter /api/v1/ops/join-graph takes, so the ops console's table and graph always describe the same set of nodes.

Heartbeats may also carry a telemetry block (NodeTelemetry): per-GPU utilization, temperature and power, the round trip the fellow observes talking to the leader, round trips to its pipeline peers, and monotonic OOM / error / restart counters. Every field is optional — a fellow built before the block still registers and heartbeats, and consumers render its missing signals as unknown rather than zero.

Pages are full server renders, so anything the browser has to fill in is rebuilt on every navigation. The sidebar used to be exactly that: a skeleton, an identity fetch, then a swap — visible on each click.

leader/web/nav_render.py renders it server-side instead. The page builders stay free of request state (the same reason the CSP nonce is substituted late): the template carries tokens — __NAV_USER__, __NAV_POOL_NAME__, __NAV_STATE_<key>__, __NAV_ADMIN_HIDDEN__, __USER_NAME__ — and the HTTP middleware substitutes them once per response, where the session cookie is available. The active pool comes from the swarm_pool cookie, so the server renders the same chip the user selected. Every value that originates with a user (display name, pool name) is HTML-escaped there.

auth_header.js no longer fetches identity or pools on load; it only wires the menus and the drawer, and loads the pool list lazily when the switcher is opened.

Pool telemetry¶

PoolTelemetry (leader/telemetry/pool_telemetry.py) keeps a bounded in-memory window of health signals per fellow and rolls them up per pool. Nothing polls a fellow; every input is something the leader already receives:

flowchart LR
    HB[heartbeat handler] -->|availability, step rate, node telemetry| PT[PoolTelemetry]
    WS[signal WS hub] -->|ping/pong RTT, disconnects| PT
    REG[register route] -->|flap counter| PT
    CHAT[chat completions + broker] --> SM[ServingMetrics]
    SM -->|tokens/s, TTFT, queue wait, errors| PT
    PT --> API["/api/v1/pools/{id}/metrics (+ SSE)"]

Design rules that the implementation holds to:

The request path never blocks on telemetry. Recording a generation is a few dict writes and perf_counter() reads into a bounded deque; a telemetry failure can never fail a generation (the broker's dispatch hook swallows its own exceptions).
One clock. Every duration is a delta of leader-side perf_counter() reads, or a delta the fellow measured about itself. No metric depends on leader and fellow clocks agreeing.
Bounded memory. Each node keeps at most ~1024 samples (≈1 h at the 5 s heartbeat interval); released nodes are forgotten so they stop skewing pool roll-ups.
Unknown is not zero. A signal that was never reported is null, so a node that simply cannot measure something is not penalised for it.

Availability is received heartbeats ÷ expected heartbeats over the window, so a node cannot hide an outage by reconnecting; it stays null until a node has been observed for at least two intervals.

The rest of leader/telemetry/ builds on that snapshot:

Module	Turns the snapshot into
`grades.py`	one explainable 0-100 score and an A-F band per node, with hysteresis so a band does not churn
`pool_metrics.py`	stored history (minute resolution for 7 days, hourly for 90) and the pool event timeline
`vram_budget.py`	what a serving stage needs — weights plus the KV cache its layers will allocate — which gates model loads
`replacement.py`	the decision to quarantine a failing node and promote a standby, guarded by dwell times and a cooldown

notify/alerts.py turns the resulting events into email/webhook notifications. The operator-facing description of all of this — grading rules, the replacement triggers and their guards, the VRAM arithmetic — is in the Pools guide.

Data plane¶

Direct fellow-to-fellow traffic uses ZeroMQ multipart messages:

flowchart LR
    F0[Fellow i] -- activation PUSH :5557 --> F1[Fellow i+1]
    F1 -- gradient PUSH :5558 --> F0

Each packet carries metadata plus raw tensor bytes. The relay path uses the same multipart semantics over leader HTTP endpoints:

Relay route	Purpose
`POST /api/v1/relay/{job_id}/{channel}/push`	Push encoded frames into a channel.
`GET /api/v1/relay/{job_id}/{channel}/pop`	Pop encoded frames, optionally blocking.

Transport selection is controlled with SWARM_PIPELINE_TRANSPORT=auto|tunnel|p2p.

Authentication notes¶

The swarm CLI can send Authorization: Bearer <token> via --token or config.
Bootstrap downloads can require a pool/invite token when SWARM_BOOTSTRAP_REQUIRE_TOKEN=1 — via X-Bootstrap-Token header or ?token= query param. The script is HTTPS-only and the manifest publishes source_archive_sha256 / script_sha256 for verification (see bootstrap).
Production deployments should enforce broader authentication at the leader or ingress layer.