Pools¶

A pool is a scheduling and access boundary: a set of nodes, members, buckets, and models that belong together. A training job only ever combines nodes from a single pool. The pool switcher at the top of the sidebar picks the active pool, the one the Training and Inference consoles act on. Opening a pool's page never changes it; the page tells you whether it is the active pool and offers Make active when it is not.

Pool detail — `/pools/{owner}/{pool}`¶

The page for a single pool, split into tabs. Owners and admins get the management controls; members get a read‑only view of most of it.

Pool detail

The pool's name with its badges: Public or Private, the pool grade (Grade C · 79.4, see Pool health), and Active pool when this is the pool the Training and Inference consoles act on.
The description, node count against the plan's quota, member count, and the pool's address as owner/slug (see Organizations and addresses). A pool owned by an organization shows it in the breadcrumb.
Make active — shown when the pool is not the active one; it switches the sidebar's active pool to this one without leaving the page.
Edit (owners and admins) — opens the Settings tab with the name field focused.
Add a node — the page's primary action. It opens a dialog: tick Join as a standby to have the machine arrive as a warm spare rather than a schedulable node, then Generate command for the pool‑scoped curl to run on it, with a Copy button. The command is valid for an hour; anyone who has it can add a machine to the pool. Visitors of a public pool they have not joined see Request to join here instead.
More actions (⋯) — open the pool in the Training or Inference console, copy the link or the pool ID, Leave pool (members other than the owner) and Delete pool (owners and admins). The destructive actions come last and ask for confirmation.

Tabs¶

Each tab has its own address (/pools/{owner}/{pool}/nodes), so a reload or a shared link opens the same tab. Old #tab-nodes links still open it.

Overview — the Pool health KPI tiles (nodes online, tokens/s, p95 TTFT, requests/min, error rate, the deployed model) with Capacity ready in the card header, and Recent events: node joins, drops, returns, role and grade changes.
Nodes — the graded node table, one row per machine. Owners and admins change a node's role from the ⋯ menu at the end of its row. Below it, Speed and network ranks the pool's GPUs by measured speed.
Jobs — training runs on this pool's nodes, with status, step, loss and start time, and a link to the Training console.
Models — models trained in this pool and their status, with a link to the Inference page.
Members — who belongs, with role and join date; the ⋯ menu on a row removes a member (or lets you leave). Owners and admins also get Invite by email, the Pending invites and Join requests lists, and for a public pool the Open join switch, which lets anyone join without approval.
Settings (owners and admins) — rename the pool or change its description, and Delete pool. Deleting detaches any attached nodes and removes members, invites and join requests; it cannot be undone.

Node health, events, grades, the node list and the member list are for members: someone who opens a public pool they have not joined sees that it exists, not whose machines are in it or who belongs to it.

Pool health¶

The Overview and Nodes tabs describe the pool as a system rather than as a list of machines.

Pool grade in the page header — an A–F letter with the 0–100 score, the mean of the graded nodes' live scores. Capacity ready says whether every stage of every deployed model is loaded on a node that is still online.
KPI tiles — nodes online/total, tokens/s, p95 TTFT, requests/min, error rate, and the deployed model. Tokens/s, TTFT and errors carry an hour‑long sparkline drawn from the stored history, so a reload does not lose the trend.
Node table — one row per machine: grade, status, role, availability, round trip, GPU utilization, free VRAM, straggler lag and failure counters. Each node's pill shows its own band only; select it to see why: the breakdown shows the score, every component with the raw number behind it (fellow-reported leader RTT 11.5ms vs 50ms target) and any penalties. Owners and admins move a node between roles from its row's ⋯ menu — Hold as spare and Promote to active first, the moves that take a machine out of service (Drain, Quarantine) last and confirmed — so a machine can leave scheduling without being removed from the pool.
Recent events — so a dip in a chart can be lined up with what happened.

How a node is graded¶

A weighted score over five components — availability (0.30), reliability (0.20), latency (0.20), throughput (0.15) and capacity (0.15) — minus penalties for flapping, crash loops, straggling and out‑of‑memory events. Override the weights with SWARM_GRADE_WEIGHTS (JSON, renormalised automatically).

Three rules keep grades trustworthy:

Unknown is dropped, not zeroed. A component a node cannot report is removed and the remaining weights are renormalised, so a machine is never punished for a signal it cannot produce.
Evidence is required. A node needs at least one behavioural signal (availability, reliability, latency or throughput) before it is graded at all; free VRAM alone describes a machine that has not done anything yet. Until then it shows —, which is a different statement from a bad grade.
Bands hold still. A band change must persist for SWARM_GRADE_DWELL_S seconds (default 120) before it takes effect, so a score oscillating around a boundary does not churn. A pending change is shown on the node's pill as B → C, and its tooltip and accessible name spell it out: the score already reads C, and the grade follows once that has held - the text names the dwell your leader is configured with and roughly how long is left. This is why a node can show B while the pool grade in the header, which follows the live scores, already says C. Going offline is exempt: that is a fact, not a noisy sample, and grades F immediately. The dwell is measured by the leader's once-a-minute roll-up (SWARM_METRICS_ROLLUP_INTERVAL_S), not by whoever is looking: opening the pool page or scraping /api/v1/metrics reads the grades without moving them, so a band settles on the same schedule whether or not anyone is watching.

Pool grades also appear as a pill on each dashboard pool tile, and as a coloured ring around each node in the training console's swarm graph.

Node roles and standbys¶

Every node in a pool holds one of five roles:

Role	Meaning
active	Schedulable. Training jobs and served models are placed on these nodes.
standby	Registered, warm and never scheduled — a spare held in reserve. Loading a model onto one by name is refused until it is promoted.
draining	On its way out of service; no new work is placed on it. A training stage it runs moves to another node at the next saved step.
quarantined	Taken out by an operator or by policy, kept registered for diagnosis. Never promoted as a replacement until someone returns it to service.
probation	Just joined and not yet through its admission self-test. Never scheduled, never promoted.

Add a spare by generating a join command with role: "standby":

curl -X POST https://<leader>/api/v1/bootstrap/join-sessions \
     -d '{"pool_id": "<pool-id>", "role": "standby"}'

Owners and admins can move a node between roles from the ⋯ menu on its row in the pool page's Nodes tab, or via the API:

curl -X POST https://<leader>/api/v1/pools/<pool-id>/nodes/<fellow-id>/role \
     -d '{"role": "standby"}'

Two things worth knowing: a standby counts against the pool's node quota (it occupies a slot in your cluster whether or not it is working), and it is graded without the throughput component — an idle spare is doing exactly what it is held in reserve to do, so it is not marked down for producing no tokens.

Admission, checkout and availability¶

Probation. A node that joins a pool on the current version starts on probation and runs a self-test the leader sends it:

each GPU must really allocate 90 % of the free memory it reports - a card that claims 24 GB and holds 8 is caught here;
its GPU must compute the known answer of a small network whose weights come from a seed the leader draws for this test, matching what the leader computes itself - a GPU that returns wrong numbers never trains anything;
its speed is measured and kept for the speed ranking: matmul TFLOPS and memory bandwidth per GPU, and its bandwidth to the leader.

A node that passes takes the role its join command named (active or standby). One that fails stays on probation, with the reason on the pool page; the pool's owner can Run its self-test again or Admit without its self-test from the node's ⋯ menu. Nodes from before the self-test are not put on probation. Switch it off with SWARM_PROBATION=0.

Checkout. A node's host can take the machine back:

fellow checkout --in 15m    # on the machine itself
fellow checkin              # give it back

or Check out for 15 min in the node's menu (the node's member, the pool's owner or an admin; POST /api/v1/pools/{id}/nodes/{fellow}/checkout). The node turns draining: a training stage it runs is moved to a standby - or the model split over the nodes left - at the next saved step, so no step is lost, before the deadline; nothing new is placed on it; and its grade holds while it is away - a checked-out machine going offline is not held against it. Check-in returns it to the role it had.

Availability windows. A node can have weekly windows it is not available, e.g. never between 18:00 and 23:00 on weekdays:

curl -X PUT https://<leader>/api/v1/pools/<pool-id>/nodes/<fellow-id>/availability \
     -d '{"windows": [{"from": "18:00", "to": "23:00", "days": [0,1,2,3,4], "tz": "Europe/Berlin"}]}'

Fifteen minutes before a window the node is checked out on its own, and checked back in after it. No new job is placed on a node within an hour of its next window. GET /api/v1/pools/{id}/gatekeeper (MCP pools_gatekeeper) lists the pool's self-tests, checkouts, windows and quarantine history.

Speed and network¶

The Speed and network card on the Nodes tab ranks every GPU in the pool by how fast it measured, and shows how fast each node's connection is.

GPUs, fastest first. Each row is one card: its rank, its name with the node, GPU index, architecture (Ampere, Ada, Turing, ...) and bf16 when it can train in bfloat16, then:

Column	What it is
Speed	A bar and a percentage: how fast this card is relative to the fastest card in this pool (100 %). Green from 75 %, amber below 40 %.
Compute	Matmul throughput in TFLOPS - float16, the precision model layers run in (float32 when a card here was measured by an older fellow). Prompt processing and training scale with it.
Memory	Memory bandwidth in GB/s. Generating a token reads every weight of a stage once, so this sets decode speed.
VRAM	The card's memory.
Measured	When, and whether at the node's admission or by a benchmark. Last try: busy or Last try failed keeps the previous numbers.

Speed is the geometric mean of the compute and memory figures, each relative to the pool's best. A card that was never measured shows Not measured and has no rank - unknown is not slow. The line above the table names the fastest and the slowest card and how many times faster one is than the other.

Network. One row per node: Direct when it can take direct links to other nodes (it is on the mesh, its ports are open, or a direct link to a peer was seen), Via leader when its pipeline traffic goes through the leader's relay; the round trip to the leader (live); and its measured upload and download to the leader, as bars against the fastest node. Under the table, Between nodes lists the links seen so far: the mesh path (direct or via relay) and the round trip measured when a training run started.

Measuring. Every node measures itself when it joins (the admission self-test above). Measure on a node's network row runs the same benchmark again - for the pool's owner, an admin, or the node's own host. It allocates no memory and changes nothing: the node keeps its role, unlike Run its self-test again, which puts it back on probation. It takes up to a minute; a node that is training or serving answers busy rather than measure under load, and a GPU that computes the known answer wrong is not ranked on those numbers. A node running a fellow from before the speed figures shows Update to measure.

Training places layers by these figures. A training run with layers weighted between fellows (partition: weighted, what the New run form and a bucket's run use) starts from the free-VRAM split. When every placed card has a figure, layers then move toward the faster cards as long as the predicted step gets shorter and every card still holds its share by the memory estimate. See Training: layers by speed. An unmeasured card keeps the whole run on the free-VRAM split, and the plan names that card: Measure it. Serving still splits by free VRAM.

API: GET /api/v1/pools/{id}/speed (members; MCP pools_speed) and POST /api/v1/pools/{id}/nodes/{fellow}/benchmark (MCP pools_benchmark_node).

Automatic replacement¶

When a pool holds standbys, the leader can replace a failing active node on its own. This is off by default (SWARM_AUTO_REPLACE=0): in dry-run mode the policy still evaluates every minute and writes what it would have done to the pool timeline, so you can read a week of its intentions before letting it act.

Triggers, each of which must hold for a dwell time before it counts:

Trigger	Default dwell	Env
No heartbeat	60 s	`SWARM_REPLACE_OFFLINE_S`
Grade F	120 s	`SWARM_REPLACE_GRADE_S`
Straggling ≥ 5 steps behind	300 s	`SWARM_REPLACE_STRAGGLER_S`
≥ 3 out-of-memory events	immediate	`SWARM_REPLACE_OOM_S`
≥ 5 re-registrations in 24 h	immediate	`SWARM_REPLACE_FLAP_S`

When one fires, the leader picks the highest-graded standby with enough free VRAM for the failing stage, quarantines the failing node and promotes the standby, so the next job or model load uses it. Every step is audit-logged and appears in the pool's event feed.

What happens to a running training job. A pipeline run whose node is gone (no heartbeat past the dwell) continues without it, from the last step every stage saved. The leader opens a recovery incident and works through it step by step - each one audit-logged and on the pool's timeline:

Fence. The job turns Recovering at its last saved step, and every stage is told to stop. The run gets a new placement epoch: whatever the stopped run still sends - a frame on the relay, a link message, a progress report, a snapshot upload - is refused or ignored, so a node that comes back from a network outage cannot write into the new run.
Pick. The best-graded standby that is online, idle, up to date and has the GPU memory for the stage takes it. Without one the job's train.on_node_loss decides: wait (the default) for a standby to appear, up to train.recovery_wait_s (10 minutes), then split the model again over the nodes left; resplit at once; fail. A new split needs at least two nodes left and is refused when the job trains a learned bottleneck between stages, which belongs to its boundary and would start over.
Move. The lost node is quarantined, the standby made active.
Resume. Every stage starts again from its saved state - or, after a new split, from the saved states of the old stages whose layers it now holds.
Verify. The first step past the saved one closes the incident: Running again. The steps trained again are recorded on the job, on the timeline and against the lost node's owner's contribution.

A job is recovered at most 3 times an hour; after that, or when a step cannot be done (no saved step yet, a node too old to take part, nothing that fits), the job is failed with the reason, as before. A run without a saved step fails like it always did, and the lost node is still replaced for the next job. Follow a job's incidents with GET /api/v1/jobs/{id}/incidents (MCP jobs_incidents) and a pool's with GET /api/v1/pools/{id}/incidents (pools_incidents); the Training page's Fellows tile says how often a run was recovered. Once a lost node is back, a stopped run can also be resumed by hand (see Training).

A node that is still running a stage but judged weak (grade F, straggling, out-of-memory errors) is migrated, not evicted: every stage of the job saves a step a little ahead and stops right after it, and the run continues from exactly that step on a standby - no step is lost. Without a standby the weak node keeps working, and is replaced once the job has ended. A node that is re-registering over and over (crash loop) counts as gone. A node whose grade is slipping (a worse band pending, or band D) makes the job it runs save twice as often from then on.

Guards, all of which have to pass:

One replacement in flight per pool, plus a cooldown afterwards (SWARM_REPLACE_COOLDOWN_S, default 300 s).
Never promote an ungraded node (no track record), one on probation, or one that was previously quarantined — returning it to service from the node's menu clears that (the history is kept across leader restarts).
Never quarantine a node when no standby can take over: turning a degraded pool into a dead one is not an improvement.

Covers. Standbys do not just wait: each one covers a stage of what the pool runs, the riskiest first (the holder's grade times the stage's size).

A hot cover holds a served model's stage loaded on the standby, out of the ring - it takes no traffic. If the stage's node is lost, the cover is switched in: the model serves again in seconds instead of after a load. Hot when 80 % of the standby's free VRAM holds the stage (SWARM_COVER_VRAM_SHARE).
A warm cover holds only the files: a served stage's files in the fellow's cache, or a training stage's base checkpoint files - so a takeover loads them and the snapshot without downloading anything (the fellow's log says Warm cover hit).

The Covers card on the Nodes tab lists which standby covers which stage, hot or warm, and which stages have none. A pool with no standby - or fewer standbys than stages - says how many stages a failover would load from scratch. A cover lives in a slot of its own on the standby: it does not count against SWARM_MAX_LOADED_MODELS and is never unloaded as idle. MCP: pools_covers.

Placement¶

The Placement card on the pool page's Nodes tab shows which node holds which part of which workload right now: each training job and each served model, its pipeline stages, the layers each stage holds, the node, and the stage's state:

State	Meaning
assigned	The leader sent the work; the node has not reported on it yet.
loading	The node is loading a model stage.
ready	Training, or serving.
failed	The node reported an error loading it (the message is shown).
lost	The node stopped heartbeating with this stage on it.

The record is kept by the leader and survives restarts. Each new start of a job or load of a model opens a new epoch; what was placed before is kept as history. Work the leader finds running without a record (a model loaded before an upgrade, say) is adopted from the nodes' heartbeats. Members only, like the rest of the node detail. The same data is at GET /api/v1/pools/{pool_id}/placements, and per training job, every epoch included, at GET /api/v1/jobs/{job_id}/placements (MCP: pools_placements, jobs_placements).

Today the placement record only observes. It is what recovering a run on a replacement node will build on.

Alerts¶

Set SWARM_ALERT_EMAILS (comma-separated) and/or SWARM_ALERT_WEBHOOK to be told when a node is quarantined, a standby is promoted, a node needs replacing but cannot be (no eligible standby, or it is still running a job), or a pool's capacity is no longer ready. Repeats of the same alert for the same node are suppressed for SWARM_ALERT_REPEAT_S (default 30 min) so a flapping machine cannot train you to ignore the alerts.

Serving VRAM¶

Loading a model onto a pool costs more than its weights: each pipeline stage also keeps a KV cache for the layers it holds, sized by the context length you serve. The leader budgets both before it loads anything, and refuses a placement that cannot hold them — stage 0 on gpu-1 needs 15.2 GB (9.0 GB weights + 4.8 GB KV cache at 8192 tokens), has 12.0 GB free — rather than letting the node run out of memory once real requests arrive.

The node table's VRAM free column (Nodes tab) carries the budget as a tooltip, and the deployed‑model tile shows what the cache costs at your configured context. Set that context with SWARM_SERVE_CONTEXT_TOKENS; the budget scales linearly with it, so halving the context halves the cache. On current fellows that context is the longest single conversation the pool takes: the requests in flight share one token budget per stage, sized for typical chats of SWARM_SERVE_TYPICAL_TOKENS (1,024), and a conversation longer than the whole budget is refused with the numbers instead of running a card out of memory.

Models exported before this existed do not record their attention shape. They are admitted on weights alone — and the budget says so rather than pretending the cache is free — until their first successful load, which teaches the manifest the shape for every load after that.

Pool metrics¶

Every pool exposes a live health snapshot describing it as a system, not as a list of machines. Any member of the pool can read it:

curl -s https://<leader>/api/v1/pools/<pool-id>/metrics | jq

The payload has four parts:

pool — nodes online/offline/training, GPUs, aggregate VRAM and RAM, mean GPU utilization, current step, worst straggler lag, average loss, mean availability, and control‑plane RTT.
serving — over a rolling window: tokens/s, requests/min, error rate broken down by class (fellow error, timeout, client disconnect, model not loaded), p50/p95 time‑to‑first‑token, p50/p95 queue wait, mean inter‑token gap, and in‑flight requests.
deployed_models — which models are loaded, on which nodes, in how many stages, and one overall state per model (the worst any stage reports, so a model with one stage still loading is not "ready").
nodes — one row per machine: availability, heartbeat age, control‑plane and leader round trips, GPU utilization/temperature/power, VRAM, step rate, stall seconds, straggler lag, OOM/error/restart counters, and its slice of the serving traffic.

Each node row also carries its grade — score, band, the components it was built from, and any penalties. A band only changes once the new one has held for the dwell time (SWARM_GRADE_DWELL_S, default 120 seconds), so a short dip does not churn the grade. The payload says so rather than leaving it to the client: pool.grade_dwell_s is the configured dwell, and a node whose score has moved into another band shows it as grade.pending_band, with grade.pending_in_s - the seconds until it takes over if the score stays there (0 means it applies on the next update; null when nothing is pending).

GET /api/v1/pools/<pool-id>/metrics/stream is the same payload as a server‑sent event stream for live dashboards.

History. GET /api/v1/pools/<pool-id>/metrics/history?metric=tokens_out_per_s&window_s=3600 returns a stored series, optionally for one node (&fellow_id=). The leader rolls metrics up once a minute, keeps minute resolution for 7 days, folds older points into hourly means for 90 days, and buckets every response down to max_points (default 200) — so a 90‑day window costs no more to read than an hour does.

Events. GET /api/v1/pools/<pool-id>/events returns the pool timeline: nodes joining, leaving, dropping offline, returning, and grade changes.

Activity across pools. GET /api/v1/pools/activity?limit=20 returns the same event records for every pool you may read the timeline of, newest first, each with an extra pool_name - one call instead of one per pool, which is what the dashboard uses. The scope is the per-pool rule applied to all pools at once: the pools you are a member of (every pool for a platform admin); a public pool you have not joined is not included. limit defaults to 20 and is capped at 100. It accepts a browser session or an API token, and a token sees exactly what its owner would:

curl -s -H "Authorization: Bearer $SWARM_API_TOKEN" \
  "https://<leader>/api/v1/pools/activity?limit=50" | jq '.[] | {at, pool_name, kind, detail}'

A signal a node never reported is null, not 0 — a machine that cannot read its own power draw is not the same as one drawing no power. Nodes that just joined report null availability until they have been observed long enough to say something honest.

Pools admin — `/admin/pools`¶

Pools admin

Admin‑only overview of every pool on the leader (not just the ones you belong to): create pools, inspect membership and node counts, and manage or delete any pool. Regular users manage their own pools from the dashboard and the pool detail page instead.