Skip to content

Operations

This section documents runtime operations for SilentSwarm clusters, especially leader/fellow startup, remote onboarding, and troubleshooting.

flowchart LR
  Start[Start leader] --> Fellows[Attach fellows]
  Fellows --> Inspect[Inspect nodes and devices]
  Inspect --> Run[Run jobs or sweeps]
  Run --> Observe[Watch metrics and logs]
  Observe --> Manage[Update, release, or purge fellows]
  Manage --> Stop[Stop cluster]

Current topics

  • Local and staging startup procedures.
  • Service health checks and observability.
  • Leader-served fellow onboarding through curl bootstrap.
  • Ops UI join flow and onboarding status.
  • Fellow release, update, and purge workflows.
  • Pipeline transport selection for tunnel and direct P2P modes.
  • Troubleshooting playbooks for bootstrap, leader connectivity, and port conflicts.

Documents

  • Bootstrap quick start: common local, remote, token, and tunnel onboarding paths.
  • Bootstrap contract: bootstrap manifest, shell contract, join flow, environment variables, and troubleshooting.
  • Kubernetes (Helm): the control plane with helm install - values, storage, backup, upgrades.
  • Fellow in a container: the fellow image - docker run --gpus all, its state volume, stopping without losing a step, upgrades.

Local cluster lifecycle

swarm start-cluster --fellows 2
swarm nodes
swarm fellows
swarm stop-cluster

Local process logs are written under .swarm/runtime-logs/. Docker Compose users normally reach the leader through the nginx frontend at http://localhost:3000.

Docker and tunnel access

docker compose --profile local-fellow up -d --build
swarm --leader-url http://localhost:3000 nodes

For temporary public access to the Ops UI and bootstrap flow:

docker compose --profile local-fellow --profile tunnel up -d --build
docker compose --profile tunnel logs -f tunnel

The tunnel log prints a public /training URL.

Upgrading an existing deployment

Two things changed in the leader image that a running deployment notices:

  • The leader runs as an unprivileged user (uid 10001). The entrypoint takes ownership of the /data volume on first start, so a normal docker compose up -d --build leader is enough; if you bind-mount extra paths into the container, chown -R 10001:10001 them yourself. Overriding user: in compose skips the entrypoint's chown entirely.
  • The frontend listens on 8080, not 80 (K1). Compose and the prod overlay's Traefik label are updated with it, so rebuild and restart the frontend together with the new compose files; a hand-written proxy that pointed at the container's port 80 must move to 8080.
  • Dependencies come from requirements.lock (pinned + hash-verified). Change requirements.txt, then regenerate with scripts/lock_deps.sh and commit both - CI fails if the lock is stale.

The enforcement flags (SWARM_REQUIRE_NODE_AUTH, SWARM_REQUIRE_OPERATOR_AUTH, SWARM_REQUIRE_CSRF) are now on unless set to 0, so fellows must be running a build that sends their node credential on the signal WebSocket before you redeploy the leader - see Users and authentication.

Where the leader keeps its state

Every piece of leader state lives under one root, SWARM_DATA_DIR. The leader image sets it to /data, the volume; outside a container it defaults to .swarm in the working directory.

Path under the root What Its own variable (wins when set)
join_state.sqlite3 users, pools, jobs, nodes, audit log SWARM_JOIN_STATE_DB
api_tokens.json, buckets/, snapshots/ tokens, bucket files, stage snapshots - next to the DB SWARM_API_TOKENS_FILE, SWARM_BUCKETS_ROOT, SWARM_SNAPSHOT_ROOT
servable_models/ models the Inference page serves SWARM_SERVABLE_MODELS_ROOT
model_exports/ trained jobs' uploaded shards SWARM_LEADER_MODEL_EXPORT_ROOT
checkpoints/ checkpoint metadata SWARM_CHECKPOINT_ROOT

Before this root existed, job model exports defaulted to /app/.swarm and checkpoint metadata to /tmp/swarm_checkpoints - inside the container, so a redeploy lost them. Back up the root and you have backed up the leader.

Behind any proxy, in any cluster

The leader and frontend images carry nothing specific to compose: the same two run under the bundled compose files and in a Kubernetes pod.

  • Frontend. Unprivileged nginx: it runs as uid 101 and listens on 8080 (compose maps SILENTSWARM_FRONTEND_PORT to it; the prod overlay's Traefik label points at 8080). Its config is baked into the image, so edits to docker/nginx/frontend.conf need docker compose build frontend, like edits to docs/. Its settings are environment variables:
Variable Default What
SWARM_LEADER_UPSTREAM leader:8080 where nginx finds the leader. In Kubernetes give the Service's full name, leader.<ns>.svc.cluster.local:8080: nginx's resolver ignores search domains
SWARM_EDGE_TRUSTED_PROXIES loopback + private ranges proxies whose X-Forwarded-For nginx believes, so edge rate limits key on the client; off believes none (Client IP)
SILENTSWARM_DOCS_URL / SILENTSWARM_DOMAIN the project's docs the docs' canonical URL, written at start (public web surface)

The DNS resolver is read from /etc/resolv.conf: Docker's embedded DNS under compose, kube-dns in a pod. - Readiness. GET /api/v1/health is liveness. GET /api/v1/ready answers 503 with a reasons list while the leader starts or shuts down, when the state DB takes no write, or when its volume has less than SWARM_MIN_FREE_DISK_GB free. Point a readiness probe at it. With one leader replica a failing probe takes the whole site out of the load balancer, so keep the disk floor below the point where you would rather be down. - First admin without a terminal. An install job can run swarm-leader admin create-user ops@example.com --password-file /secrets/admin --if-missing (or take the password from SWARM_ADMIN_PASSWORD); a rerun leaves the account alone. - Headscale's port. HEADSCALE_LISTEN_ADDR overrides listen_addr in docker/headscale/config.yaml (compose: SWARM_HEADSCALE_LISTEN_ADDR, default 0.0.0.0:8080). In a pod beside the leader, which owns 8080, use 0.0.0.0:8081. The leader reaches Headscale through its socket (SWARM_MESH_HEADSCALE_SOCKET), not the port.

Relay memory. The HTTP relay holds queued frames in the leader's memory. A busy channel peaks at about SWARM_RELAY_MAX_FRAME_MB × SWARM_RELAY_QUEUE_MAX, which is 512 MB × 64 = 32 GB by default, a ceiling no real job comes near: a serving prompt of 526 tokens crosses as 1.4 MB compressed or 10.8 MB uncompressed (benchmarks), and a training boundary carries batch × sequence × hidden × 2 bytes per step. Under a container memory limit, lower both so their product, times the channels you expect to be busy at once, fits beside the leader's own few hundred MB. A frame above the limit is refused with 413, a full channel with 503, rather than the leader being OOM-killed.

Remote fellow onboarding

Use the Training console at /training or the bootstrap API documented in Bootstrap contract. Join sessions can carry a token and stream logs back to the leader while the remote host installs and starts the fellow runner.

Runtime management

  • swarm update [--fellow-id ID]: ask connected fellows to re-bootstrap from the leader.
  • swarm release --fellow-id ID: remove a fellow from the leader registry.
  • swarm release --fellow-id ID --purge-install: release and request local cleanup.
  • swarm pending-purges and swarm retry-purge: inspect and retry cleanup for fellows that were offline during purge delivery.
  • swarm watch <job_id>: stream live merged metrics from the leader.

Observability

  • Training console: /training (or a pool's /pools/{owner}/{pool}/training) is a read-only join and monitoring surface. Selecting a fellow in the graph or table opens its runtime logs automatically and shows status, network speed bars, step/loss, VRAM used/free, and a live VRAM bar per GPU. Selecting the leader shows cluster resources: aggregate VRAM, pipeline-link throughput (activations and gradients, from the act_bytes/grad_bytes counters /api/v1/cluster/nodes carries), leader (control-plane) traffic, and the active job with one card per pipeline stage.
  • Worker logs: GET /api/v1/workers/{fellow_id}/logs/stream; the Ops UI hydrates logs from the REST snapshot first and keeps polling as a tunnel-safe fallback.
  • Fellow progress logs: each training progress post is logged at info level with job ID, fellow ID, step, loss, status code, HTTP latency, request/response bytes, and cumulative fellow byte counters. Fellows also update their local step/loss immediately so heartbeat metrics and the UI do not wait for completion.
  • Computation placement: leader dispatch ranks connected fellows by aggregate free GPU VRAM across eligible GPUs on each node. GPUs with less than placement_min_gpu_free_gb free memory are ignored; the default is 4.0 GB and can be set globally with SWARM_PLACEMENT_MIN_GPU_FREE_GB or per job config. The floor is there to skip busy cards, so a card smaller than it only has to be idle: it needs min(placement_min_gpu_free_gb, total VRAM - 0.25 GB) free, which lets an idle 4 GB card (it reports about 3.9 GB free) take work under the default. When a job cannot be placed, its placement_blocked_reason (and swarm train --plan) names every GPU the floor excluded, with what it had free and what it needed. With --partition weighted, node layer counts are weighted by aggregate eligible free VRAM - then, when every placed GPU has a measured speed (telemetry/speed.measured_tflops), moved toward the faster cards while the predicted step shortens and the memory check still passes (speed SP4, training/speed_split.py; the plan's auto.split says which and why) - and each node placement contains a gpu_slots plan that splits the node's layer range across eligible GPUs by that GPU's currently free VRAM. The fellow exposes its assigned GPUs through CUDA_VISIBLE_DEVICES, and the torch worker - a single-fellow job as well as each pipeline stage - spreads the model over all of them. The per-GPU split is made on the fellow at load time, from each card's free VRAM measured then and the model's real sizes: every card is filled to the same share of its free memory, counting the embeddings on the first card and the final norm, output head and the loss's logits (batch x sequence x vocabulary, in fp32) on the last one - both on the first when the head is tied to the embeddings. A full fine-tune counts gradients and AdamW state on every weight, LoRA the weights alone. So on two equal cards the one holding the embeddings and head takes fewer blocks, and a bigger card takes more. The leader's gpu_slots decide which GPUs a job gets; their per-GPU layer ranges are a free-VRAM-weighted preview, not the split the worker loads.
  • Training memory check: before a job starts, the leader estimates what every placed GPU will hold and keeps the job pending when one would run out, instead of letting both fellows load their shards and fail at step 1 with a CUDA out-of-memory error. The parameter counts come from the checkpoint itself: the leader reads the JSON headers of its safetensors files from the Hub with two small range requests per file (about 100 KB for a 4B model; only the text decoder is counted, so a multimodal checkpoint's vision tower does not inflate it). Each stage's layers are then spread over its cards exactly as the fellow will spread them (the same arithmetic, pipe/split/memory.py), and each card is priced as: its blocks' weights in the dtype they are stored in (model.weight_dtype, else the job dtype; about 0.55 bytes per weight with model.quantization 4-bit), gradients and AdamW state on top when the weights train (not with LoRA or freeze_backbone), activations for backward (train.batch_size x train.seq_len), the embeddings on the first card, the final norm, head and the loss's logits on the last (with tied embeddings the last stage holds the embedding matrix too), plus 0.4 GB of CUDA context and a 5 % margin. Activations are priced per block type from measurements on Qwen3.5-4B: about 24 x batch x sequence x hidden x activation bytes for an attention block, 72 x for a linear-attention (Gated DeltaNet) block on transformers' plain-torch path. The first, uniform guess (16 x) let two jobs through that then ran a GTX 1050 Ti out of memory at step 1; with the measured figures and the 5 % margin the check matches what that cluster did (1 x 256 and 2 x 128 tokens per step trained, 3 x 128 and 4 x 128 did not). A card over its free VRAM (from the last heartbeat) blocks the job with placement_blocked_reason naming each such card, its layers and the parts, and what to change - a smaller train.batch_size when one fits, model.weight_dtype "float16", LoRA, 4-bit quantization or more cards. swarm train --plan and swarm jobs show print the estimate per GPU; it is stored on a started job as memory_estimate. The check is skipped (and says so) when the model is a local path, has no safetensors weights, or the Hub cannot be reached (SWARM_LEADER_HF_LOOKUP=0 turns the lookup off); "memory_check": false in the job config starts a job regardless. The logits often decide it for large vocabularies: Qwen3.5's 248k-token vocabulary costs about 3 GB at batch 1 x 1024 tokens.
  • Pipeline stage ordering: by default, SWARM_PIPELINE_LAST_STAGE_PLACEMENT=largest_vram moves the strongest selected fellow to the final pipeline stage, because that stage owns final_norm, the output head, loss, and gradient workspace. Set pipeline_last_stage_placement=preserve in a job config to keep the raw placement order for diagnostics.
  • GPU runtime check: a fellow that should train on NVIDIA GPUs needs the PyTorch runtime extras (pip install -e ".[runtime]", or requirements-runtime.txt - what the bootstrap installs). Hardware inventory and training capability are independent: a fellow reports GPUs from nvidia-smi/NVML even when the installed torch build has no CUDA support and every job silently runs on CPU. Confirm with python -c "import torch; print(torch.cuda.is_available())" on the host before interpreting a stress run.
  • GPU memory cleanup: training runs in a fresh torch subprocess (fellow/torch_dispatch.py), so a completed, failed, or cancelled job releases its CUDA context by exiting - an idle fellow does not sit on a model's VRAM. The worker pins MKL_THREADING_LAYER=GNU (torch links the GNU OpenMP layer; an Intel libiomp on the path aborts it), and starts with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True unless the host sets that variable itself: a stage close to the end of its card's memory otherwise fails on fragmentation alone (a resumed 14B stage on a 6 GB card could not place 216 MiB with about 450 MiB free or reserved but unused). If a deployment is supervised by K8s, systemd, or another restart manager and wants the whole fellow recycled after each job, set SWARM_FELLOW_EXIT_AFTER_TRAINING=1 so it exits after posting the final job status and is restarted by the supervisor.
  • Live VRAM in the Ops UI comes from heartbeat hardware telemetry. Registration-time GPU inventory is only the initial snapshot; while fellows are running, each heartbeat refreshes per-GPU free VRAM so node bars and GPU placement rows show current used/total memory.
  • Pool telemetry: GET /api/v1/pools/{pool_id}/metrics (snapshot) and .../metrics/stream (SSE). One payload describes the pool as a system: serving throughput (tokens/s, TTFT, queue wait, error classes), training progress and straggler lag, per-node liveness (availability, control-plane RTT, heartbeat age), capacity, and which models are deployed on which nodes. Member-scoped: per-node telemetry never crosses a pool boundary. SWARM_POOL_METRICS_INTERVAL_S (default 5) sets the stream cadence.
  • Node health signals come from the fellow's heartbeat telemetry block: per-GPU utilization/temperature/power, the fellow's own leader round trip, peer round trips, OOM/error/restart counters, and gpu_processes: who holds memory on each GPU (from nvidia-smi --query-compute-apps, sampled at most every 30 s) - pid, name, MB, fellow (this fellow's own process tree: its training and serving workers) and swarm (any SilentSwarm process). A process with swarm: true but fellow: false is a worker left behind by an earlier fellow instance (kill it on that machine); one with neither flag belongs to something else; null means the fellow could not see it (another PID namespace). It shows in GET /api/v1/workers/{fellow_id} under telemetry and in the MCP tool fellows_get. The block is optional - a fellow that predates it heartbeats normally and its missing signals read as unknown, never as zero.
  • Control-plane RTT: the leader pings connected fellows over the signal WebSocket every SWARM_CONTROL_PING_INTERVAL_S seconds (default 15, 0 disables) and times the pong. This is latency measurement, not liveness - the heartbeat already covers liveness. Only fellows that advertised the ping capability at registration are pinged: an older build cannot parse an unknown control frame and drops its socket on one, so pinging everything would turn a latency probe into a reconnect storm. Nodes that predate the capability simply report an unknown control RTT until they are updated.
  • Metric history: GET /api/v1/pools/{pool_id}/metrics/history?metric=&window_s=&max_points= and the pool timeline at .../events. A roll-up runs every SWARM_METRICS_ROLLUP_INTERVAL_S seconds (default 60, 0 disables) writing minute-resolution rows; a daily retention pass folds rows older than 7 days into hourly means and drops anything past 90 days, so the tables stay bounded. Reads are always bucketed to at most max_points.
  • Node grades: each node scores 0-100 (bands A-F) from availability, reliability, latency, throughput and capacity, minus penalties for flapping, crash loops, straggling and OOM. Tune with SWARM_GRADE_WEIGHTS (JSON) and SWARM_GRADE_DWELL_S (default 120 - how long a band change must hold before it takes effect). Unreported components are dropped and the rest renormalised; a node with no behavioural evidence stays ungraded rather than being scored badly. Only the metrics roll-up advances the dwell clock; the metrics API, SSE stream and Prometheus scrape read grades without moving them.
  • Node roles: every pool node is active (schedulable), standby (warm spare, never placed), draining, or quarantined. Mint a spare with POST /api/v1/bootstrap/join-sessions {"pool_id": ..., "role": "standby"}; change a role with POST /api/v1/pools/{pool_id}/nodes/{fellow_id}/role (pool owner or admin, audit-logged). Standbys count against the pool's node quota. Serving follows the same rule as training: a model loads onto active nodes only, and POST /api/v1/models/{id}/load with a standby or quarantined fellow_id answers 409 node-not-schedulable. A served stage whose fellow has missed heartbeats past the registry timeout (15 s) no longer counts as ready: the model status says which stage stopped answering, and chats are refused at once instead of waiting for a relay timeout.
  • Automatic replacement: off by default. SWARM_AUTO_REPLACE=1 lets the leader quarantine a failing active node and promote the best-graded standby that has enough free VRAM for the stage. If the node is gone (offline trigger) while it holds a stage of a running pipeline job, the job goes to the Recovery Coordinator instead (next item); a job it cannot recover (no committed snapshot yet, a fellow without recovery) is failed with the reason and its other fellows paused, and the node is replaced as usual. If the node is alive but weak and still holds a stage, the replacement is deferred until the job ends. Dwell times per trigger: SWARM_REPLACE_OFFLINE_S (60), SWARM_REPLACE_GRADE_S (120), SWARM_REPLACE_STRAGGLER_S (300), SWARM_REPLACE_OOM_S, SWARM_REPLACE_FLAP_S; SWARM_REPLACE_COOLDOWN_S (300) spaces replacements out. A node counts as straggling at 5+ steps behind the pool - pipeline stages legitimately run a step or two apart. Repeated identical decisions are recorded once, not on every tick. In dry-run the policy still records what it would have done on the pool timeline - review that before enabling it.
  • Serving VRAM budget: a model needs its weights plus the KV cache its layers allocate. The leader budgets both before loading (2 x layers x kv_heads x head_dim x dtype_bytes x context x concurrency, per stage) and refuses a placement that cannot hold them, with the numbers in the message. A pipeline of current fellows shares one token budget per stage across its requests in flight, sized for typical chats (SWARM_SERVE_TYPICAL_TOKENS, default 1024) and never below one conversation of SWARM_SERVE_CONTEXT_TOKENS; stage 0 admits by it and refuses a conversation that can never fit (a 413). Tune with SWARM_SERVE_CONTEXT_TOKENS (default 4096), SWARM_SERVE_CONCURRENCY (default 1 - one generation per node today), and SWARM_SERVE_VRAM_SLACK (default 0.10). The attention shape comes from the servable manifest; a model exported before this existed records its shape the first time a fellow loads it, and until then is admitted on weights alone and says so.
  • Placement ledger: GET /api/v1/pools/{pool_id}/placements (members) and GET /api/v1/jobs/{job_id}/placements list which node holds which shard (layer range) of which job or model, per placement epoch, with the latest state changes. Rows live in the state DB (shard_placements, placement_events); training dispatch and model load/unload write them, and the sweep loop (every heartbeat interval, 5 s) reconciles them against heartbeats: assigned -> loading/ready, lost once a node is past the heartbeat timeout, retired when the job ends or the model is unloaded. Events older than 90 days are pruned; rows are kept. Read-only: nothing acts on it yet.
  • Stage snapshots (robustness R2): each stage of a pipeline job saves its trainable state, optimizer, schedule, step and RNG every SWARM_SNAPSHOT_EVERY_STEPS steps (default 50; a job's train.snapshot_every_steps wins, 0 turns it off for that job, SWARM_SNAPSHOTS=0 for the deployment) and uploads it to POST /api/v1/jobs/{id}/snapshots/{step}/stages/{stage} (node credential, only from the node running that stage, chunked like shard uploads, bounded by SWARM_MAX_UPLOAD_GB and SWARM_MIN_FREE_DISK_GB). Only when every fellow in the job declares stage-snapshot. Files live under SWARM_SNAPSHOT_ROOT (default snapshots/ beside the state DB, so /data/snapshots in the leader container); a step is committed once every stage's file passed its size, SHA-256 and header check, and the newest SWARM_SNAPSHOT_KEEP (2) committed steps of a running job are kept. A finished job keeps its newest for SWARM_SNAPSHOT_KEEP_DAYS (7), then none. Bytes count against the job's bucket owner's storage quota (507 storage-quota when full). Read: GET /api/v1/jobs/{id}/snapshots, a file at .../snapshots/{step}/stages/{stage} (operators) or .../node (a node of the job's pool). On the fellow, files go to SWARM_FELLOW_SNAPSHOT_ROOT (default .swarm/snapshots, or snapshots/ under SWARM_FELLOW_STATE_DIR - /state in the fellow image), uploaded on a background thread; the run waits up to SWARM_SNAPSHOT_UPLOAD_WAIT_S (120) at its end for the last one.
  • Resume (robustness R3): POST /api/v1/jobs/{id}/resume (operators, scoped like the job; swarm resume, MCP jobs_resume, the Resume button on the Training page) starts a FAILED or CANCELLED pipeline job - or a RUNNING one none of whose fellows still trains it, as after a leader restart - again from its newest committed snapshot, every stage on the node that ran it. The job is RECOVERING until a stage reports a step past the snapshot, then RUNNING. Refusals are 409 with a type naming the reason: no-snapshot, another-job-active, stage-nodes-offline, node-not-schedulable, capability-missing (a fellow without resume), job-still-training, stages-busy (a stage still runs the stopped run; it is sent PAUSE, retry once it is ready), job-finished. Uncommitted snapshot steps after the resumed one are deleted first. Every resume is audit-logged (job.resumed), leaves a job_resumed event on the pool timeline and a new placement epoch in the ledger, and is recorded on the job (config.resume, config.resumes: step, steps lost, previous status and error).
  • Recovery (robustness R4): a pipeline job that lost a node gets an incident (incidents table in the state DB) whose saga runs fence -> pick -> move -> resume -> verify, advanced by the sweep loop every heartbeat interval and resumed from its row after a leader restart. Fence: the job turns RECOVERING at its newest committed snapshot, its config.placement_epoch goes up, the relay lane is cleared, uncommitted later snapshot steps are dropped, every stage gets PAUSE. Pick: the best-graded standby that is connected, ready, declares resume and recovery, was never lost before and fits the stage (the start's memory estimate); without one the job's train.on_node_loss - wait (default, up to train.recovery_wait_s or SWARM_RECOVERY_WAIT_S, 600 s), resplit, fail - and a re-split re-plans the job over the nodes left (at least two, no learned bottleneck, uniform boundary compression). Move: lost node quarantined, standby active. Resume: START with each stage's ResumePoint once every node is ready (SWARM_RECOVERY_STEP_TIMEOUT_S, 300 s, per waiting step). Verify: the first progress past the snapshot closes it recovered (SWARM_RECOVERY_VERIFY_S, 1800 s). At most SWARM_RECOVERY_MAX_PER_HOUR (3) incidents per job and hour, then the job fails. Audit events recovery.opened|fence|pick|move|resume|closed, pool events of kind recovery, alerts job_recovered and recovery_failed, node_losses / steps_lost on the lost node owner's contribution. Read: GET /api/v1/jobs/{id}/incidents, GET /api/v1/pools/{id}/incidents (members). Fencing (also after a manual resume): relay channels of a restarted run carry its epoch (...-act.e2) and a push or pop on an older one answers 409 stale-epoch; status reports with an older epoch are ignored ("stale": true); snapshot uploads from an older epoch answer 409 stale-epoch; fellow links drop frames of another epoch.
  • Gatekeeper (robustness R5): judges nodes and writes their roles (leader/resilience/gatekeeper.py; tables quarantine_history, admissions, node_availability). Probation: a node joining with the selftest capability is probation until its self-test passes (SWARM_PROBATION, default on; SWARM_SELFTEST_MAX_GB 64 per GPU, SWARM_SELFTEST_TIMEOUT_S 900, SWARM_SELFTEST_ATTEMPTS 3, SWARM_SELFTEST_BUSY_RETRY_S 120); returning a probation or quarantined node to active/standby through the role route releases it. Urgency: offline and flapping are evict_now (R4 recovery); grade F, straggler and OOM in a running job are migrate - every stage gets STAGE_CONTROL stop_at a few steps ahead (SWARM_MIGRATE_LEAD_STEPS, 3), saves it and stops, and the job continues from that step (0 lost) if a standby can take the stage, else nothing moves; SWARM_MIGRATE_BOUNDARY_S (900) bounds the wait before it falls back to the last committed snapshot. Drain: an operator drain (role draining) or a host checkout (POST /api/v1/workers/{id}/checkout from fellow checkout, POST|DELETE /api/v1/pools/{pool}/nodes/{fellow}/checkout) does the same with the host's deadline, holds the node's grade (PoolTelemetry.excuse) and is not charged to its owner. Windows: PUT .../availability weekly windows; checked out SWARM_CHECKOUT_LEAD_S (900) ahead, checked back in after, and no job is placed within SWARM_AVAILABILITY_MARGIN_S (3600) of a window. Watch: a slipping node (worse band pending, or D) halves the job's snapshot cadence once (min SWARM_WATCH_MIN_EVERY_STEPS, 5). All require every stage to declare stage-control; without it migrate defers and drain falls back to the R4 path. Events node_admitted, node_admission_failed, node_checked_out, node_checked_in, node_watched; audit gatekeeper.*.
  • Speed (speed SP1): measured speed per GPU and link (leader/telemetry/speed.py; table node_benchmarks, the newest 20 rows per node). An admission that passes is recorded there; POST /api/v1/pools/{pool}/nodes/{fellow}/benchmark (the pool's owner or the node's host) sends the same SELFTEST probe with nothing to allocate and leaves the role alone - a busy node answers busy, a wrong known answer is not ranked, and no answer within SWARM_SELFTEST_TIMEOUT_S counts as failed. GET /api/v1/pools/{pool}/speed ranks the cards; audit gatekeeper.benchmark.
  • Serving failover (robustness R6, leader/resilience/serving_failover.py): every distributed model's stage payloads are kept in serving_plans at load and dropped at unload. The sweep loop opens a serve incident when exactly one stage's fellow is past its heartbeat timeout and every other stage declares serve-rewire (SWARM_SERVE_FAILOVER, default on): fence - survivors get REWIRE {epoch} (they end every session and switch to relay channels named act.e2, tok.e2, ...), the relay lane is cleared, the model is recovering; pick - a standby of the pool, else an active node with nothing loaded, connected, ready, never quarantined, with the lost stage's VRAM free (SWARM_SERVE_FAILOVER_WAIT_S, 300, before it gives up); move - LOAD_MODEL for that stage with transport.epoch, direct links recomputed and survivors given their new direct_peer; verify - closed once the model is ready (SWARM_SERVE_FAILOVER_LOAD_S, 1800). Giving up unloads every stage, as before R6. Meanwhile the broker holds requests that fail and replays them on the new ring with continue_text (the text so far) and the tokens left, up to 3 times, and new chats wait (SWARM_SERVE_REPLAY_WAIT_S, 600). A replacement fetches GET /api/v1/models/{id}/shards/{stage}/weights?shards= (node credential, its pool's models; a tar cached under the model's .stages/, served with Range). Audit serving.failover.*, pool events serving_failover.
  • Covers (robustness R7, leader/resilience/covers.py, table covers): every SWARM_COVER_INTERVAL_S (30) the sweep loop plans per pool which standby (connected, cover capability, never quarantined) covers which stage of the served models and the running job, riskiest first (1 - grade score, 0.5 ungraded, times layers); serving stages get a hot cover when SWARM_COVER_VRAM_SHARE (0.8) of the standby's free VRAM holds the stage's reported VRAM, everything else warm. COVER {action: hold|drop, kind, level, workload_id, stage_index}: hot loads the stage with --cover (no links until a REWIRE with activate), warm serving fetches its files, warm training runs runtime/torch/prefetch.py (the stage's base checkpoint files into the HF cache, SWARM_COVER_PREFETCH_TIMEOUT_S 7200). Fellows report covers in the heartbeat (covers). Serving failover activates a ready hot cover instead of loading; the training coordinator tries a warm cover first. GET /api/v1/pools/{id}/covers (members) lists covers and uncovered stages with a note.
  • Alerts: SWARM_ALERT_EMAILS (comma-separated) and/or SWARM_ALERT_WEBHOOK receive node-quarantined, node-promoted, replacement-blocked (no eligible standby, or deferred behind a running job; not sent in dry-run), capacity-not-ready, job-recovered and recovery-failed notifications, de-duplicated per node for SWARM_ALERT_REPEAT_S (default 1800).
  • Bootstrap logs: GET /api/v1/bootstrap/join-sessions/{id}/logs/stream.
  • Job metrics: GET /api/v1/jobs/{job_id}/metrics/stream.
  • Job loss history: GET /api/v1/jobs/{job_id}/metrics/history; the Ops UI uses this stored curve when a job is selected or the page is reloaded, and polls it as a fallback while live metrics are streaming.
  • Prometheus text metrics: GET /api/v1/metrics.

Transport modes

  • SWARM_PIPELINE_TRANSPORT=auto: use direct P2P when ports are reachable and leader relay as fallback.
  • SWARM_PIPELINE_TRANSPORT=tunnel: force the leader relay.
  • SWARM_PIPELINE_TRANSPORT=p2p: force direct fellow-to-fellow links.
  • Serving a model across two fellows follows the same policy: under auto (and p2p, mesh) the leader plans a direct link from stage 0 to stage 1's serving port (SWARM_SERVE_PORT on the fellow, default 5559) when both fellows declare serve-direct; stage 0 checks it can reach the port and falls back to the relay otherwise. tunnel keeps serving on the relay.
  • Direct links without open ports (mesh). docker compose ... up -d headscale leader frontend runs Headscale beside the leader; add a DNS record mesh.<SILENTSWARM_DOMAIN> pointing at the host (Traefik gets its certificate). The prod overlay sets SWARM_HEADSCALE_URL=https://mesh.<domain>. Once https://mesh.<domain>/health answers, fellows on the current version join on their own (userspace Tailscale, no root) and pipeline links between them run through the tunnel. SWARM_MESH_PORTS (default 5557-5600) are the ports the access policy opens between fellows of one pool. Headscale's data lives in $SILENTSWARM_DATA_DIR/headscale; its control socket in $SILENTSWARM_DATA_DIR/headscale-run (SILENTSWARM_DATA_DIR defaults to ./data/docker; production keeps it at /srv/silentswarm) (shared with the leader only). Switch it off on Admin > Direct links. Details: Mesh networking.
  • SWARM_SERVING_HISTORY_DAYS (default 7): how long every finished chat request is kept in the leader's state DB for the Inference page's longer Performance ranges (serving_requests table; older rows are deleted as new ones arrive).
  • SWARM_LEADER_HF_LOOKUP=0: don't look a training model's layer count up on the Hugging Face Hub (an air-gapped leader). Jobs then need model.num_layers for a weighted split and for truthful per-stage layer ranges; without it they split halfway. HF_ENDPOINT points the lookup at a mirror, HF_TOKEN authorizes gated repos.

Useful commands

swarm info
swarm start-cluster --fellows 2
swarm nodes
swarm fellows
swarm jobs
swarm watch <job_id>
swarm --leader-url http://localhost:3000 sweep --config exp/configs/exp_large_unlimited_compute_v1.json --partition weighted
swarm --leader-url http://localhost:3000 sweep --config exp/configs/exp_full_sweep_unlimited_v1.json
python exp/scripts/run_benchmark_suite.py --suite exp/configs/benchmarks/multinode_training_v1.json --leader-url http://localhost:3000 --dry-run
swarm update --fellow-id <id>
swarm release --fellow-id <id>
swarm stop-cluster

Benchmark operations

Run benchmark suites only when the cluster is otherwise idle, because each scenario measures the whole placement and network path. Start with --dry-run to verify that the connected fellows satisfy the infra profile's fellow count, GPU count, and minimum free VRAM requirements. The runner stores artifacts below exp/runs/benchmarks/{suite_run_id}/.

The single_gpu_baseline and two_gpu_one_node scenarios use the job field max_gpus_per_worker so dispatch limits the number of eligible GPU slots on the selected fellow. Two-node scenarios leave that field unset so all eligible GPUs on each selected fellow can participate. The constrained-network scenario records its target profile with enforcement=external; apply real inter-node shaping outside SilentSwarm before running it if you need hard bandwidth or latency limits.

Distributed benchmark jobs export trained shard weights when the job payload contains artifact_paths.model_export_dir. The export keeps the training split visible in model_export/run_manifest.json and writes one weights.npz per stage/fellow. Jobs can also set shard_push_interval_steps to make each fellow export and upload its latest shard to the leader every N training steps. The default benchmark stress profile enables this every 20 steps with shard_push_to_leader=true.

The leader stores pushed shards in the job's artifact_paths.model_export_dir when that path is present, or below .swarm/model_exports/{job_id}/model_export otherwise. Each upload refreshes model_export/run_manifest.json and appends a receipt to job.config.latest_shard_pushes. This does not yet perform automatic resume, but it gives the leader the latest stage-local weights needed for a future K8s replacement fellow to reload a shard.

Serving a trained run does not need a glue step: the leader assembles the uploaded per-stage shards into a servable model itself (servable.json under SWARM_SERVABLE_MODELS_ROOT, default $SWARM_DATA_DIR/servable_models), and it appears in GET /api/v1/models and on the Inference page. A model that shows as incomplete is missing a shard - POST /api/v1/models/{model_id}/push-export asks the fellows to re-upload whatever stages they still hold locally. Only the ready fellows of the model's pool are asked; a fellow_id outside that pool is refused with 409, and a model outside the caller's pools answers 404 (as POST /v1/chat/completions does for it).

  • scripts/
  • docker/
  • src/silent_swarm/bootstrap/
  • src/silent_swarm/leader/
  • src/silent_swarm/fellow/
  • tests/