Skip to content

Operations

This section documents runtime operations for SilentSwarm clusters, especially leader/fellow startup, remote onboarding, and troubleshooting.

flowchart LR
  Start[Start leader] --> Fellows[Attach fellows]
  Fellows --> Inspect[Inspect nodes and devices]
  Inspect --> Run[Run jobs or sweeps]
  Run --> Observe[Watch metrics and logs]
  Observe --> Manage[Update, release, or purge fellows]
  Manage --> Stop[Stop cluster]

Current topics

  • Local and staging startup procedures.
  • Service health checks and observability.
  • Leader-served fellow onboarding through curl bootstrap.
  • Ops UI join flow and onboarding status.
  • Fellow release, update, and purge workflows.
  • Pipeline transport selection for tunnel and direct P2P modes.
  • Troubleshooting playbooks for bootstrap, leader connectivity, and port conflicts.

Documents

Local cluster lifecycle

swarm start-cluster --fellows 2
swarm nodes
swarm fellows
swarm stop-cluster

Local process logs are written under .swarm/runtime-logs/. Docker Compose users normally reach the leader through the nginx frontend at http://localhost:3000.

Docker and tunnel access

docker compose --profile local-fellow up -d --build
swarm --leader-url http://localhost:3000 nodes

For temporary public access to the Ops UI and bootstrap flow:

docker compose --profile local-fellow --profile tunnel up -d --build
docker compose --profile tunnel logs -f tunnel

The tunnel log prints a public /ops/join-v2 URL.

Upgrading an existing deployment

Two things changed in the leader image that a running deployment notices:

  • The leader runs as an unprivileged user (uid 10001). The entrypoint takes ownership of the /data volume on first start, so a normal docker compose up -d --build leader is enough; if you bind-mount extra paths into the container, chown -R 10001:10001 them yourself. Overriding user: in compose skips the entrypoint's chown entirely.
  • Dependencies come from requirements.lock (pinned + hash-verified). Change requirements.txt, then regenerate with scripts/lock_deps.sh and commit both - CI fails if the lock is stale.

The enforcement flags (SWARM_REQUIRE_NODE_AUTH, SWARM_REQUIRE_OPERATOR_AUTH, SWARM_REQUIRE_CSRF) are now on unless set to 0, so fellows must be running a build that sends their node credential on the signal WebSocket before you redeploy the leader - see Users and authentication.

Remote fellow onboarding

Use the Ops UI at /ops/join-v2 or the bootstrap API documented in Bootstrap contract. Join sessions can carry a token and stream logs back to the leader while the remote host installs and starts the fellow runner.

Runtime management

  • swarm update [--fellow-id ID]: ask connected fellows to re-bootstrap from the leader.
  • swarm release --fellow-id ID: remove a fellow from the leader registry.
  • swarm release --fellow-id ID --purge-install: release and request local cleanup.
  • swarm pending-purges and swarm retry-purge: inspect and retry cleanup for fellows that were offline during purge delivery.
  • swarm watch <job_id>: stream live merged metrics from the leader.

Observability

  • Ops UI: /ops/join-v2 is a read-only join and monitoring surface. Selecting a fellow in the graph or table opens its runtime logs automatically and shows status, network speed bars, step/loss, VRAM used/free, and per-GPU telemetry or active GPU layer placement. Selecting the leader shows cluster performance, including total RAM, CPU cores, aggregate VRAM, network throughput, observed step time, steps/minute, and the active placement plan.
  • Worker logs: GET /api/v1/workers/{fellow_id}/logs/stream; the Ops UI hydrates logs from the REST snapshot first and keeps polling as a tunnel-safe fallback.
  • Fellow progress logs: each training progress post is logged at info level with job ID, fellow ID, step, loss, status code, HTTP latency, request/response bytes, and cumulative fellow byte counters. Fellows also update their local step/loss immediately so heartbeat metrics and the UI do not wait for completion.
  • Computation placement: leader dispatch ranks connected fellows by aggregate free GPU VRAM across eligible GPUs on each node. GPUs with less than placement_min_gpu_free_gb free memory are ignored; the default is 4.0 GB and can be set globally with SWARM_PLACEMENT_MIN_GPU_FREE_GB or per job config. With --partition weighted, node layer counts are weighted by aggregate eligible free VRAM, and each node placement contains a gpu_slots plan that splits the node's layer range across eligible GPUs by that GPU's currently free VRAM. The fellow exposes its assigned GPUs through CUDA_VISIBLE_DEVICES and the torch stage spreads its transformer blocks across them weighted by each GPU's VRAM, so the layer split is an actual intra-node multi-GPU execution plan rather than UI metadata. The final pipeline stage also places final_norm, the output head, loss, and gradient workspace on the local slot with the most free VRAM; it can still need more memory than its layer count alone suggests.
  • Pipeline stage ordering: by default, SWARM_PIPELINE_LAST_STAGE_PLACEMENT=largest_vram moves the strongest selected fellow to the final pipeline stage, because that stage owns final_norm, the output head, loss, and gradient workspace. Set pipeline_last_stage_placement=preserve in a job config to keep the raw placement order for diagnostics.
  • GPU runtime check: a fellow that should train on NVIDIA GPUs needs the PyTorch runtime extras (pip install -e ".[runtime]", or requirements-runtime.txt - what the bootstrap installs). Hardware inventory and training capability are independent: a fellow reports GPUs from nvidia-smi/NVML even when the installed torch build has no CUDA support and every job silently runs on CPU. Confirm with python -c "import torch; print(torch.cuda.is_available())" on the host before interpreting a stress run.
  • GPU memory cleanup: training runs in a fresh torch subprocess (fellow/torch_dispatch.py), so a completed, failed, or cancelled job releases its CUDA context by exiting - an idle fellow does not sit on a model's VRAM. The worker pins MKL_THREADING_LAYER=GNU (torch links the GNU OpenMP layer; an Intel libiomp on the path aborts it). If a deployment is supervised by K8s, systemd, or another restart manager and wants the whole fellow recycled after each job, set SWARM_FELLOW_EXIT_AFTER_TRAINING=1 so it exits after posting the final job status and is restarted by the supervisor.
  • Live VRAM in the Ops UI comes from heartbeat hardware telemetry. Registration-time GPU inventory is only the initial snapshot; while fellows are running, each heartbeat refreshes per-GPU free VRAM so node bars and GPU placement rows show current used/total memory.
  • Pool telemetry: GET /api/v1/pools/{pool_id}/metrics (snapshot) and .../metrics/stream (SSE). One payload describes the pool as a system: serving throughput (tokens/s, TTFT, queue wait, error classes), training progress and straggler lag, per-node liveness (availability, control-plane RTT, heartbeat age), capacity, and which models are deployed on which nodes. Member-scoped: per-node telemetry never crosses a pool boundary. SWARM_POOL_METRICS_INTERVAL_S (default 5) sets the stream cadence.
  • Node health signals come from the fellow's heartbeat telemetry block: per-GPU utilization/temperature/power, the fellow's own leader round trip, peer round trips, and OOM/error/restart counters. The block is optional - a fellow that predates it heartbeats normally and its missing signals read as unknown, never as zero.
  • Control-plane RTT: the leader pings connected fellows over the signal WebSocket every SWARM_CONTROL_PING_INTERVAL_S seconds (default 15, 0 disables) and times the pong. This is latency measurement, not liveness - the heartbeat already covers liveness. Only fellows that advertised the ping capability at registration are pinged: an older build cannot parse an unknown control frame and drops its socket on one, so pinging everything would turn a latency probe into a reconnect storm. Nodes that predate the capability simply report an unknown control RTT until they are updated.
  • Metric history: GET /api/v1/pools/{pool_id}/metrics/history?metric=&window_s=&max_points= and the pool timeline at .../events. A roll-up runs every SWARM_METRICS_ROLLUP_INTERVAL_S seconds (default 60, 0 disables) writing minute-resolution rows; a daily retention pass folds rows older than 7 days into hourly means and drops anything past 90 days, so the tables stay bounded. Reads are always bucketed to at most max_points.
  • Node grades: each node scores 0-100 (bands A-F) from availability, reliability, latency, throughput and capacity, minus penalties for flapping, crash loops, straggling and OOM. Tune with SWARM_GRADE_WEIGHTS (JSON) and SWARM_GRADE_DWELL_S (default 120 - how long a band change must hold before it takes effect). Unreported components are dropped and the rest renormalised; a node with no behavioural evidence stays ungraded rather than being scored badly.
  • Node roles: every pool node is active (schedulable), standby (warm spare, never placed), draining, or quarantined. Mint a spare with POST /api/v1/bootstrap/join-sessions {"pool_id": ..., "role": "standby"}; change a role with POST /api/v1/pools/{pool_id}/nodes/{fellow_id}/role (pool owner or admin, audit-logged). Standbys count against the pool's node quota.
  • Automatic replacement: off by default. SWARM_AUTO_REPLACE=1 lets the leader quarantine a failing active node and promote the best-graded standby that has enough free VRAM for the stage - checkpoint, role swap, re-place, resume. Dwell times per trigger: SWARM_REPLACE_OFFLINE_S (60), SWARM_REPLACE_GRADE_S (120), SWARM_REPLACE_STRAGGLER_S (300), SWARM_REPLACE_OOM_S, SWARM_REPLACE_FLAP_S; SWARM_REPLACE_COOLDOWN_S (300) spaces replacements out. A node counts as straggling at 5+ steps behind the pool - pipeline stages legitimately run a step or two apart. Repeated identical decisions are recorded once, not on every tick. In dry-run the policy still records what it would have done on the pool timeline - review that before enabling it.
  • Serving VRAM budget: a model needs its weights plus the KV cache its layers allocate. The leader budgets both before loading (2 x layers x kv_heads x head_dim x dtype_bytes x context x concurrency, per stage) and refuses a placement that cannot hold them, with the numbers in the message. Tune with SWARM_SERVE_CONTEXT_TOKENS (default 4096), SWARM_SERVE_CONCURRENCY (default 1 - one generation per node today), and SWARM_SERVE_VRAM_SLACK (default 0.10). The attention shape comes from the servable manifest; a model exported before this existed records its shape the first time a fellow loads it, and until then is admitted on weights alone and says so.
  • Alerts: SWARM_ALERT_EMAILS (comma-separated) and/or SWARM_ALERT_WEBHOOK receive node-quarantined, node-promoted and capacity-not-ready notifications, de-duplicated per node for SWARM_ALERT_REPEAT_S (default 1800).
  • Bootstrap logs: GET /api/v1/bootstrap/join-sessions/{id}/logs/stream.
  • Job metrics: GET /api/v1/jobs/{job_id}/metrics/stream.
  • Job loss history: GET /api/v1/jobs/{job_id}/metrics/history; the Ops UI uses this stored curve when a job is selected or the page is reloaded, and polls it as a fallback while live metrics are streaming.
  • Prometheus text metrics: GET /api/v1/metrics.

Transport modes

  • SWARM_PIPELINE_TRANSPORT=auto: use direct P2P when ports are reachable and leader relay as fallback.
  • SWARM_PIPELINE_TRANSPORT=tunnel: force the leader relay.
  • SWARM_PIPELINE_TRANSPORT=p2p: force direct fellow-to-fellow links.

Useful commands

swarm info
swarm start-cluster --fellows 2
swarm nodes
swarm fellows
swarm jobs
swarm watch <job_id>
swarm --leader-url http://localhost:3000 sweep --config exp/configs/exp_large_unlimited_compute_v1.json --partition weighted
swarm --leader-url http://localhost:3000 sweep --config exp/configs/exp_full_sweep_unlimited_v1.json
python exp/scripts/run_benchmark_suite.py --suite exp/configs/benchmarks/multinode_training_v1.json --leader-url http://localhost:3000 --dry-run
swarm update --fellow-id <id>
swarm release --fellow-id <id>
swarm stop-cluster

Benchmark operations

Run benchmark suites only when the cluster is otherwise idle, because each scenario measures the whole placement and network path. Start with --dry-run to verify that the connected fellows satisfy the infra profile's fellow count, GPU count, and minimum free VRAM requirements. The runner stores artifacts below exp/runs/benchmarks/{suite_run_id}/.

The single_gpu_baseline and two_gpu_one_node scenarios use the job field max_gpus_per_worker so dispatch limits the number of eligible GPU slots on the selected fellow. Two-node scenarios leave that field unset so all eligible GPUs on each selected fellow can participate. The constrained-network scenario records its target profile with enforcement=external; apply real inter-node shaping outside SilentSwarm before running it if you need hard bandwidth or latency limits.

Distributed benchmark jobs export trained shard weights when the job payload contains artifact_paths.model_export_dir. The export keeps the training split visible in model_export/run_manifest.json and writes one weights.npz per stage/fellow. Jobs can also set shard_push_interval_steps to make each fellow export and upload its latest shard to the leader every N training steps. The default benchmark stress profile enables this every 20 steps with shard_push_to_leader=true.

The leader stores pushed shards in the job's artifact_paths.model_export_dir when that path is present, or below .swarm/model_exports/{job_id}/model_export otherwise. Each upload refreshes model_export/run_manifest.json and appends a receipt to job.config.latest_shard_pushes. This does not yet perform automatic resume, but it gives the leader the latest stage-local weights needed for a future K8s replacement fellow to reload a shard.

Serving a trained run does not need a glue step: the leader assembles the uploaded per-stage shards into a servable model itself (servable.json under SWARM_SERVABLE_MODELS_ROOT, default .swarm/servable_models), and it appears in GET /api/v1/models and on the Inference page. A model that shows as incomplete is missing a shard - POST /api/v1/models/{model_id}/push-export asks the fellows to re-upload whatever stages they still hold locally.

  • scripts/
  • docker/
  • src/silent_swarm/bootstrap/
  • src/silent_swarm/leader/
  • src/silent_swarm/fellow/
  • tests/