Kubernetes (Helm)¶

The chart in deploy/helm/silentswarm/ installs the control plane: the leader, the nginx frontend that also serves these docs, and optionally the Headscale mesh server. GPU machines outside the cluster join the way they always do, with the one-line curl ... | bash command. That is the product's core case: ordinary machines, often behind NAT, connecting out. GPU nodes of the cluster itself can run fellows from the same chart (In-cluster fellows).

flowchart LR
    Internet --> Ingress[Ingress controller]
    Ingress -->|domain| FE[frontend Deployment<br/>nginx + docs :8080]
    Ingress -->|mesh.host| HS
    FE --> LS[leader Service :8080]
    subgraph Pod["leader StatefulSet, 1 pod"]
        L[leader :8080] -- socket --> HS[headscale :8081]
    end
    LS --> L
    L --> PVC[(data PVC<br/>ReadWriteOnce)]
    F[fellows, anywhere] -->|https| Ingress

Requirements: Kubernetes 1.28 or newer, an Ingress controller, and a block-backed ReadWriteOnce storage class (see Storage).

Install¶

helm install swarm deploy/helm/silentswarm \
  --namespace swarm --create-namespace \
  --set domain=swarm.example.org \
  --set mesh.host=mesh.swarm.example.org \
  --set admin.email=you@example.org \
  --set ingress.className=nginx \
  --set-string ingress.annotations."cert-manager\.io/cluster-issuer"=letsencrypt

Then follow what helm install prints:

DNS. Point domain (and mesh.host, if set) at the Ingress controller.
Admin password. With admin.email set, the leader pod creates that account on start, and the chart generates its password into the Secret <release>-silentswarm-admin (kept across upgrades). Read it with kubectl get secret ... -o jsonpath='{.data.password}' | base64 -d, sign in, and change it. Without admin.email, the first person to register becomes the admin, so register before you publish the address.
Fellows. The Training console shows the one-line join command; it works from any network that reaches https://<domain>.

helm test swarm -n swarm checks the install from inside the cluster: the leader reports the installed version, the docs carry this domain's canonical URL, and the admin can sign in.

The images come from ghcr.io/franzelfx/silentswarm-{leader,frontend} at the chart's appVersion (Releases and images). For private images, set imagePullSecrets.

Values¶

Value	Default	What
`domain`	(required)	public host of the UI, API and docs
`mesh.host`	empty	Headscale's host; the mesh server runs only when set. Off, every job uses the leader relay
`admin.email`, `admin.existingSecret`	empty	the first admin, and a Secret with its password (else generated)
`existingSecret` / `secrets`	empty	a Secret whose keys become leader environment: `SWARM_SMTP_PASSWORD`, `HF_TOKEN`, `SILENTSWARM_INDEXNOW_KEY`, `SWARM_ALERT_WEBHOOK`
`leader.env`	`{}`	any other `SWARM_*` setting, e.g. `SWARM_SMTP_HOST`, `SWARM_REQUIRE_ADMIN_APPROVAL`
`leader.resources`	512Mi request, 4Gi limit	the relay holds frames in memory - see below
`leader.relay.maxFrameMB`, `.queueMax`	128, 8	relay limits sized to the memory limit
`persistence.data.size`, `.storageClass`	20Gi	the leader's state
`persistence.models.enabled`, `.size`	off, 200Gi	a separate claim for servable models (GBs each)
`image.registry`, `.namespace`, `.tag`	ghcr.io, franzelfx, appVersion	where the images come from
`ingress.className`, `.annotations`, `.tls.*`	cluster default, none, on	TLS and host routing only
`frontend.replicas`, `.trustedProxies`	2, private ranges	nginx replicas; whose `X-Forwarded-For` it believes
`metrics.serviceMonitor.*`	off	a Prometheus Operator ServiceMonitor
`networkPolicy.enabled`, `.ingressFrom`, `.metricsFrom`	off	ingress policies
`fellows.enabled`, `.replicas`, `.gpusPerFellow`	off, 2, 1	in-cluster fellows
`fellows.idBase`, `.enrollmentSecret`	10000, `swarm-enrollment`	node IDs from `idBase` up; the Secret with their enrollment token
`fellows.terminationGracePeriodSeconds`, `.drainSeconds`	180, 150	how long a stopping fellow waits for its stage to move
`fellows.storage.size`, `.resources`, `.nodeSelector`, `.tolerations`, `.affinity`	200Gi, 16-32Gi, -, GPU taint, -	per pod
`fellows.mesh.enabled`	off	join the mesh too (only for jobs mixing in outside fellows)
`fellows.standby.replicas`, `.idBase`, `.enrollmentSecret`	0, 20000, `swarm-enrollment-standby`	a StatefulSet of standbys

The chart sets SWARM_LEADER_URL, SILENTSWARM_DOMAIN and SWARM_HEADSCALE_URL from these values. It keeps SWARM_TRUSTED_PROXY_HOPS=1: the frontend resolves the client past the Ingress controller and appends only that address (Client IP). The three enforcement flags (SWARM_REQUIRE_NODE_AUTH, ..._OPERATOR_AUTH, ..._CSRF) stay on.

Relay memory. A busy relay channel holds up to maxFrameMB × queueMax in the leader's memory. The defaults (128 MB × 8 = 1 GB) leave room for two or three busy channels under the 4Gi limit; a measured serving frame is 1.4 to 11 MB and a training boundary is batch × sequence × hidden × 2 bytes (42 MB for 1 × 4096 × 5120). Raise the limit and the relay values together: an OOM kill mid-job breaks every relay link at once.

Metrics. /api/v1/metrics needs an operator credential. Create a read-scoped API token in the UI (Profile, API tokens), store it in a Secret, and name it in metrics.serviceMonitor.tokenSecret.

What runs¶

The leader is a StatefulSet with exactly one replica. A StatefulSet never starts a second pod while the first might still run, even after a node loss; two processes on one SQLite file, or two sets of sweep loops, would corrupt state and double-count. More replicas are not supported: the fellows' connections, the relay and the job loops live in one process.
Headscale, when mesh.host is set, runs as a second container in the leader pod on port 8081 and shares only its control socket (an emptyDir) with the leader. It has its own small volume.
The frontend is a stateless Deployment of unprivileged nginx. It keeps the per-route timeouts and buffering (relay, SSE, MCP, chat streams), so the Ingress does TLS and host routing only and any controller works.

Every pod meets the restricted Pod Security Standard: non-root, no capabilities, RuntimeDefault seccomp. The leader and Headscale also run with a read-only root filesystem; all their writes go to their volumes and /tmp. Probes: liveness on /api/v1/health, readiness on /api/v1/ready (503 while starting or stopping, without a writable DB, or low on disk), and a startup probe that allows five minutes for schema migrations.

Storage¶

The leader keeps everything in one SQLite file plus files beside it, all under /data on the data claim. Use block-backed ReadWriteOnce storage: a cloud disk, Longhorn, local-path, Ceph RBD. NFS and other network file systems do not honour POSIX locks reliably, and SQLite on them corrupts under contention.

Backup is a copy of the claim. The simplest consistent one is taken with the leader stopped:

kubectl -n swarm scale statefulset swarm-silentswarm-leader --replicas=0
# snapshot the PVC data-swarm-silentswarm-leader-0 (VolumeSnapshot or your backup tool)
kubectl -n swarm scale statefulset swarm-silentswarm-leader --replicas=1

Without a stop, SQLite's online backup gives a consistent copy of the database (the image has Python but no sqlite3 binary):

kubectl -n swarm exec swarm-silentswarm-leader-0 -c leader -- python -c \
  "import sqlite3; sqlite3.connect('/data/join_state.sqlite3').backup(sqlite3.connect('/data/backup.sqlite3'))"

Then copy /data/backup.sqlite3 and the rest of /data (bucket files, models, exports) off the pod.

Upgrade¶

helm upgrade swarm deploy/helm/silentswarm -n swarm --reuse-values

Users, pools, buckets and models live on the claims and survive. A leader restart is not zero-downtime: relay links, chats in flight and MCP sessions drop for the few seconds it takes (8 s in the CI test), and fellows reconnect by themselves.

Do not upgrade while fellows upload shards. After a job COMPLETED, its fellows upload their trained shards for minutes. A restart in that window leaves the model incomplete (recover with POST /api/v1/models/{model_id}/push-export, or models_push_export over MCP). Check first:

kubectl -n swarm logs statefulset/swarm-silentswarm-leader -c leader --since=10m | grep -i shard

A rollout stuck on a broken pod. A StatefulSet does not replace a pod that never became ready - a known Kubernetes limitation. If an upgrade fixes such a pod (a wrong image, say), delete the old pod once: kubectl -n swarm delete pod swarm-silentswarm-leader-0.

Fellows¶

External fellows need nothing from the chart: they connect out to https://<domain> and, with the mesh on, to https://<mesh.host> - joined with the one-line command, or as the fellow image (Fellow in a container) with an enrollment token from the pool page kept in a Secret.

In-cluster fellows¶

With fellows.enabled the chart runs the fellow image on the cluster's GPU nodes, as a StatefulSet: each pod is one node of a pool, with gpusPerFellow cards (nvidia.com/gpu, so the NVIDIA device plugin must run) and its own volume. Kubernetes 1.28 or newer.

On the pool page, Add a node -> "An enrollment token" (active role, a node cap if you like). It prints the commands for the next two steps.

Store the token where the chart expects it, in the release's namespace:

kubectl -n swarm create secret generic swarm-enrollment \
  --from-literal=SWARM_JOIN_TOKEN=<token>

Turn the fellows on:

helm upgrade swarm deploy/helm/silentswarm -n swarm --reuse-values \
  --set fellows.enabled=true --set fellows.replicas=2

The pods show up under the pool's Nodes within a minute.

Aspect	How
Identity	node ID = `idBase` + the pod's ordinal (`swarm-silentswarm-fellow-0` is 10000), from the `apps.kubernetes.io/pod-index` label; the fellow adds the two (`SWARM_FELLOW_ID_BASE` + `SWARM_FELLOW_INDEX`). A rescheduled pod is the same node, with its volume and the node credential on it. Keep the range clear of your other fellows' IDs
Leader	the leader's Service in the namespace (`fellows.leaderUrl` to change it): registration, the signal socket, the relay and servable models fetched from the leader stay inside the cluster (Hugging Face weights still come from the Hub)
Peers	each pod advertises its pod IP; the leader's port probe answers, so a job of in-cluster fellows runs P2P. The mesh stays off (`fellows.mesh.enabled`): turned on, every pod joins it, which helps a job that also includes outside fellows but routes in-cluster traffic through the tunnel too. Without it, such a mixed job uses the relay
Storage	a `ReadWriteOnce` claim per pod at `/state`: the model and Hugging Face caches, and trained shards until their upload - possibly their only copy
Stopping	a pod deletion, eviction or node drain sends SIGTERM: a fellow holding a training stage checks out and waits `drainSeconds` for the leader to move the stage at its next snapshot, then exits. Size `drainSeconds` (and `terminationGracePeriodSeconds` above it) to your jobs' snapshot cadence; a stage not moved in time resumes from the last snapshot instead. A PodDisruptionBudget lets a drain take one fellow at a time
Upgrades	the image is the code (the leader's update skips these fellows). `helm upgrade` replaces the pods one by one, each draining first
Liveness	local (`health_check`): a leader restart never restarts the fellows
Security	non-root (uid 10001), read-only root filesystem, the "restricted" Pod Security Standard

Standbys. fellows.standby.replicas adds a second StatefulSet whose token (swarm-enrollment-standby) was minted with the standby role: warm nodes that covers and recovery move a lost node's stage to. A node is replaced automatically only with SWARM_AUTO_REPLACE=1 on the leader:

helm upgrade swarm deploy/helm/silentswarm -n swarm --reuse-values \
  --set fellows.standby.replicas=1 --set leader.env.SWARM_AUTO_REPLACE=1

NetworkPolicy. With networkPolicy.enabled, fellows reach the leader's API, and each other and the leader reach the fellows' ports 5557-5600.

Scaling. More replicas are more nodes: a running job keeps its split (it re-splits only through recovery), and the next job uses them. Fewer replicas remove the highest ordinals - each a node loss to a job it was part of - and the node stays listed, offline, until you remove it from the pool. Every node counts against the pool's node limit, and the token's cap.

Testing the chart with CPU fellows (gpusPerFellow: 0, an image built with --build-arg TORCH_INDEX_URL=https://download.pytorch.org/whl/cpu) works but trains slowly; scripts/helm_kind_fellows.sh does that on kind.