Fellow in a container¶

The one-line curl ... | bash join builds a fellow in a virtualenv on the machine itself. The fellow image is the other way to run one: everything the fellow needs (Python, PyTorch with its CUDA libraries, the pinned runtime libraries, the mesh tunnel's binaries) is in the image, and the machine needs only Docker, an NVIDIA driver and the NVIDIA Container Toolkit.

Run¶

docker run -d --name fellow --gpus all --restart unless-stopped \
  --stop-timeout 120 \
  -e SWARM_LEADER_URL=https://swarm.example.org \
  -e SWARM_JOIN_TOKEN=<from the pool's join command> \
  -e SWARM_FELLOW_ID=7001 \
  -e SWARM_HOST=<an address other fellows can reach> \
  -e SWARM_FELLOW_DRAIN_S=100 \
  -p 5557:5557 -p 5558:5558 -p 5559:5559 \
  -v fellow-state:/state \
  ghcr.io/franzelfx/silentswarm-fellow:<version>

SWARM_FELLOW_ID is the node's identity. Keep it the same across restarts and recreates: a new ID is a new node, and enough of them fill the pool's node limit. The bootstrap derives one from the host name; the image takes what you give it, or SWARM_FELLOW_ID_BASE plus SWARM_FELLOW_INDEX (a StatefulSet's pod ordinal, which is how the Helm chart numbers its in-cluster fellows).
SWARM_JOIN_TOKEN binds the node to its pool. Use an enrollment token (the pool page's Add a node, "An enrollment token for containers and Kubernetes"): it does not expire and admits every container started with it, up to an optional number of nodes, until you revoke it. A one-hour join command's token works too, for the first start; after that the fellow comes back with the node credential it keeps in /state/node_token.
Ports. The fellow advertises the ports it binds, so publish them unchanged. A taken port is an error in the image (the bootstrap moves to the next free pair instead): another port would be one nothing routes to. Without open ports the leader's relay carries the traffic, and with the mesh on the fellow joins it on its own (SWARM_FELLOW_MESH=0 declines it).
GPUs. --gpus all, or --gpus '"device=0,1"' for some. The image sets CUDA_DEVICE_ORDER=PCI_BUS_ID so GPU indices match nvidia-smi, and NVIDIA_DRIVER_CAPABILITIES=compute,utility so the fellow can read the cards' memory and temperature.

State¶

The fellow runs as uid 10001 and writes only to /state (SWARM_FELLOW_STATE_DIR):

Under `/state`	What
`models/`	servable model cache
`hf/`	Hugging Face cache (`HF_HOME`, unless you set it)
`servable_exports/`	a training run's exported shards - possibly the only copy until the upload to the leader finishes
`snapshots/`, `offload/`	stage snapshots, disk offload for a stage too big for its cards
`control/`	`fellow checkout` requests, the liveness file
`mesh/`	the mesh tunnel's node state
`node_token`	the node credential, so a restart re-registers as the same node of the same pool
`logs/fellow.out`	the log the leader's log view shows

Use a named volume (as above), or a host directory owned by uid 10001. Each path still takes its own variable (SWARM_FELLOW_MODEL_CACHE, SWARM_FELLOW_SERVABLE_EXPORT_ROOT, ...) when you want it elsewhere.

Stopping¶

docker stop (and a Kubernetes pod deletion) sends SIGTERM. A fellow holding a training stage then checks itself out: the leader moves the stage to another node at its next snapshot boundary, and the fellow waits up to SWARM_FELLOW_DRAIN_S seconds (default 20) for that before it stops its worker and exits 0. A stage not moved in time is the job's recovery to handle, from its last snapshot. Give the container longer than the drain: --stop-timeout (Docker's default is 10 s) or terminationGracePeriodSeconds (30 s on Kubernetes). A serving stage needs no drain: the leader's serving failover replaces it.

The checkout keeps the node out of new work; the next start checks it back in.

fellow checkout works the same in a container:

docker exec fellow fellow checkout --in 15m
docker exec fellow fellow checkin

Health and logs¶

The image's health check, and the probe to use as a Kubernetes livenessProbe, is python -m silent_swarm.fellow.health_check. It is local: the runner touches /state/control/alive while it runs, and the check fails when that is older than SWARM_FELLOW_ALIVE_MAX_S (60 s). It does not ask the leader, so a leader restart never restarts the fellows and kills their work. health_check --leader checks leader reachability instead, for diagnosis.

Logs go to stdout and stderr (docker logs fellow) and to /state/logs/fellow.out, which the fellow ships to the leader's log view.

Upgrades¶

The image is the code. swarm update and the leader's update skip image-managed fellows (image_managed in the response, "(image)" beside the version on the Training page), and a fellow refuses an UPDATE that reaches it anyway - the bootstrap would start a second fellow beside it. Upgrade by pulling the next version and recreating the container with the same volume and SWARM_FELLOW_ID.

The image pins its runtime libraries (torch, transformers, accelerate, peft, bitsandbytes, ...) in docker/fellow-constraints.txt; a bootstrap fellow takes the newest its requirements allow. A pool can mix both: the leader only uses a feature every fellow in a job declares, which keeps the wire format compatible, but two transformers versions need not compute identical numbers.