Fellow Bootstrap contract

This page documents the full bootstrap contract for leader-managed fellow onboarding.

Current capabilities

Leader-managed onboarding provides:

• a leader-served bootstrap manifest (manifest.json)
• a leader-served shell bootstrap script (fellow.sh)
• a leader-served source archive (source.tar.gz)
• join sessions with optional token authentication
• per-session log collection and SSE streaming
• a web UI, the Training console at /training, for creating join commands and monitoring connected fellows
• live swarm graph with role lanes, stabilized interpolation, and per-fellow status in the Training console
• non-interactive install and start flow with retry logic
• macOS-tolerant dependency installation that lets a fellow join even when PyTorch wheels are unavailable for the local Python version
• ZMQ port overrides and idempotent re-runs
• automatic fallback to the next free ZMQ port pair in non-interactive mode
• SQLite-backed persistence for join sessions, bootstrap logs, and worker registry state across leader restarts

flowchart TD
  Operator[Operator or automation] --> Session[Create join session]
  Session --> Token[Receive session ID and token]
  Token --> Curl[Run leader-served curl command]
  Curl --> Manifest[Fetch manifest and bootstrap script]
  Manifest --> Archive[Download source archive]
  Archive --> Install[Create venv and install package]
  Install --> Start[Start fellow runner]
  Start --> Register[Register with leader]
  Register --> Ready[Fellow appears ready in Ops UI]
  Curl -. session logs .-> Logs[Bootstrap log stream]
  Logs -. SSE .-> Operator

Endpoints

Bootstrap resources

Endpoint	Description
GET /api/v1/bootstrap/manifest.json	JSON description of the bootstrap contract, incl. source_archive_sha256 and script_sha256
GET /api/v1/bootstrap/fellow.sh	Shell script that installs and starts a fellow
GET /api/v1/bootstrap/fellow.sh.sha256	Published sha256 of fellow.sh (sha256sum -c format) for read-before-run verification
GET /api/v1/bootstrap/source.tar.gz	Deterministic gzip archive of the project source

When SWARM_BOOTSTRAP_REQUIRE_TOKEN=1 is set, all bootstrap endpoints require a valid token — passed either as an X-Bootstrap-Token header or a ?token= query param (the header form is preferred; it stays out of leader access logs). This is the recommended setting for beta so the curl … | bash surface cannot be fetched anonymously — only holders of a live pool/invite token (join session) can onboard.

Security hardening

The curl … | bash bootstrap runs as a high-trust surface — often as root on a GPU box — so it is defended at several layers:

• HTTPS only. The script refuses a plain-http:// leader URL (except loopback) unless you pass --allow-http explicitly. Serve the leader behind TLS; see deployment.
• Source archive is checksum-verified. source.tar.gz is built deterministically, and the manifest publishes its source_archive_sha256. The script verifies the downloaded archive against that hash with sha256sum before extracting or executing any of it — a mismatch aborts the install.
• Invite-token gating (beta). With SWARM_BOOTSTRAP_REQUIRE_TOKEN=1, the script and archive can't be fetched without a valid pool/invite token, so onboarding is never anonymous.
• No imports from shared temp directories. The script works from its own private mktemp -d directory, never from /tmp itself, and runs its inline Python helpers with python3 -I. Python puts the working directory first on sys.path, so a stray or planted /tmp/enum.py would otherwise replace the standard library inside the installer.
• Read-before-run. You never have to pipe an unread script to a shell — see below.

Read before you run

Instead of curl … | bash, split fetch and run so you can inspect the script and confirm its checksum first (all over HTTPS):

LEADER=https://your-leader
TOKEN=<pool/invite token from the Ops UI>   # omit if the leader isn't token-gated

# 1. Fetch the script and the leader's published checksum for it
curl -fsSL -H "X-Bootstrap-Token: $TOKEN" "$LEADER/api/v1/bootstrap/fellow.sh" -o fellow.sh
curl -fsSL -H "X-Bootstrap-Token: $TOKEN" "$LEADER/api/v1/bootstrap/fellow.sh.sha256" -o fellow.sh.sha256

# 2. Verify the script matches the published checksum
sha256sum -c fellow.sh.sha256        # must print: fellow.sh: OK

# 3. Read it, then run it
less fellow.sh
bash fellow.sh --leader-url "$LEADER" --token "$TOKEN"

The script then downloads source.tar.gz and verifies it against the manifest's source_archive_sha256 automatically, so the whole chain from script to source is checksum-verified end to end.

Join sessions

Endpoint	Description
POST /api/v1/bootstrap/join-sessions	Create a join session with a token and expiry (the only response that carries the token)
GET /api/v1/bootstrap/join-sessions	List the sessions of your pools (no tokens)
GET /api/v1/bootstrap/join-sessions/{id}	Get a session by ID (no token)
PATCH /api/v1/bootstrap/join-sessions/{id}	Update session status or host
DELETE /api/v1/bootstrap/join-sessions/{id}	Revoke a session

Bootstrap logs

Endpoint	Description
POST /api/v1/bootstrap/join-sessions/{id}/logs	Append a log line
GET /api/v1/bootstrap/join-sessions/{id}/logs	Retrieve buffered log lines
GET /api/v1/bootstrap/join-sessions/{id}/logs/stream	SSE stream of log lines

Operations UI

Endpoint	Description
GET /pools/{owner}/{pool}/training	Web UI (/training opens the active pool's) that generates join commands and shows the swarm graph, training panel, logs, and connected fellows table
GET /api/v1/ops/join-graph	Snapshot payload for leader + worker graph rendering
GET /api/v1/ops/join-graph/stream	SSE stream of graph snapshot updates (1 s interval)

The Training console is organized into three full-width vertical sections to use wide screens efficiently:

• Add Fellow
• Swarm Graph
• Connected Fellows

Graph readability model

The Training console's graph uses a lane-based layout to keep topology readable as the swarm grows:

• leader is fixed on the left as a stable anchor
• fellows are arranged on the right in a radial lane around a fixed center
• node positions are interpolated between snapshots to avoid abrupt jumps

Node semantics:

• labels show compact identity (#id + hostname) directly on the node
• hostnames prefer the registered fellow host when the raw hardware hostname is only a container ID
• full details (version, GPUs/VRAM, throughput, last seen) appear in tooltip and side panel
• worker nodes use larger symbols and a custom hover card so state stays legible on wide operator screens
• graph node details are refreshed from the same live worker inventory as the fellows table so status and VRAM stay aligned
• status is encoded by both color and shape/ring
• ready: circle with calm ring
• training: rounded shape with pulsing highlight
• offline: gray triangle (stale heartbeat)

Edge semantics:

• default links are thin and subdued
• links to actively training fellows are brighter and pulse subtly
• leader-to-fellow links represent tunnel-capable control/data relay paths
• fellow-to-fellow links are rendered only when direct P2P connectivity is available

Pipeline transport selection:

• SWARM_PIPELINE_TRANSPORT=auto (default): tunnel fallback, auto-upgrade to direct P2P when fellow ports are reachable
• SWARM_PIPELINE_TRANSPORT=tunnel: force leader relay tunnel
• SWARM_PIPELINE_TRANSPORT=p2p: force direct fellow-to-fellow links

The graph header includes a small legend plus live counters for connected, training, and offline fellows.

flowchart LR
  subgraph Auto[SWARM_PIPELINE_TRANSPORT=auto]
    A[Fellow A] --> Probe{Peer ports reachable?}
    Probe -- yes --> P2P[Direct ZMQ P2P]
    Probe -- no --> Relay[Leader HTTP relay]
    Relay --> B[Fellow B]
    P2P --> B
  end

Sidepanel actions

The graph sidepanel includes three actions per fellow:

• Logs: opens bootstrap session logs via the existing join-session log stream
• Restart: prepares a fresh join session and restart command for the selected fellow ID
• Update: sends the existing POST /api/v1/workers/update signal to the selected fellow

Action feedback is rendered in a dedicated result area in the sidepanel and is kept stable across graph refreshes.

The Connected Fellows table uses the same action-button visual language (icon, color coding, and token-based styling) for logs, restart, and update actions.

Join UI state persistence

The leader persists join sessions, bootstrap log entries, and worker registry entries to SQLite, so the join UI and worker inventory can recover their state after a restart.

• default DB path: $SWARM_DATA_DIR/join_state.sqlite3 (/data in the leader image, .swarm otherwise)
• override via env var: SWARM_JOIN_STATE_DB=/path/to/join_state.sqlite3

Persisted worker fields include fellow ID, host, ZMQ ports, hardware inventory, last known heartbeat/status, and current throughput counters. After a leader restart, workers remain visible immediately and subsequent heartbeats continue to update the restored entries.

Join commands

Unsecured (local/dev)

curl -fsSL http://<leader>:8080/api/v1/bootstrap/fellow.sh | bash -s -- \
  --leader-url http://<leader>:8080 --dry-run

Remove --dry-run for a real install.

With a join session (recommended)

1. Create a session from the Training console (/training) or via API:

curl -sX POST http://<leader>:8080/api/v1/bootstrap/join-sessions \
  -H "Content-Type: application/json" \
  -d '{"ttl_seconds": 3600}' | python3 -m json.tool

1. Use the returned session_id and token:

curl -fsSL http://<leader>:8080/api/v1/bootstrap/fellow.sh | bash -s -- \
  --leader-url http://<leader>:8080 \
  --session-id <SESSION_ID> \
  --token <TOKEN>

Token-Gated mode

Set SWARM_BOOTSTRAP_REQUIRE_TOKEN=1 on the leader to require a valid token for all bootstrap requests. Without the env var the leader operates in local/dev mode (no token required).

Bootstrap script flags

Flag	Default	Description
--leader-url URL	(from script)	Leader base URL
--fellow-id ID	auto (hostname CRC32)	Explicit fellow ID
--install-dir DIR	$HOME/.silent-swarm	Installation root
--session-id ID	—	Join session ID for log callbacks
--token TOKEN	—	Session token for authenticated bootstrap
--zmq-act-port PORT	5557	ZMQ activation port
--zmq-grad-port PORT	5558	ZMQ gradient port
--no-autostart	off	Do not start the fellow when the machine boots (kept across updates)
--dry-run	off	Print steps without installing
--help	—	Show help

Serving port. A fellow that holds the second stage of a model served across two fellows also listens on SWARM_SERVE_PORT (default 5559, set in the fellow's environment) for a direct link from the first stage. It is only used when the first stage's fellow can reach it - same LAN or mesh, port open between the two - and the leader relay carries the model otherwise. Open 5559/tcp between your fellows (not to the internet) to let served models skip the leader - or open nothing and switch on Direct links (Admin > Direct links): fellows then reach each other through an encrypted tunnel they set up themselves, without root. SWARM_PIPELINE_TRANSPORT=tunnel on the leader turns direct links off.

Script behavior

The script performs these steps on the remote host:

1. Fetch the bootstrap manifest from the leader.
2. Download the source archive from the leader.
3. Extract the archive into <install-dir>/app.
4. Create or reuse a Python virtual environment at <install-dir>/.venv.
5. Install dependencies via the virtualenv interpreter (python -m pip). Core dependencies are required; the PyTorch runtime dependencies are best-effort on macOS. Core dependency installs retry up to 3 times.
6. Start python -m silent_swarm.fellow.runner in the background, in a session of its own.
7. Make the fellow start again when the machine boots (see below), unless --no-autostart.

Additional behaviors:

• Writes the process PID to <install-dir>/fellow.pid. A re-run stops the running process (SIGTERM, then SIGKILL) and starts a fresh one.

• Leaves one start at boot, the first of these that works on the host:

  • root with systemd - a system unit, /etc/systemd/system/silentswarm-fellow.service;
  • a user with systemd - a user unit in ~/.config/systemd/user/, when the user lingers (otherwise systemd starts a user's units at login, not at boot). The bootstrap asks for linger itself - loginctl enable-linger, then sudo -n - and never prompts for a password;
  • cron - an @reboot line in the user's crontab, marked with the install directory.

  Each runs fellow restart. The unit is a oneshot that stays active (RemainAfterExit=yes): it starts the fellow and does not supervise it, so a swarm update - which stops the old process by fellow.pid and starts a new one - keeps working, and systemctl stop silentswarm-fellow stops the node. Installing never starts the unit. Another install directory gets a unit named after it (silentswarm-fellow-<hash>.service). The choice is kept in fellow.env as AUTOSTART, so an update keeps a fellow that was turned off, off; fellow autostart on|off|status changes or shows it. - A fellow.pid left from before a reboot is only acted on when it still names a fellow process: the number may belong to another process by then. - Writes runtime logs to <install-dir>/logs/fellow.out. - Writes runtime config to <install-dir>/fellow.env. - Installs a local helper command fellow (symlink at ~/.local/bin/fellow) for follow-up operations on the host. - Posts each log line back to the leader via the session log endpoint when --session-id is provided. - Updates the session status to completed or failed on exit. - The --token is passed as X-Bootstrap-Token header in all curl calls to the leader. - The shell wrapper prefers python3.13, python3.12, or python3.11 when present and falls back to any Python 3.11+ interpreter. - If the requested ZMQ ports are already occupied, the fellow runner automatically switches to the next free incremental pair in non-interactive bootstrap mode. - If python3 -m venv is unavailable, or it creates an environment without pip, the script tries ensurepip, then falls back to python3 -m virtualenv, and if needed tries python3 -m pip install --user virtualenv before failing.

Fellow local CLI

After bootstrap, the host gets a local fellow command with a minimal operational UI:

• fellow update – fetch latest bootstrap script from leader and re-install.
• fellow jobs – show recent job-related events parsed from local fellow logs.
• fellow logs --attach – attach/follow fellow.out (Ctrl+C to detach).
• fellow checkout --in 15m – take the machine back: its running training stage moves to another node at the next saved step before then, nothing new is placed on it, and its grade holds. fellow checkin gives it back.
• fellow stop / fellow restart – stop the node process, or start it again from fellow.env.
• fellow autostart [on|off|status] – whether the node starts when the machine boots (on after the bootstrap unless --no-autostart). status names the systemd unit or crontab line in use; on installs one, off removes it.
• fellow purge – leave for good: stops the node and deletes its start at boot, the install directory, the fellow command on the PATH and the ufw rules the bootstrap recorded (--keep-firewall leaves those). Asks first unless --yes. The Hugging Face model cache stays. The host's side of Day 1 for hosts.

fellow update is interactive by default and asks:

Would you like to update <current_version> -> <leader_version> [y/N]

Use fellow update --yes for non-interactive execution.

Join session lifecycle

stateDiagram-v2
  [*] --> pending: session created
  pending --> active: first log line
  active --> completed: script exits 0
  active --> failed: script exits non-zero
  pending --> revoked: operator DELETE
  active --> revoked: operator DELETE
  pending --> expired: TTL reached
  active --> expired: TTL reached
  completed --> [*]
  failed --> [*]
  revoked --> [*]
  expired --> [*]

Restarts and enrollment tokens

A join token admits a machine for its TTL (one hour by default). After that a fellow still comes back as the same node: it keeps the node credential the leader issued at its first registration (~/.silent-swarm/node_token, or node_token under SWARM_FELLOW_STATE_DIR; SWARM_FELLOW_NODE_TOKEN_FILE overrides it, mode 0600) and presents it on every registration. A valid credential keeps the node in the pool it was issued for, even when the join token in its environment has expired or been revoked. Releasing a node (DELETE /api/v1/workers/{id}) revokes its credential.

An enrollment token is a join session for containers and Kubernetes: POST /api/v1/bootstrap/join-sessions with "enrollment": true (and optionally "max_nodes"). It has no expiry (expires_at is null, the TTL is ignored), admits every node started with it into its pool with its role, up to max_nodes distinct nodes (a node that joined before always comes back; one more answers 409 enrollment-full), and lists the nodes that joined in enrolled. The pool page creates one under Add a node and lists them under Nodes, where they are revoked; over MCP it is enrollment_tokens_create. Revoking one stops new nodes; nodes already in keep their credential and stay until they are released. A pod that lost its state volume (and with it its credential) cannot come back with a revoked token.

Remote update via swarm update

The swarm update command pushes an UPDATE control signal to one or more connected fellows, instructing them to re-run the bootstrap install script from the leader.

Usage

# Update all connected fellows
swarm update

# Update a specific fellow
swarm update --fellow-id 7001

# Update multiple fellows
swarm update --fellow-id 7001 --fellow-id 7002

How it works

1. The CLI posts to POST /api/v1/workers/update on the leader.
2. The leader broadcasts (or targets) a {"signal": "UPDATE", "payload": {"leader_url": "..."}} frame over the fellow's control WebSocket.
3. On receiving the signal, the fellow re-runs the bootstrap script:

   curl -fsSL <leader_url>/api/v1/bootstrap/fellow.sh | bash -s -- --fellow-id <ID>
   

4. The bootstrap script replaces the installed source and restarts the runner process. The fellow's local state survives: it runs from <install dir>/app, so what it writes under app/.swarm/ - a trained model's exported shards waiting to be uploaded, offload folders - is moved aside (<install dir>/.swarm-keep, a rename on the same filesystem) while the source is replaced, and put back. Updates used to delete it, and with it the only copy of a stage whose upload had failed.
5. Like any bootstrap run it sets up the start at boot, unless fellow.env says AUTOSTART=0 - so an update is also how fellows installed before it existed get one.

The signal is only processed when the fellow is in IDLE or PAUSED state. Fellows in TRAINING ignore the signal with a warning in their logs.

Fellows that run from the container image are not updated this way: their code is the image, so the leader skips them and they upgrade by pulling a new image (Fellow in a container).

Response shape

{"sent": [7001, 7002], "not_connected": [7003], "image_managed": [7010]}

sent — fellows that had an active WS connection and received the signal. not_connected — registered fellows that were not connected via WS at the time (they get the update at their next heartbeat). image_managed — fellows skipped because their container image is their code.

Security notes

• In local/dev mode (SWARM_BOOTSTRAP_REQUIRE_TOKEN unset) the bootstrap endpoints are openly accessible.
• Set SWARM_BOOTSTRAP_REQUIRE_TOKEN=1 to require valid tokens in staging or shared environments.
• Tokens are random 32-byte URL-safe strings generated with secrets.token_urlsafe.
• Sessions expire automatically after the configured TTL (default 1 hour).
• Expired and revoked tokens are rejected immediately.
• Log endpoints and the PATCH endpoint do not require authentication (the caller needs to know the session ID).

Troubleshooting

python3 not found

Install Python 3.11+ via your OS package manager before running the script.

partially initialized module 'enum' (or another stdlib name)

A traceback that ends in a file under /tmp (for example File "/tmp/enum.py") means a leader older than this fix served the script: it ran its Python helpers from /tmp, where a file named like a standard-library module shadowed the real one. Redeploy the leader, or work around it on the host by pointing TMPDIR at a private directory:

curl -fsSL -H "X-Bootstrap-Token: <token>" https://<leader>/api/v1/bootstrap/fellow.sh \
  | TMPDIR="$(mktemp -d)" bash -s -- --leader-url https://<leader> --token <token>

Also check who put the file there (ls -l /tmp/*.py): on a shared host, a module planted in /tmp is a way to run code as whoever runs the installer.

pip install fails

The script retries pip install up to 3 times with a 5-second delay. If it still fails, check network connectivity and PyPI reachability from the target host. Common causes: missing build tools (gcc, make) for compiled dependencies.

On macOS, the PyTorch runtime install is best-effort. If PyPI has no matching torch wheel for the selected Python version, bootstrap logs a warning and continues so the fellow can register with the leader. Training jobs on that host still need torch; install a Python version with a matching wheel and rerun the bootstrap command or fellow update --yes.

Virtual environment creation fails with ensurepip is not available

The bootstrap script first tries python3 -m venv. If that creates an environment without pip, it tries python -m ensurepip --upgrade, then falls back to python3 -m virtualenv, and finally tries to install virtualenv with python3 -m pip --user.

If you see No module named pip on the target host, that means the initial venv was created without bundled pip and all recovery paths failed.

If all fallbacks fail, install one of the OS packages below and rerun the bootstrap command:

• Debian/Ubuntu: sudo apt install python3-venv
• Alternative package on some distros: python3-virtualenv

Fellow process exits immediately

Check <install-dir>/logs/fellow.out for the error. Common causes: incorrect SWARM_LEADER_URL, leader not reachable, or ZMQ port conflict.

If the conflict is only on the default ports 5557/5558, the bootstrap path now auto-falls back to the next free pair, for example 5561/5562. You can still force explicit values with --zmq-act-port and --zmq-grad-port.

Node stays offline after its machine rebooted

Every fellow says at start, in the log the leader shows for it, how it starts at boot (Start at boot: user (...)); Start at boot: none is a warning. On the machine, fellow autostart status says the same. nothing means the node was installed before the start at boot existed, or without one: fellow restart brings it back now and fellow autostart on makes it come back by itself. A user unit whose note says the user does not linger starts only when someone logs in: sudo loginctl enable-linger <user>. With cron, the start's own output is in <install-dir>/logs/autostart.log.

Fellow reconnects after a leader restart

If the leader restarts and drops the signal WebSocket (for example close code 1012 service restart), the fellow now automatically reconnects with exponential backoff and re-registers on the leader.

Script hangs at step 2

The source archive is built on demand. On large repos this can take a few seconds. Ensure the leader host has sufficient disk space for a temporary in-memory gzip.

Session shows expired before bootstrap finishes

Increase ttl_seconds when creating the session. The default is 3600 s (1 hour).

Manifest shape

{
  "format_version": "bootstrap-manifest-v1",
  "leader_url": "http://leader:8080",
  "leader_version": "0.1.0",
  "ready": true,
  "registered_workers": 0,
  "bootstrap_script_url": "http://leader:8080/api/v1/bootstrap/fellow.sh",
  "recommended_join_command": "curl -fsSL ... | bash -s -- --leader-url ...",
  "install": {
    "strategy": "leader_source_archive",
    "source_archive_url": "http://leader:8080/api/v1/bootstrap/source.tar.gz",
    ...
  },
  "script": { "endpoint": "...", "supported_flags": [...] }
}