Fellow isolation¶

What a fellow runs can reach what its user can reach. This page says what stands between the two, what does not, and how to turn on the parts that are off by default.

Three things can go wrong, and they have different answers:

Who	How	What helps
The coordinator	Its `UPDATE` signal makes a fellow run a script it serves	Update policy
A model or dataset	A crafted file exploits a parser in the worker	Worker limits
A machine in the pool	It reads or changes the work it was given	Not this page: a host can always read what it runs. Sealed links and audits narrow it, and nothing makes it impossible on these cards

If the people running the machines and the coordinator all trust each other, you need none of this.

What is always true¶

These hold on every fellow, with nothing to configure:

A job runs no code of its own. trust_remote_code is always off, tensors cross the wire in a format with no pickle, and a checkpoint from another machine is read with weights_only=True, which a fellow refuses to do on a torch older than 2.6 (CVE-2025-32434).
A worker has no node token. A bucket dataset is downloaded by the fellow. A relay link goes through a proxy on the fellow's loopback, which holds the token and passes on relay calls and nothing else. SWARM_FELLOW_RELAY_PROXY=0 goes back to handing the worker the token, on a fellow where the worker is not limited.
A worker has no signing key. A stage that seals its links reads node_link_key, a file with the link key alone. The key that signs this node's work receipts stays in node_key.
A worker's environment is an allow list. Cloud keys, TOKEN, SWARM_JOIN_TOKEN and anything else named like a secret stay out. A Hub token passes unless the worker is limited.

On an unlimited worker these are depth, not a boundary: it runs as you, so it can still read ~/.silent-swarm/node_token from disk. The next section is what closes that.

Update policy¶

SWARM_FELLOW_UPDATES, set with --updates when joining or in fellow.env:

Value	A coordinator's `UPDATE`
`auto` (default)	Runs the script the coordinator serves. Whoever controls the coordinator controls this machine.
`off`	Is refused. You update with `fellow update`.
`signed`	Installs only what a key you pinned signed, running the installer inside the verified archive. The coordinator's script is never fetched.

A value that cannot be read means off, and signed with no usable key refuses everything. Both settings outlive an update. A fellow from a container image always refuses UPDATE.

UPDATE is the only signal that runs code a coordinator chooses. A coordinator can still choose which models you load, and it can delete this install with PURGE; off stops neither.

The first install still trusts the coordinator

The curl | bash that joins a machine runs what the coordinator serves, and that includes writing these settings. Read the script first (read-before-run), or start from the container image.

Signing releases¶

The signing key never goes on the leader, or the signature proves nothing.

# Once, on a machine you trust:
swarm-leader release keygen --out release.key     # prints the public key

# On the leader, for each version of the source it serves:
swarm-leader release digest                       # prints a SHA-256

# Back where the key is:
swarm-leader release sign --key release.key --digest <sha256> --out release.sig

# Copy release.sig to the leader's data directory (SWARM_RELEASE_SIGNATURE_FILE).

Join with the public key pinned:

curl -fsSL "$LEADER/api/v1/bootstrap/fellow.sh" | bash -s -- --leader-url "$LEADER" \
    --updates signed --release-key <public key>

A signature covers the archive and the moment it was signed, and a fellow refuses anything signed before what it last installed, so a coordinator cannot hand it an old release with a known hole. To roll back on purpose, sign the old source again. Repeat --release-key to accept two keys while you rotate.

Python dependencies still come from PyPI, within the ranges in requirements-runtime.txt: a signature covers this project's source and not what it installs.

Worker limits¶

SWARM_FELLOW_SANDBOX, set with --sandbox when joining or in fellow.env:

Value	Meaning
`off` (default)	The worker runs as you, with your permissions.
`on`	Applies what this kernel offers, and goes on without what it lacks.
`required`	Refuses to start work without Landlock and the seccomp filter.

A value that cannot be read means required. It needs no root, container or extra package: the worker limits itself before it starts, using Linux features (Landlock from 5.13, TCP port limits from 6.7). macOS has none of them.

Try it first. This starts a sandboxed process that tries to reach what it should not, then runs a tensor on your GPU, and changes nothing:

fellow sandbox-check

A failure here means a job would have failed too. Then add SWARM_FELLOW_SANDBOX=required to fellow.env and run fellow restart.

What a limited worker can and cannot do¶

Access	What it covers
Read	The system's libraries, the Python environment, the source, `/proc`, `/sys`, the Hub cache, this job's model and data
Write	Its own scratch directory, `/dev/shm`, this job's exports, snapshots and offload directory
Not reach	Your home directory, SSH keys, `node_token`, `node_key`, the Hub token, the fellow's `/proc/<pid>/environ`
Not do	Open a Unix socket by path (`docker.sock`, a desktop session's bus), start a program outside the system and Python directories, ptrace, make namespaces, load kernel modules, use BPF or `io_uring`

Models come from a separate process with the Hub token and HTTPS, no GPU and no pipeline ports. It downloads and parses no weights; the worker then reads them with HF_HUB_OFFLINE=1.

Where this stops¶

Read this before relying on it.

It is not a boundary against the kernel or the GPU driver. A limited worker still talks to both, and a bug in either is a way out. For strangers' work, use a virtual machine.
The network limits are ports, not hosts (SWARM_FELLOW_SANDBOX_NET, limited by default). A worker can reach any machine listening on a pipeline port (5557-5656), so an attacker who runs a listener there can receive what the worker can read. On a mesh, the tunnel's local SOCKS proxy dials what it is asked to, including addresses on the internet. These limits shut out the easy ways out, not a determined one.
A job that streams its dataset from the Hub keeps HTTPS in its worker (hf, and the default WikiText). A job on a bucket or local file does not. The log says which.
What it can read, it can still send over those paths: this job's model and data. The limits protect the rest of the machine.
on limits less on an old kernel, silently apart from the log and the report below. Use required when that matters.

If something breaks¶

Everything is refused by default, so a missing path is the likely failure. The worker's error reaches the fellow's log.

Symptom	Fix
A GPU is not found, on a machine where it works otherwise	`/dev/nvidia-uvm` is usually missing: a limited process cannot run the setuid helper that makes it. Run `nvidia-modprobe -u -c=0` at boot.
A library or a file under `/opt` is not found	Add its directory to `SWARM_FELLOW_SANDBOX_PATHS` (colon-separated, read-only)
A peer cannot be reached on an unusual port	Add it to `SWARM_FELLOW_SANDBOX_PORTS` (comma-separated)
A model under a local path is not found	It must be named by the job, or listed in `SWARM_FELLOW_SANDBOX_PATHS`

SWARM_FELLOW_SANDBOX_NET=open lifts the port limits and leaves the rest in place.

Containers¶

The container image runs as a non-root user, holds its state on one volume and refuses UPDATE, which already closes the coordinator's path. Run it with --cap-drop=ALL --security-opt=no-new-privileges --read-only --tmpfs /tmp; the Helm chart and the compose profile do.

A container shares the host's kernel and GPU driver, so it has the same limit as the worker sandbox.

Virtual machines¶

For work from people you do not trust, the boundary is a virtual machine with the GPU passed through (VFIO): the guest has its own kernel and driver, and a bug in either stays inside.

Enable IOMMU in the firmware and on the kernel command line (intel_iommu=on or amd_iommu=on).
Bind the card to vfio-pci. It then belongs to the guest, and the host cannot use it.
Give the guest the card (Proxmox: Add PCI Device, libvirt: <hostdev>), and install the NVIDIA driver inside.
Join from inside the guest, with nothing from the host mounted in it.

Expect GPU compute speed close to bare metal, with the guest's memory reserved up front. We have not measured this on the cards in our benchmarks, and consumer cards differ in how cleanly they pass through.

gVisor's GPU support (nvproxy) is limited to specific driver versions and, as far as its documentation lists them, data-center cards. Confidential computing needs recent data-center GPUs too. Neither helps on ordinary machines.

What a node reports¶

A fellow reports what it runs under when it registers; swarm fellows show <id> and fellows_get show it:

Level	Meaning
`none`	Nothing beyond your user account. Older fellows count as this.
`restricted`	Workers run limited, or the fellow is in a container
`vm`	The node is a guest of a hypervisor. WSL does not count: it can read the Windows disk.

The report also names the update policy, because a limited worker on auto still leaves the coordinator in control of the machine.

A pool can require a level: Node isolation (minimum) in its settings, swarm pools update --min-isolation restricted, or min_isolation on PATCH /api/v1/pools/{id}. Nodes below it take none of the pool's work.

This is a node's own word

The leader cannot check a report, and a machine that wants to lie can. A minimum holds honest owners to a rule. It is not evidence about a stranger's machine, and it does nothing to protect the work from the host.