Encrypted links¶

Stages of a pipeline send each other activations, gradients and token ids. With SWARM_WIRE_ENCRYPTION on, those links are sealed: encrypted and authenticated end to end between the two stages, on a direct link, over the mesh and through the leader's relay alike.

What each route protects¶

Route	Links plain (`off`)	Links sealed (`prefer`, `require`)
Direct, off the mesh	Readable on the network. Anyone who can reach the port can inject frames	Encrypted. Forged frames are dropped
Direct, on the mesh	Encrypted by WireGuard between the machines	Encrypted twice
Leader relay	Encrypted by TLS to the leader, readable on the leader	The leader only ever holds ciphertext

Turning it on¶

Set this on the leader:

Value	Effect
`off` (default)	Links are plain, as before
`prefer`	Sealed when every stage can do it. Otherwise plain, with a warning in the leader's log. Use it while a cluster updates fellow by fellow
`require`	A fellow that cannot seal is not placed at all, even for work that would run on it alone. A pipeline never starts in the clear. Any unknown value also means `require`, so a typo cannot leave links open

A fellow can seal when it runs current code, with the cryptography package that swarm fellows update installs, and the leader has accepted its keys. Each placement decides afresh. Check a node with swarm fellows show <id>, or see Encrypted links on its panel in the training console: that shows its most recent placement.

What it does not protect¶

A host can read what it computes on. A stage has to decrypt a frame to use it. Sealing protects data between machines, not from the machine doing the work:

Stage	What its host can read
First stage	The prompt, or the training inputs
Middle stages	Activations and their share of the weights. Activations are not plain text, but they are not proven safe from being turned back into their input
Last stage	The generated token ids, so the reply, or the training labels

Run the first and last stages on machines you trust if that matters for your data. Cards with confidential computing are not what most fellows have, and computing on encrypted data is far too slow for this, so neither is on offer.

Other limits, by design:

The leader hands out the keys. Peers learn each other's public keys from the leader, which could swap one. Sealing protects against a stranger on the network and against reading what passes through the relay, not against whoever runs the leader. Key changes are refused and audited (below).
No forward secrecy against a stolen node key. Each direction starts from a fresh ephemeral key, but also from the receiver's long-lived key. Someone who records traffic and later steals that key can read what was sent to it.
A stranger can still block a direct link. A ZeroMQ PAIR socket takes one peer and frames are checked after connecting, so a stranger reaching the port can occupy it. They can neither read nor inject. Serving falls back to the relay; a firewall or the mesh closes the port to them.
Stages only. Prompts travel to the first stage over the fellow's connection to the leader (use HTTPS: a fellow warns when a public leader URL is plain HTTP), and the leader sees them there, as it does the reply.

How it works¶

Node keys. On first start, a fellow creates an Ed25519 signing key and an X25519 link key in its key file (node_key under its state directory, or SWARM_FELLOW_NODE_KEY_FILE; mode 0600). They never leave the machine. Back the file up with the node token if you back up a fellow; a new key needs a rejoin.

Pinning. At registration the fellow sends the public keys, signed. The leader pins the first keys a node shows. Other keys later are refused unless the fellow presents a valid join token, because a stolen node token alone must not let anyone swap in a key of their own. A refused node keeps registering but cannot seal (node_key_status: changed), and the audit log records node.keys_changed. To accept new keys on purpose, rejoin the node with a fresh join token.

Streams. Each direction of each link is its own one-way stream, using RFC 9180 (HPKE): DHKEM(X25519, HKDF-SHA256) in Auth mode, HKDF-SHA256, ChaCha20-Poly1305, with the sequence number carried in each frame. An interactive handshake would not fit: on a serving ring a stage sends to one neighbour and receives from another, a relay link may have nobody at the other end yet, and a peer can restart without notice. The code is held to the RFC's published test vectors.

What a stream is bound to. A random context per job start or model load, the placement epoch, the route (relay or direct), and both nodes' keys. So traffic recorded from an earlier run opens nothing, a frame cannot be moved between routes or nodes, and replays within a stream are dropped. Moving a stage to another node sends the ring the new key with its REWIRE, and a sealed ring accepts only a replacement that can seal.

Cost. On the leader host, an AMD EPYC with AES instructions, ChaCha20-Poly1305 took about 0.5 ms per MB each way (about 480 MB/s). A frame adds under 100 bytes. The effect on tokens per second has not been measured on a GPU cluster yet.