Skip to content

Mesh networking — direct P2P by default (E7)

Status: in production as the userspace mesh below: the leader runs Headscale, mints a single-use key per fellow, keeps the access policy in step with the pools, and fellows join on their own without root. The older system-Tailscale path (bootstrap --mesh-* flags with one shared key) still works for nodes that already run Tailscale.

Userspace mesh (the default)

What a cluster owner does: add one DNS record, mesh.<domain> pointing at the leader's host. Nothing else - no ports opened anywhere, no root on the fellows.

  1. Leader side. docker-compose.yml runs headscale (pinned v0.29.4) beside the leader; the prod overlay routes https://mesh.<domain> to it through Traefik. The leader image carries the headscale CLI and reaches the server through its control socket on a shared volume (the container runs with the leader's group, socket mode 0770 - no API key to provision). leader/cluster/mesh_control.py checks https://mesh.<domain>/health (SWARM_HEADSCALE_URL, set by the prod overlay; the SWARM_MESH_* variables belong to the older system-Tailscale path below and stay empty); once it answers, the mesh is offered (an admin can switch it off on Admin > Direct links).
  2. The offer. A fellow that declares mesh-userspace gets {"login_server": ...} in its registration answer and in every heartbeat answer (mesh_known marks a leader that sends it; null means leave), so a switch or a DNS record going live reaches running fellows within seconds.
  3. Joining. fellow/mesh_agent.py downloads the pinned Tailscale release into the install directory, checks its SHA-256 before unpacking (only the two binaries, by base name), starts tailscaled --tun=userspace-networking as the fellow's own user, asks POST /api/v1/mesh/key (node credential) for a single-use, 10-minute key tagged tag:pool-<pool id>, and runs tailscale up. It re-registers with its mesh address, which becomes its host; the node state persists, so a restart rejoins without a new key. A failure leaves the fellow on the relay and retries after ten minutes.
  4. Traffic. Without a network interface, a mesh address is reachable only through the SOCKS5 proxy tailscaled opens on loopback. The fellow exports it as SWARM_MESH_SOCKS; runtime/netpath.py makes links to 100.64.0.0/10 dial through it (a SOCKS5 CONNECT for stage 0's reachability check, and for training and serving links a local forwarder, pipe/link/socks.py, that the ZeroMQ socket dials on 127.0.0.1 - libzmq's own SOCKS_PROXY client aborts the process with Bad file descriptor (src/epoll.cpp:86) when a proxied connection fails or drops, which every job's end is). Inbound needs nothing: the tunnel hands connections to the host's ports, so the listening stage binds the port on every interface rather than the mesh address, which is on none (pipe/link/resolve.bind_endpoint).
  5. Access policy. A userspace tunnel would hand any inbound connection to the host, so the policy is what protects the rest of a fellow's machine: the leader sets it (Headscale database policy mode) to let fellows of one pool reach each other on SWARM_MESH_PORTS (default 5557-5600) and nothing else. Fellows of other pools do not even see each other.

Headscale's default DERP map (Tailscale's public relays) carries traffic that cannot be hole-punched, encrypted end to end; the leader's host needs no UDP port. Admin > Direct links shows each fellow's tunnel and, per peer, whether the path is direct or via relay (from tailscale status, reported as telemetry.mesh_peers).

Why

SilentSwarm's data plane wants direct fellow-to-fellow ZMQ links: activations flow stage 0 → stage 1, gradients flow back, and nothing else needs to touch that traffic. But fellows usually live behind NAT/firewalls and can't reach each other, so the cluster historically relayed every activation and gradient tensor through the leader (tunnel transport). That makes the leader a star bottleneck — double the bandwidth crosses it, and large activations bump into HTTP body-size caps.

A mesh VPN removes the trade-off. Every node — leader and fellows — gets a stable, mutually-routable address on an overlay network. The leader's existing reachability probe then succeeds, and auto transport selection upgrades to direct P2P by default, while keeping the "works behind NAT with no config" promise via the mesh's own NAT traversal + relay fallback.

We use Tailscale (WireGuard data plane + DERP relay fallback + NAT hole punching) with an optional self-hosted Headscale control server, so the whole coordination plane can stay self-hosted.

Topology

The data plane is a pipeline (chain), not a mesh of application traffic — each fellow only exchanges tensors with its adjacent stage. The VPN is a mesh at L3; the SilentSwarm traffic riding on it is still a chain.

flowchart LR
  subgraph mesh["Mesh VPN overlay (100.64.0.0/10)"]
    L["Leader<br/>100.64.0.1<br/>(control plane + relay fallback)"]
    F0["Fellow stage 0<br/>100.64.0.10"]
    F1["Fellow stage 1<br/>100.64.0.11"]
  end
  F0 -- "activations (ZMQ, direct)" --> F1
  F1 -- "gradients (ZMQ, direct)" --> F0
  F0 -. "register / heartbeat / logs" .-> L
  F1 -. "register / heartbeat / logs" .-> L
  • Control plane (register, heartbeat, signals, logs) stays a star to the leader — it's tiny and never the bottleneck.
  • Data plane goes direct fellow↔fellow over the overlay. The leader relay remains only as a fallback for off-mesh nodes.

How the default flips to P2P

The leader already probes each fellow's ZMQ ports and auto upgrades to p2p when all fellows are reachable (select_transport_mode in src/silent_swarm/leader/training/dispatch.py, which applies the rule in pipe/link/resolve.py). The mesh makes that path the default in two ways:

  1. The fellow advertises its mesh IP as its host. The bootstrap runs tailscale up, reads tailscale ip -4, and uses that as SWARM_HOST, so register reports the overlay address. The leader stores the mesh IP as the worker's routable host, so all existing probe/peer/routing code transparently uses the overlay.
  2. Mesh membership counts as reachable. auto treats a node with an up mesh handshake as reachable even if a just-in-time TCP probe is momentarily blocked, because the overlay guarantees a routable path (with DERP relay fallback).

Transport policy values (SWARM_PIPELINE_TRANSPORT):

Value Behaviour
auto (default) P2P when every fellow is reachable or on the mesh, else tunnel.
p2p Force direct; fails if a peer is unreachable.
tunnel Force leader relay (legacy).
mesh Assert the mesh: P2P only if every fellow is on the overlay, else refuse and fall back to tunnel.

Data flow through the code

sequenceDiagram
    participant BS as fellow bootstrap
    participant TS as tailscale/Headscale
    participant FR as fellow runner
    participant LD as leader
    participant UI as Ops UI
    BS->>TS: tailscale up --authkey --login-server (shared.mesh.mesh_up)
    TS-->>BS: mesh ip 100.64.0.x
    BS->>FR: SWARM_HOST=mesh ip, SWARM_MESH_IP, SWARM_MESH_BACKEND
    FR->>LD: POST /register {host=mesh ip, mesh_ip, mesh_backend}
    LD->>LD: store host=mesh_ip, auto selects p2p
    FR->>LD: POST /heartbeat {mesh_online} (re-checks tailscale status)
    LD->>UI: /cluster/nodes + /join-graph {mesh_ip, mesh_online, p2p_capable}
    UI->>UI: mesh badge, On mesh count, P2P edges

Integration points (files touched)

Layer File What it does
Mesh helpers src/silent_swarm/shared/mesh.py mesh_up, mesh_ipv4, mesh_status, is_mesh_ip — thin tailscale CLI wrappers.
Contract src/silent_swarm/shared/contracts.py RegisterRequest.mesh_ip/mesh_backend, HeartbeatRequest.mesh_online.
Fellow src/silent_swarm/fellow/runner.py, fellow/heartbeat.py Detect/advertise mesh IP; report live handshake each heartbeat.
Bootstrap src/silent_swarm/bootstrap/entry.py, leader/cluster/fellow_bootstrap.sh, leader/cluster/bootstrap_contract.py --mesh-* flags, configure_mesh join step, host precedence, join-command injection.
Registry src/silent_swarm/leader/cluster/registry.py WorkerEntry mesh fields; host = mesh_ip or host; heartbeat updates mesh_online.
Persistence src/silent_swarm/leader/cluster/join_persistence.py workers.mesh_ip/mesh_backend/mesh_online columns + additive migration.
Transport src/silent_swarm/leader/training/dispatch.py, pipe/link/resolve.py select_transport_mode gathers reachability (probe, mesh); choose_route applies the auto/mesh policy.
Ops UI data src/silent_swarm/leader/web/join_ui.py, app.py (/cluster/nodes) Mesh fields + mesh-aware p2p_capable.
Ops UI view leader/templates/ops_join.{html,js,css}, pool_detail.html Mesh badge, "On mesh" count, legend, table column, join-command mesh flags.

Address plan

Tailscale/Headscale assign from the CGNAT block 100.64.0.0/10. We treat membership in that block as "this is a mesh address" (shared.mesh.is_mesh_ip). No manual IPAM: Headscale allocates, and nodes are identified on the overlay by hostname fellow-<id> (and the leader by its own tailnet name).

Leader on the tailnet

The leader must also be on the overlay to probe and route to fellows. Two options:

  • Sidecar (recommended for Docker): run a tailscale container in the leader's network namespace (network_mode: service:tailscale) with TS_AUTHKEY and, for Headscale, TS_EXTRA_ARGS=--login-server=…. The leader process then sees the tailscale0 interface.
  • Host tailscale: if the leader runs on a host already on the tailnet, nothing extra is needed.

Enable via .env:

SWARM_MESH_BACKEND=tailscale
SWARM_MESH_LOGIN_SERVER=https://headscale.example.com   # omit for Tailscale SaaS
SWARM_MESH_AUTHKEY=tskey-auth-...                        # reusable pre-auth key
SWARM_PIPELINE_TRANSPORT=auto                            # already the default

The leader injects --mesh-backend/--mesh-authkey/--mesh-login-server into the curl … | bash join command shown in the Ops UI and pool pages, so a new node joins the overlay and comes up P2P-capable with no extra operator steps.

Security model

  • Data plane is WireGuard-encrypted end to end; ZMQ tensor traffic never traverses the public internet in cleartext.
  • ACLs: restrict the tailnet so fellows may reach each other only on the ZMQ ports and the leader, via Headscale ACL policy (out of band).
  • Auth keys: the scaffold ships a reusable pre-auth key in the join command. That key is a bearer secret — anyone with it can join the tailnet. Treat it as sensitive and rotate it; prefer the per-node minting below.
  • Firewall rules on the ZMQ ports (the existing --allow-from ufw path) can be scoped to the mesh CIDR so the ports are reachable only over the overlay.

Production hardening

Done for the userspace mesh: single-use, short-TTL keys per fellow, pool tags, pool-scoped access policy, and per-edge direct-vs-relayed reporting. Still open: expiring a node in Headscale when it is purged or released, and the same key minting for the system-Tailscale path below, where bootstrap_contract.mesh_join_flags() still passes one shared reusable key:

  1. In the join-session flow (POST /api/v1/bootstrap/join-sessions), call the Headscale API to mint a single-use, short-TTL pre-auth key bound to that session, and inject that into the returned mesh_join_flags.
  2. Tag the node in Headscale (--advertise-tags) so ACLs apply automatically.
  3. On node purge/release, expire the node in Headscale so its overlay address is reclaimed.

Other follow-ups: DERP/relay health surfaced in the Ops UI; per-edge "direct vs DERP-relayed" indication (Tailscale exposes this in tailscale status --json); and scoping the ZMQ firewall rules to the mesh CIDR automatically during bootstrap.

Fallback behaviour

If tailscale is missing, the auth key is absent, or the join fails, the bootstrap logs a warning and the node stays off-mesh — it advertises its LAN/FQDN host and the cluster falls back to the leader relay for that node. Mesh is an upgrade, never a hard dependency.