Mesh networking — direct P2P by default (E7)¶

Status: scaffold. The control-plane wiring, persistence, transport selection, bootstrap join step, and Ops UI surfacing are implemented. The one deliberate stub is per-node key minting (see Production hardening).

Why¶

SilentSwarm's data plane wants direct fellow-to-fellow ZMQ links: activations flow stage 0 → stage 1, gradients flow back, and nothing else needs to touch that traffic. But fellows usually live behind NAT/firewalls and can't reach each other, so the cluster historically relayed every activation and gradient tensor through the leader (tunnel transport). That makes the leader a star bottleneck — double the bandwidth crosses it, and large activations bump into HTTP body-size caps.

A mesh VPN removes the trade-off. Every node — leader and fellows — gets a stable, mutually-routable address on an overlay network. The leader's existing reachability probe then succeeds, and auto transport selection upgrades to direct P2P by default, while keeping the "works behind NAT with no config" promise via the mesh's own NAT traversal + relay fallback.

We use Tailscale (WireGuard data plane + DERP relay fallback + NAT hole punching) with an optional self-hosted Headscale control server, so the whole coordination plane can stay self-hosted.

Topology¶

The data plane is a pipeline (chain), not a mesh of application traffic — each fellow only exchanges tensors with its adjacent stage. The VPN is a mesh at L3; the SilentSwarm traffic riding on it is still a chain.

flowchart LR
  subgraph mesh["Mesh VPN overlay (100.64.0.0/10)"]
    L["Leader<br/>100.64.0.1<br/>(control plane + relay fallback)"]
    F0["Fellow stage 0<br/>100.64.0.10"]
    F1["Fellow stage 1<br/>100.64.0.11"]
  end
  F0 -- "activations (ZMQ, direct)" --> F1
  F1 -- "gradients (ZMQ, direct)" --> F0
  F0 -. "register / heartbeat / logs" .-> L
  F1 -. "register / heartbeat / logs" .-> L

Control plane (register, heartbeat, signals, logs) stays a star to the leader — it's tiny and never the bottleneck.
Data plane goes direct fellow↔fellow over the overlay. The leader relay remains only as a fallback for off-mesh nodes.

How the default flips to P2P¶

The leader already probes each fellow's ZMQ ports and auto upgrades to p2p when all fellows are reachable (_select_pipeline_transport_mode in src/silent_swarm/leader/app.py). The mesh makes that path the default in two ways:

The fellow advertises its mesh IP as its host. The bootstrap runs tailscale up, reads tailscale ip -4, and uses that as SWARM_HOST, so register reports the overlay address. The leader stores the mesh IP as the worker's routable host, so all existing probe/peer/routing code transparently uses the overlay.
Mesh membership counts as reachable. auto treats a node with an up mesh handshake as reachable even if a just-in-time TCP probe is momentarily blocked, because the overlay guarantees a routable path (with DERP relay fallback).

Transport policy values (SWARM_PIPELINE_TRANSPORT):

Value	Behaviour
`auto` (default)	P2P when every fellow is reachable or on the mesh, else `tunnel`.
`p2p`	Force direct; fails if a peer is unreachable.
`tunnel`	Force leader relay (legacy).
`mesh`	Assert the mesh: P2P only if every fellow is on the overlay, else refuse and fall back to `tunnel`.

Data flow through the code¶

sequenceDiagram
    participant BS as fellow bootstrap
    participant TS as tailscale/Headscale
    participant FR as fellow runner
    participant LD as leader
    participant UI as Ops UI
    BS->>TS: tailscale up --authkey --login-server (shared.mesh.mesh_up)
    TS-->>BS: mesh ip 100.64.0.x
    BS->>FR: SWARM_HOST=mesh ip, SWARM_MESH_IP, SWARM_MESH_BACKEND
    FR->>LD: POST /register {host=mesh ip, mesh_ip, mesh_backend}
    LD->>LD: store host=mesh_ip, auto selects p2p
    FR->>LD: POST /heartbeat {mesh_online} (re-checks tailscale status)
    LD->>UI: /cluster/nodes + /join-graph {mesh_ip, mesh_online, p2p_capable}
    UI->>UI: mesh badge, On mesh count, P2P edges

Integration points (files touched)¶

Layer	File	What it does
Mesh helpers	`src/silent_swarm/shared/mesh.py`	`mesh_up`, `mesh_ipv4`, `mesh_status`, `is_mesh_ip` — thin `tailscale` CLI wrappers.
Contract	`src/silent_swarm/shared/contracts.py`	`RegisterRequest.mesh_ip/mesh_backend`, `HeartbeatRequest.mesh_online`.
Fellow	`src/silent_swarm/fellow/runner.py`, `fellow/heartbeat.py`	Detect/advertise mesh IP; report live handshake each heartbeat.
Bootstrap	`src/silent_swarm/bootstrap/entry.py`, `leader/cluster/fellow_bootstrap.sh`, `leader/cluster/bootstrap_contract.py`	`--mesh-*` flags, `configure_mesh` join step, host precedence, join-command injection.
Registry	`src/silent_swarm/leader/cluster/registry.py`	`WorkerEntry` mesh fields; `host = mesh_ip or host`; heartbeat updates `mesh_online`.
Persistence	`src/silent_swarm/leader/cluster/join_persistence.py`	`workers.mesh_ip/mesh_backend/mesh_online` columns + additive migration.
Transport	`src/silent_swarm/leader/app.py`	`_select_pipeline_transport_mode` mesh-awareness + `mesh` policy.
Ops UI data	`src/silent_swarm/leader/web/join_ui.py`, `app.py` (`/cluster/nodes`)	Mesh fields + mesh-aware `p2p_capable`.
Ops UI view	`leader/templates/ops_join.{html,js,css}`, `pool_detail.html`	Mesh badge, "On mesh" count, legend, table column, join-command mesh flags.

Address plan¶

Tailscale/Headscale assign from the CGNAT block 100.64.0.0/10. We treat membership in that block as "this is a mesh address" (shared.mesh.is_mesh_ip). No manual IPAM: Headscale allocates, and nodes are identified on the overlay by hostname fellow-<id> (and the leader by its own tailnet name).

Leader on the tailnet¶

The leader must also be on the overlay to probe and route to fellows. Two options:

Sidecar (recommended for Docker): run a tailscale container in the leader's network namespace (network_mode: service:tailscale) with TS_AUTHKEY and, for Headscale, TS_EXTRA_ARGS=--login-server=…. The leader process then sees the tailscale0 interface.
Host tailscale: if the leader runs on a host already on the tailnet, nothing extra is needed.

Enable via .env:

SWARM_MESH_BACKEND=tailscale
SWARM_MESH_LOGIN_SERVER=https://headscale.example.com   # omit for Tailscale SaaS
SWARM_MESH_AUTHKEY=tskey-auth-...                        # reusable pre-auth key
SWARM_PIPELINE_TRANSPORT=auto                            # already the default

The leader injects --mesh-backend/--mesh-authkey/--mesh-login-server into the curl … | bash join command shown in the Ops UI and pool pages, so a new node joins the overlay and comes up P2P-capable with no extra operator steps.

Security model¶

Data plane is WireGuard-encrypted end to end; ZMQ tensor traffic never traverses the public internet in cleartext.
ACLs: restrict the tailnet so fellows may reach each other only on the ZMQ ports and the leader, via Headscale ACL policy (out of band).
Auth keys: the scaffold ships a reusable pre-auth key in the join command. That key is a bearer secret — anyone with it can join the tailnet. Treat it as sensitive and rotate it; prefer the per-node minting below.
Firewall rules on the ZMQ ports (the existing --allow-from ufw path) can be scoped to the mesh CIDR so the ports are reachable only over the overlay.

Production hardening¶

The deliberate stub is per-node key minting. bootstrap_contract.mesh_join_flags() currently passes one shared reusable key. The production path:

In the join-session flow (POST /api/v1/bootstrap/join-sessions), call the Headscale API to mint a single-use, short-TTL pre-auth key bound to that session, and inject that into the returned mesh_join_flags.
Tag the node in Headscale (--advertise-tags) so ACLs apply automatically.
On node purge/release, expire the node in Headscale so its overlay address is reclaimed.

Other follow-ups: DERP/relay health surfaced in the Ops UI; per-edge "direct vs DERP-relayed" indication (Tailscale exposes this in tailscale status --json); and scoping the ZMQ firewall rules to the mesh CIDR automatically during bootstrap.

Fallback behaviour¶

If tailscale is missing, the auth key is absent, or the join fails, the bootstrap logs a warning and the node stays off-mesh — it advertises its LAN/FQDN host and the cluster falls back to the leader relay for that node. Mesh is an upgrade, never a hard dependency.