Skip to content

Training console — /ops/join-v2

The operations hub: add machines to the swarm, launch training runs, and watch the cluster live. Everything here is scoped to the pool selected in the sidebar.

Training console

Train from a bucket

Pick a bucket to launch a run using its base model and datasets. The bucket bundles what to train; the console runs it on the pool's nodes.

Add fellow

Generate join command creates a short‑lived, pool‑scoped invite and shows a one‑line curl … | bash you run on any GPU machine to add it to the swarm. The node bootstraps itself and connects back. The bootstrap is HTTPS‑only, verifies a published checksum, and (in beta) can be locked to invite tokens — see bootstrap hardening.

Swarm graph

A live force‑directed graph of the cluster. The Leader sits at the center; each connected fellow is a node whose shape/colour encode status. The legend and counters track Connected, Training, Direct P2P, On mesh, and Offline. Edges show the transport between stages — a solid P2P link (direct fellow‑to‑fellow, the default on a mesh) or a relayed hop through the leader. Click any node to inspect its GPUs, free VRAM, transport, mesh IP/handshake, and throughput.

The graph, the counters, the cluster telemetry, and the fellows table all show the pool selected in the sidebar — nothing else. A node joins the pool whose join command it was bootstrapped with, so if a machine you just added is missing, it most likely landed in another pool: the console says so above the graph and links straight to the pool that holds it.

Cluster performance

Aggregate telemetry for the selected scope: fellow/GPU counts, current step, step time, throughput, average loss, and cluster VRAM + network‑throughput meters.

Training & connected fellows

  • Training — queued and running jobs with live step, loss, and a loss curve; stop a run from here.
  • Connected fellows — a table of the pool's nodes with hostname, version, status, Mesh membership, GPUs, VRAM, TX/RX throughput, step, loss, and last‑seen. Select a row to stream that fellow's live logs.
  • Log stream — sits directly under the fellows table and follows whichever node you selected there (or clicked in the graph). It tails automatically; scroll up to read back and the follow pauses until you hit Jump to latest. Copy puts the buffered lines on the clipboard, Clear empties the view without dropping the stream, Close stops it.