Skip to content

Benchmarks

There are no published numbers on this page yet. That is deliberate: the measurements below need a real multi-machine cluster, and we would rather have an empty table than a number produced on a laptop and rounded in our favour.

This page is the methodology, so that when the numbers arrive you can tell whether to believe them - and so you can run the same measurements on your own pool and disagree with ours.

The three questions worth measuring

Everything else is a detail of these.

1. Serving throughput

Tokens per second, and time to first token, for a given model on a given pool. The number people actually feel.

Report separately, because they answer different questions:

  • generation rate - tokens divided by the time actually spent generating, which is what the pipeline can sustain;
  • end-to-end rate - tokens divided by wall-clock from request to last token, which is what a user experiences and includes queueing;
  • time to first token - dominated by the depth of the pipeline and the slowest hop in it.

The leader already records all three per request (leader/telemetry/serving_metrics.py), so this is a matter of driving load and reading GET /api/v1/pools/{pool_id}/metrics, not of new instrumentation.

2. Boundary cost: relay versus direct

The interesting claim in this project is that a pipeline across ordinary internet links is viable. That stands or falls on what crosses the stage boundary and how it gets there.

Measure, for the same model and the same prompts:

  • bytes per token crossing each boundary;
  • added latency per hop, direct peer-to-peer versus relayed through the coordinator;
  • the same pair over a mesh VPN, which is a third path with different characteristics again.

Force the transport rather than letting it choose, so the comparison is a comparison: SWARM_PIPELINE_TRANSPORT=p2p, =tunnel, =mesh.

3. What the learned bottleneck buys

The compression layer at the stage boundary is the reason the previous number is supposed to be tolerable. So it needs its own measurement, with and without:

  • boundary bytes per token, compressed and uncompressed;
  • the resulting change in tokens per second on a link that is actually constrained (a LAN measurement will show almost nothing, which is the point);
  • and the quality cost, on the same evaluation, in the same table. A compression number without the quality it cost is not a result.

Rules we are holding ourselves to

These exist so that a number on this page means something:

  1. Hardware and link, stated. GPU model and VRAM per node, and the actual measured bandwidth and RTT between them - not the marketing speed of the connection.
  2. Model and precision, stated. "7B" is not a specification.
  3. Warm, not first-run. Weight loading and CUDA init are a separate number, reported separately, never averaged into throughput.
  4. Distributions, not means. p50 and p95 with the run count. A mean tokens-per-second on eight requests says nothing.
  5. The unflattering configuration too. If relay costs 3x on a slow link, that number goes on the page next to the fast one.
  6. Reproducible. Commands and configuration in the repository, so the run can be repeated by someone who does not trust us.

Reproducing a run today

The training side already has a suite that plans, runs and writes artefacts:

.venv/bin/python exp/scripts/run_benchmark_suite.py \
  --suite exp/configs/benchmarks/multinode_training_v1.json \
  --leader-url https://your-leader

For serving, drive load against the OpenAI-compatible endpoint and read the pool's metrics afterwards:

# one pool, one loaded model, n concurrent clients
curl -s -H "Authorization: Bearer $SWARM_API_TOKEN" \
     -H 'Content-Type: application/json' \
     -d '{"model":"<model-id>","messages":[{"role":"user","content":"..."}]}' \
     https://your-leader/v1/chat/completions

# then the rolled-up window for that pool
curl -s -H "Authorization: Bearer $SWARM_API_TOKEN" \
     https://your-leader/api/v1/pools/<pool-id>/metrics

The aggregate carries generation rate, end-to-end rate, time to first token, per-request percentiles and the error breakdown, scoped to that pool's nodes.

Results

Date Nodes Model Transport Tokens/s (p50) TTFT (p50) Boundary bytes/token
- - - - - - -

Empty, on purpose. When a row appears here it will have a linked artefact directory behind it under exp/runs/, containing the configuration and the raw records the row was computed from.