Benchmarks¶
One model, measured end to end on real home hardware: Qwen3-14B served across
two ordinary PCs on one home internet connection, through the leader's public
chat API, the way a client sees it. Every number below comes from a raw result
file in the repository, and the scripts that produced them are in
exp/spikes/, so you can run the same measurements on your own pool and
disagree with ours.
In one paragraph
Two home PCs with four older GPUs between them (none can hold the model alone) serve Qwen3-14B at 3.3 tokens/s to one user and 11.6 tokens/s in total to eight users at once, each seeing a first token after about 6 seconds. When a reply repeats its prompt, as a code edit does, guessing ahead raises one user to 5.4 tokens/s - and the same model on one of the PCs to 9.2 (section 4). On 2026-10-16 the same PCs served 3.2 and 9.9 tokens/s. Through the leader's internet relay instead of a direct link, eight users get the same 9.9 tokens/s. The learned bottleneck sends a seventh of the bytes across the boundary, and the fine-tune behind it scores better than the uncompressed base model on text it never saw (58.3 % against 57.0 % next-token accuracy).
Setup¶
| Item | Value |
|---|---|
| Date | 2026-10-16 |
| Code | commit 1a9e328 (main) |
| Model | Qwen3-14B-c2483d93: Qwen3-14B with 4-bit NF4 weights, LoRA adapters on attention and MLP, and a learned bottleneck at layer 17 (half the width, 8-bit codes). Trained on the same cluster: 1,000 steps on OpenAssistant Guanaco. |
| Stage 0 | layers 0-16, on PC 1: 2x NVIDIA GTX 1050 Ti, 4 GB each, computing in float32 (Pascal, compute capability 6.1) |
| Stage 1 | layers 17-39 and the output head, on PC 2: 2x NVIDIA GTX 1660 SUPER, 6 GB each, computing in float16 (Turing, compute capability 7.5) |
| One PC | the whole model on PC 2's two cards (num_stages: 1) |
| Network | both PCs on one home internet connection; the leader in a datacentre, 21 ms round trip away. Direct: the PCs talk over the Headscale mesh (userspace Tailscale), the default. Relay: every frame goes PC -> leader -> PC, two trips across the internet through the home upload (SWARM_PIPELINE_TRANSPORT=tunnel). |
| Serving settings | the defaults of this commit: up to 8 requests in flight, decode batches of at most 2 (both PCs have pre-Ampere cards), prompts in pieces of 256 tokens, a KV budget measured per stage (3,586 tokens on stage 0) |
Method¶
exp/spikes/serving/bench_serving.pysends streaming requests to/api/v1/chat/completionswith an API token, as an OpenAI client would: greedy decoding, thinking off, 128 new tokens per reply, short prompts (one of eight everyday questions).- Each concurrency level is two rounds of n simultaneous requests: 2, 8 and 16 requests for 1, 4 and 8 users. One short request first pays for the warm-up; loading is not in any number.
- TTFT is from sending the request to the first streamed token. Tokens/s per user is a reply's tokens after the first, over the time after the first. Tokens/s total is all tokens of a level over its wall-clock time. Percentiles are over the individual requests.
- The fellows report where a token's time went (
swarm.timingon the last chunk): each stage's compute and the link, which includes the time a position waited behind other requests' positions. exp/spikes/serving/summarize.pyturns the raw files into the tables below.
1. Serving throughput¶
| Setup | Users | TTFT p50 / p95 | Tokens/s per user p50 / p95 | Tokens/s total | Errors |
|---|---|---|---|---|---|
| Two PCs, direct | 1 | 1.4 / 1.4 s | 3.2 / 3.2 | 3.2 | 0 of 2 |
| Two PCs, direct | 4 | 3.4 / 4.2 s | 2.3 / 2.6 | 8.7 | 0 of 8 |
| Two PCs, direct | 8 | 5.7 / 6.8 s | 1.3 / 1.4 | 9.9 | 0 of 16 |
| Two PCs, relay | 1 | 1.5 / 1.5 s | 2.9 / 2.9 | 2.8 | 0 of 2 |
| Two PCs, relay | 4 | 3.6 / 4.4 s | 2.3 / 2.3 | 8.5 | 0 of 8 |
| Two PCs, relay | 8 | 5.9 / 7.1 s | 1.4 / 1.4 | 9.9 | 0 of 16 |
| One PC | 1 | 1.2 / 1.3 s | 4.2 / 4.3 | 4.1 | 0 of 2 |
| One PC | 4 | 46.7 / 91.6 s | 4.4 / 4.5 | 4.2 | 0 of 8 |
A 1,526-token prompt with 32 new tokens: TTFT 58.7 s on two PCs over the mesh, 61.2 s through the relay, 67.7 s on one PC. Prompts are compute-bound on these cards: about 40 ms per prompt token.
A 4,026-token conversation is refused before it starts, in under a second:
this conversation needs 4058 tokens (4026 in the prompt and up to 32 in the
reply); the pool serving this model holds 3586. Before the KV budget was
measured on the stage, the same request ran stage 0's 4 GB card out of memory
part way through its prompt.
One PC serves one request at a time, so with 4 users the later ones wait for whole replies: 47 s to the first token at the median, 92 s at p95.
Where a token's time goes, two PCs over the mesh, one user: stage 0 computes for 112 ms, stage 1 for about 172 ms, and the link adds about 5 ms. Corrected on 2026-10-18: this page first said stage 1 137 ms and link 58 ms. Stage 1 timed its compute before it waited for the GPU, so about a third of its time was reported as link; the stages' own logs showed the real split, and the fellows report it correctly since. The throughput and TTFT figures were never affected. With 8 users, stage 0 works on batches of 2 for 156 ms each and stays busy; the link figure (488 ms) is mostly positions waiting their turn.
2. Boundary cost¶
Relay against direct. For one user the relay costs about 10 %: 2.8 against
3.2 tokens/s. The fellows reported 107 ms of link per token against 58 ms;
both figures held about 56 ms of stage 1's compute (see the correction above),
so the relay's own cost is about 50 ms a token against about 5 ms direct. With 4 or 8 users it
costs nothing measurable (8.5 against 8.7, and 9.9 against 9.9): enough requests
are in flight to hide the link behind the compute. That took a pipelined relay
link that bundles frames; before it, the relay capped every load at about
4.5 tokens/s (exp/spikes/serving/README.md).
Compressed against not. The learned bottleneck sends 2.7 KB per token
across the boundary; the full hidden state is 20.5 KB. While decoding, that
changes nothing - a token is too small for bandwidth to matter, and the link
costs round trips. It matters for prompts, and only over the internet: a
526-token prompt crosses as 1.4 MB compressed or 10.8 MB uncompressed, and
through the relay that was 25.7 s against 35.3 s to the first token,
9.5 s of upload through the home connection. Over the mesh the two were
within half a second. These compressed-against-uncompressed numbers are from
2026-10-15 (results/direct_*.json, results/relay_*.json), on code from before
the batched decode; the uncompressed copy was not re-run on this commit.
3. Quality cost¶
Scored offline by replaying the exported model layer by layer on the CPU, with
NF4-rounded weights, the adapters and both bottleneck halves applied at layer 17
(exp/spikes/compression/score_offline.py). 128-token windows, 8,192 tokens per
set; the standard error of a set's mean window loss is 0.04 to 0.08.
| Text | Base model, uncompressed: loss / top-1 | Fine-tune behind the 4x boundary: loss / top-1 |
|---|---|---|
| Guanaco, held-out rows (never read in training) | 2.056 / 57.0 % | 1.847 / 58.3 % |
| WikiText-103 validation | 2.655 / 50.7 % | 2.662 / 47.4 % |
On unseen text from the training domain, the compressed fine-tune beats the
uncompressed base. On out-of-domain text the loss is unchanged and top-1 is 3.3
points lower - plausibly the cost of fine-tuning on chat data, but this replay
cannot separate that from the bottleneck's share. The compression measurements
on a smaller model (Qwen3-0.6B) that led to this design are in
exp/spikes/compression/README.md.
4. Faster since: batches of 4 and guessing ahead¶
Measured 2026-10-19 on the same PCs, model and settings, on the code of branch
feat/ts0-serving-speed-spike (commit 1f4d013). Two changes:
- Batches of 4 on old cards. Each fellow times its 4-bit matmuls at load and splits calls of 4 or more rows where its cards are slow at them (the GTX 1660s: 4.7 ms a block for 3 rows, 26-32 ms for 4). The leader then lets these cards decode 4 requests at once instead of 2.
- Guessing ahead (speculative decoding). Stage 0 proposes up to 2 next
tokens by finding the reply's last few tokens earlier in the conversation,
and the pipeline checks them in the same pass; every guess the model would
not have made is thrown away, so the reply is the same. On by
SWARM_SERVE_SPECULATIVE=2, off by default.
Throughput, bench_serving.py as in section 1 (results/m1_two_pc_default.json
without guessing, results/m2_two_pc_levels_spec_on.json with):
| Setup | Users | TTFT p50 / p95 | Tokens/s per user p50 / p95 | Tokens/s total | Errors |
|---|---|---|---|---|---|
| Batches of 4 | 1 | 1.4 / 1.4 s | 3.3 / 3.3 | 3.2 | 0 of 2 |
| Batches of 4 | 4 | 3.4 / 4.1 s | 2.5 / 2.6 | 9.5 | 0 of 8 |
| Batches of 4 | 8 | 5.8 / 6.9 s | 1.6 / 1.6 | 11.6 | 0 of 16 |
| Batches of 4, guessing ahead | 1 | 1.4 / 1.4 s | 3.3 / 3.4 | 3.2 | 0 of 2 |
| Batches of 4, guessing ahead | 4 | 3.4 / 4.1 s | 2.7 / 2.8 | 10.4 | 0 of 8 |
| Batches of 4, guessing ahead | 8 | 5.7 / 6.8 s | 1.5 / 1.6 | 11.6 | 0 of 16 |
Guessing ahead, by kind of request. 18 prompts, one at a time, greedy, up
to 256 tokens each: 8 everyday questions, 4 code edits (rename a variable, add
type hints, fix a bug, in 15-40 lines of Python), 2 summaries and 4 questions
about a pasted passage. Medians per kind (exp/spikes/serving/spec_acceptance.py,
results/m2_two_pc_spec_{off,on}.json, results/m1_one_pc_spec_{off,on}.json):
| Kind | Replies | Two PCs: off / on | One PC: off / on | Tokens per pass |
|---|---|---|---|---|
| Code edits | 4 | 3.2 / 5.4 (1.7x) | 4.3 / 9.2 (2.1x) | 2.1 |
| Summaries | 2 | 3.2 / 3.9 (1.2x) | 4.4 / 6.2 (1.4x) | 1.4-1.5 |
| Questions about a passage | 4 | 3.1 / 3.3 (1.05x) | 4.4 / 4.9 (1.1x) | 1.2 |
| Everyday questions | 8 | 3.3 / 3.3 (1.0x) | 4.3 / 4.7 (1.1x) | 1.1 |
- It pays where a reply repeats what it was given. In open conversation the model rarely says what it said before, so there is little to guess and nothing is lost.
- One PC gains more than two because its GTX 1660s check 3 positions in about the time of 1; on two PCs the GTX 1050 Tis of stage 0 pay about 58 % per extra position.
- With 4 or 8 users it changes nothing at 8 and helps at 4: stage 0 guesses only while the pipeline has at most one request per stage, so a full pipeline decodes in batches as before.
- The one-PC run with guessing was a load that also measured its own step costs; the others were not.
Limits, next to the numbers¶
- Prompts are slow on these cards: about 40 ms per prompt token, so a 1,500-token prompt waits a minute for its first token.
- One PC beats two for a single user. With no link and no handoff, PC 2 alone serves 4.2 tokens/s to one user against 3.2 across both PCs. Two PCs serve more users at once because they have twice the cards - and they are the only way to serve a model none of the cards holds alone, which is the point.
- Old cards batch poorly. A decode batch of 4 took the GTX 1660 SUPERs 671 ms against about 120 ms for one session - the 4-bit kernel bitsandbytes picks from 4 rows on is slow on a Turing card without tensor cores. Since 2026-10-18 the fellows split such calls into calls of 3 rows, a batch of 4 takes about 211 ms, and the cap is 4 (section 4). The GTX 1050 Tis compute in float32 and get about 58 % slower per extra position. Newer cards are expected to batch far better, and are not measured here.
- Two PCs on one home connection are not two homes. The relay numbers cross the internet twice per position, but both ends share one upload. Two homes in different cities would add their own round trip to every token of a single user.
- Small samples. Two rounds per level, 2 to 16 requests: p95 is close to the maximum. The numbers show the shape, not a guarantee.
- The KV budget is small on stage 0. 3,586 tokens fit beside the weights on two 4 GB cards, so a conversation longer than that is refused on this pool, and 8 users with long replies queue rather than run at once.
Reproducing a run¶
# 1. load the model (two stages, or one PC with --num-stages 1 --fellow-id <id>)
swarm model-load <model-id> --wait
# 2. drive it: 1, 4 and 8 users, two rounds, 128 tokens, then a 1,500-token prompt
.venv/bin/python exp/spikes/serving/bench_serving.py <model-id> out.json \
--levels 1,4,8 --rounds 2 --max-tokens 128 --long-tokens 1500
# 3. the tables
.venv/bin/python exp/spikes/serving/summarize.py out.json
The script reads the leader URL and an API token from ~/.swarm/config.toml.
For the relay, start the leader with SWARM_PIPELINE_TRANSPORT=tunnel and load
the model again. Raw results of this page: exp/spikes/serving/results/v7_*.json.
The history of how the numbers got here - and every fix the measurements forced
- is in exp/spikes/serving/README.md.
The training side has its own suite, which plans, runs and writes artefacts:
.venv/bin/python exp/scripts/run_benchmark_suite.py \
--suite exp/configs/benchmarks/multinode_training_v1.json \
--leader-url https://your-leader
Rules we hold these numbers to¶
- Hardware and link, stated. GPU model and VRAM per node, and the measured round trip - not the marketing speed of the connection.
- Model and precision, stated. "14B" is not a specification.
- Warm, not first-run. Loading and CUDA start-up are never averaged into throughput.
- Distributions, not means. p50 and p95, with the number of requests.
- The unflattering configuration too. One PC beating two for one user is on this page because it is true.
- Reproducible. Commands, configuration and raw results in the repository.