Running one LLM across mismatched GPUs: measured results¶
Four old GPUs in two home PCs, none able to hold the model alone, served Qwen3-14B at 3.3 tokens/s to one user and 11.6 tokens/s in total to eight. It works, and it costs you something: the same model on one of those PCs gave one user 4.2 tokens/s.
The setup¶
| Item | Value |
|---|---|
| Model | Qwen3-14B, 4-bit weights, fine-tuned with LoRA adapters |
| PC 1 | 2x GTX 1050 Ti, 4 GB each, holding layers 0-16 |
| PC 2 | 2x GTX 1660 SUPER, 6 GB each, holding layers 17-39 and the output head |
| Network | Both PCs on one home connection |
| Client | Streaming requests through the public API, 128 tokens per reply |
The layers are split by what each card can hold, so the mismatch is the point of the test rather than an obstacle to it.
The numbers¶
| Users | Tokens/s per user | Tokens/s in total | First token |
|---|---|---|---|
| 1 | 3.3 | 3.2 | 1.4 s |
| 4 | 2.5 | 9.5 | 3.4 s |
| 8 | 1.6 | 11.6 | 5.8 s |
There were no errors in any of these runs. They are small samples, of 2 to 16 requests per level. The benchmarks page has the method and all of the results.
Why eight users get more than one¶
Every request has to pass through every card, so one user waits for the whole chain. With more users, several requests run through a card in one batch, and the weights are only read once. So the cluster does more in total even as each user's share falls.
Batches have a limit on old cards. A batch of 4 on the GTX 1660 SUPERs took 671 ms against about 120 ms for one request, until we split such calls into 3 rows: that brought it to about 211 ms.
Where the time goes¶
For one user, over the direct link: the first PC computes for 112 ms a token, the second for about 172 ms, and the link adds about 5 ms. Compute is what limits this, not the network.
The internet relay confirms it. Sending everything through the coordinator costs one user about 10 %: 2.8 tokens/s against 3.2. With eight users the difference disappears, at 9.9 tokens/s either way, in the earlier run that compares the two.
Where it stops¶
- Long prompts are slow. About 40 ms per prompt token, so a 1,500-token prompt waits a minute for its first token.
- Context is small on small cards. 3,586 tokens fit beside the weights on the two 4 GB cards. Longer conversations are refused.
- One PC is faster for one user, if the model fits it at all. Splitting is for a model no card can hold, or for serving more people.
- One connection is not two homes. Both PCs share a home connection here. Two homes in different cities would add a round trip to every token of a single user.
If your model fits on one card, use a single-machine tool. How it compares says which, and where those tools are simply better.
SilentSwarm is in pre-evaluation. We are looking for companies, institutions and individuals who want to try it on their own machines: Apply for the evaluation.