What fits on my GPUs?
Enter your machines and see which open language models they can hold: on one card, on one PC or across several. It works in your browser, with no login and no install.
Your machines
For NVIDIA cards on Linux, the only hardware SilentSwarm runs on today. What you enter stays in your browser: it is kept in the page's address after the # sign, which browsers do not send, so you can share the result as a link.
What fits
An estimate for serving with 4-bit weights and the default context of 4,096 tokens, deliberately on the cautious side. Fine-tuning needs more memory. The coordinator checks every real load against the machines it would run on.
What each model size needs
On a single card, with 4-bit weights and the default context. Across several cards, each one also keeps a little memory for itself.
| Model size | Parameters | Memory on one card |
|---|---|---|
| 8B | 8 billion | 9.2 GB |
| 14B | 14 billion | 12.6 GB |
| 32B | 32 billion | 22.3 GB |
| 70B | 70 billion | 41.8 GB |
How fast will it be?
This tool does not predict speed, because that depends on the cards and we only quote what we measured. These are our published numbers for a 14-billion-parameter model on old cards. Newer cards are faster, and we will add them once our measurements are complete.
| Setup | Users | Tokens/s per user | Tokens/s in total |
|---|---|---|---|
| Two PCs: 2x GTX 1050 Ti and 2x GTX 1660 SUPER | 1 | 3.3 | 3.2 |
| Two PCs: 2x GTX 1050 Ti and 2x GTX 1660 SUPER | 8 | 1.6 | 11.6 |
| One PC: 2x GTX 1660 SUPER | 1 | 4.2 | 4.1 |
Those old cards held the model with a context of 3,586 tokens, shorter than the 4,096 this planner assumes, so it does not count them as fitting. The benchmark page has the method and the limits. For more on choosing hardware, see LLM selbst hosten: Was reicht an Hardware? (in German).