Node validation¶
How the cluster checks that a machine did the work it was given. It rests on work metering, and it moves no money.
SWARM_AUDITS is off by default. shadow runs everything and punishes nobody.
enforce also quarantines a node after a confirmed failure. Don't turn on enforce
before the tolerances have been measured on real cards (see below).
What a host can do, and what answers it¶
| Attack | Answer |
|---|---|
| Skips computing, hands on its input, returns garbage | Replay audits |
| Runs cheaper weights or a cheaper dtype | Replay audits, within the tolerance below |
| Behaves only when it detects a test | Audits replay real traffic, drawn after the reply has gone |
| Two machines of one owner vouch for each other | Audits by another owner, and a raised rate on such pairs |
| Farms by sending itself requests | That work counts as not payable |
| A broken card returning wrong numbers | Re-tests at random times |
Cheating cannot inflate pay, because pay would come from the tokens the leader delivered. These checks are about work that was not really done.
Replay audits¶
A finished request is drawn with a chance the host cannot see or predict:
| Node | Chance per request, per stage |
|---|---|
| On probation | 15 % (SWARM_AUDIT_RATE_PROBATION) |
| Established | 1.5 % (SWARM_AUDIT_RATE_ESTABLISHED) |
| Raised: same owner on the ring, a signal, or a recent failure | 50 % (SWARM_AUDIT_RATE_RAISED) |
For a drawn stage:
- Receipts first. Both neighbours must have signed matching roots.
- Frames from its neighbours. Stages keep recent frames in RAM
(
SWARM_AUDIT_BUFFER_MB, 256). The audited stage is never asked for them, and they must hash to the signed roots. - A verifier replays the whole session from an empty cache: a hot
cover of another owner, or the reference node
(
SWARM_AUDIT_REFERENCE_FELLOW). It never sees the claim, and it sends back what it computed, so it cannot simply agree. - The leader compares, per position, and the worst one counts.
| Outcome | Means |
|---|---|
passed |
They agree. The frames are deleted at once |
disagreed |
A verifier disagrees, and the reference node has not decided yet |
cleared |
The reference sided with the host |
suspect |
The reference node disagreed, with no third party. One more failing session decides, and the audit rate is raised meanwhile |
confirmed_failed |
Confirmed by the reference node |
abandoned |
It could not be carried out, which says nothing against the host |
One disagreement never punishes anyone, because honest hardware does fail. Without a
hot cover of another owner or a reference node, audits end abandoned, with "no
verifier holds this stage" as the reason.
What cannot be audited yet: a session going on from a kept cache, a replayed one, or a drafting one, since their frames alone do not reproduce them. Training is not audited. Its proof is still the snapshot commit.
The last stage is checked less strictly¶
A last stage sends sampled tokens, so there is nothing to compare exactly. The verifier
works out what honest sampling under the request's own temperature and top-k would score,
and the leader scores the tokens claimed against that. On a real 0.6B model
(exp/spikes/audit_tokens.py, 6 sessions of 24 tokens each):
| Run | Passed |
|---|---|
| Honest: greedy, and temperature 0.7, 1.0, 1.5 and 2.0 | 6/6 each |
| Random tokens, asked for greedy or 0.7 | 0/6 |
| Second-best token, asked for greedy | 0/6 |
| Half the layers on other weights, asked for 0.7 or 1.0 | 0/6 |
| Half the layers on other weights, asked for greedy | 1/6 |
| Random tokens, asked for 1.5 | 5/6 |
The higher the temperature, the less this can see: a sampler that hot is close to random itself. Subtly cheaper weights can also pass, and replies under 8 tokens decide nothing. Hidden states, which every other stage is judged on, do not have these limits.
Tolerances¶
From exp/spikes/audit_tolerance.py: a middle stage of a real 0.6B model, per position,
on CPU only.
| Frames | Honest worst | Weakest cheat that saves work | Tolerance |
|---|---|---|---|
| float32 | 1.4e-6 | 8.1e-3 (bf16 instead) | 1e-3 |
| bfloat16 | 1.9e-2 | 5.3e-2 (8-bit weights) | 3e-2 |
| float16 | 2.3e-3 | 5.1e-2 (8-bit weights) | 3e-2 |
Skipped layers, 4-bit weights, stale caches and passthrough all land above 0.3. A 0.1 % noise on the weights does not separate from honest bf16, but it saves nothing either. Compare only runs of the same dtype and quantization.
Different cards, drivers and GPU against CPU are not in these numbers. To measure
them, run the spike on two cards with the same inputs, then set
SWARM_AUDIT_TOLERANCES (JSON per dtype):
python exp/spikes/audit_tolerance.py --device cuda --dtype bfloat16 --inputs in.npz --dump a.npz
python exp/spikes/audit_tolerance.py --compare a.npz b.npz
Other checks¶
- Re-tests. The admission self-test's known-answer probe runs again at random times
(
SWARM_AUDIT_RETEST_HOURS, about daily), with a fresh seed, on idle nodes. A wrong answer raises the node's audit rate. - Card changes. Fellows report each GPU's UUID. A change raises the audit rate. The UUID is self-reported, so it is a signal and proves nothing.
- Weight challenges. A node hashes random byte ranges of its stage's files with a
fresh nonce, and the answer is compared with the leader's copy and any cover's
(
SWARM_AUDIT_CHALLENGE_HOURS). This shows it holds the bytes, not that it computes with them. Weights loaded from a Hugging Face cache are outside the stage directory, so a stage made only of those has nothing to compare. - Canaries (
SWARM_AUDIT_CANARIES, off). A machine with no traffic has nothing to audit, so the leader sends it a synthetic request and audits that. They count as nobody's usage. A host that recognises one can pass it, so these are a second line.
Trust and consequences¶
Nodes start on probation and become established after 20 passed audits and 7 days.
A confirmed failure always records the failure, raises the rate for all of that owner's
nodes for 30 days, and shows the work done in the
SWARM_EARNINGS_MATURITY_DAYS (14) before it as reversed_tokens in the shadow report.
Under enforce it also quarantines the node. Earnings and refunds do not exist yet.
An owner sees their machines' audits with GET /api/v1/me/audits and can appeal a failure
(POST /api/v1/me/audits/{id}/appeal). An admin reads the evidence and decides
(swarm admin audits show, swarm admin audits resolve). Overturning restores the node.
What is not payable¶
The shadow report's not_payable_tokens separates work that was done but could not earn:
| Reason | Means |
|---|---|
own_use |
The consumer owns the machine |
org_use |
The consumer and the machine's owner share an organization |
private_pool |
The pool is private, so it keeps its accounting to itself |
over_probation_limit |
A node on probation passed SWARM_PROBATION_DAILY_TOKENS (200,000) in a day |
What an audit shows to whom¶
An audit shows a request's activations to one more machine, and an audit of stage 0 shows the prompt too. Encryption between machines does not change that, because a verifier has to compute on them. So:
- frames stay in RAM on fellows, and are overwritten as new work comes in;
- on the leader they are deleted as soon as a replay agrees. Disputed ones stay for
SWARM_AUDIT_EVIDENCE_DAYS(14); - a stage 0 audit's prompt stays in the leader's memory only, so a restart abandons it;
- usage events and audit rows hold counts, hashes and errors, never text.
Turning audits on therefore changes what your data note must say. Tell your users first.