Work metering (shadow)¶
What the cluster's machines did for whom, measured so that it could be paid for later. Nothing here moves money or names a price. The result is a report for the deployment's admins, in tokens and training steps.
What counts¶
The numbers come from what the leader can see itself, because everything a host reports comes from software the host controls.
| Work | Unit | Counted when |
|---|---|---|
| Serving | Tokens the leader passed on to the caller | As they are delivered. A request that delivered nothing counts for nothing |
| Serving | Prompt tokens | A stage's own figure, because the leader has no tokenizer. Capped by the size of what was sent |
| Training | Steps past a committed snapshot | Every stage has delivered that step, or the job finished |
Busy time is not a unit: it is self-reported, and it pays a slow card more for the
same work. gpu_hours on the profile page is time online, and nothing is based
on it.
Training counts by a high-water mark. A step counts the first time a commit passes it, so steps lost with a node were never counted, and steps trained again after a rollback count for nothing.
Who gets what¶
Each stage's share is its parameters over the placement's, from the leader's own
plan: embeddings on the first stage, norm and head on the last. Without a readable
checkpoint it falls back to layers, and the event's basis says so. A one-stage
placement gets everything. Shares are whole micro-tokens, split by largest remainder,
so they always add up to exactly what was delivered.
The profile page's Tokens served now uses the same split, so every stage's owner sees their share. It used to credit only whoever ran stage 0.
Usage events¶
usage_events is append-only: the store has no update, and SQLite triggers refuse
one. Each row has a key per piece of work (serve:<request id>,
train:<job>:<from>-<to>), so a failover's replay counts once. It keeps counts and
ids, never a prompt or a reply, for SWARM_USAGE_RETENTION_DAYS (400).
Signed receipts¶
With SWARM_WORK_RECEIPTS=on on the leader, models loaded from then on have their
stages hash what their links carry. The setting is off by default, because the hashing
runs on every frame and its effect has not been measured on a GPU cluster. On the
leader's EPYC CPU it took about 21 µs for a decode frame and 3.8 ms for a 2 MB prompt
piece, at each end of a link.
Each stage keeps a rolling hash (BLAKE2b) per session, direction and route, over the
frames' plaintext, above any encryption. Its fellow signs the
result with the node's key and sends it to POST /api/v1/workers/{id}/receipts, where
it is checked against the key the leader pinned. Nothing changes on the wire, so
fellows that cannot do this simply have no receipts to show.
For each request, every edge of the ring is compared: stage N's out chain with stage
N+1's in chain, and the last stage's with stage 0's.
| Status | Meaning |
|---|---|
matched |
Same frame count and same root on every edge |
pending |
Not all reported yet, within two minutes of the request |
no_receipts |
A stage sent nothing in time. Its share of this request is withheld |
frames_lost |
The counts differ: a link lost a frame, which links are allowed to do |
root_differs |
Same count, different bytes: corruption or a false claim |
not_asked |
The model was loaded without receipts |
single_stage |
There are no links to compare |
Chains are per session because one lost frame then costs one request's match, not every later root of the link. Receipts prove a stage took part, and they make audits possible. They create no money: a host's receipts cannot raise what was delivered.
They do not prove a stage computed correctly. A stage can hash garbage it really sent. Catching that takes re-computing samples of the work, which is a later phase.
Training has no receipts. Its proof is the snapshot commit: every stage must deliver the same step, with a checked file.
The shadow report¶
GET /api/v1/admin/work/report?days=14 (admins only; swarm admin work-report, or the
MCP tool admin_work_report):
hosts: per fellow, its tokens and steps,withheld_tokens,reversed_tokensandnot_payable_tokens(see node validation);check.exact: whether the split adds up to exactly what was delivered;receipts: how requests came out, with a reason for each that did not match.
Signals¶
Each host also gets signals, which compare its claims with its other figures:
| Signal | Means |
|---|---|
work_above_measured_speed |
More FLOPs per busy second than twice the card's measured speed |
gpu_idle_while_working |
Its GPUs read under 5 % over at least 20 requests |
token_count_disagrees |
Stage 0's own token count is over 1 % from the leader's |
These come from the host's own software, so they can never decide anything. The next phase may let them raise how often a host is audited. A fast card with stale telemetry can raise one honestly, so read them as a reason to look.
Limits¶
- The figures have not yet been checked against a live GPU cluster.
- A request's share goes to whoever held each stage when it finished, so a stage moved
during a request credits the new host with all of it. Such requests show
replays. - Benchmarks go through the chat path, so their tokens count as delivered, for whoever ran them.
- Whoever operates the leader can still change these tables. Publishing their hashes is a later phase.