The public web surface¶

Everything a leader serves to someone who is not logged in: the landing page, the documentation you are reading, the files crawlers and security researchers look for, and the first-party numbers that say whether any of it worked.

This page is about a public deployment. A leader on a trusted LAN serves the same files; they just have no audience.

What is public¶

Path	Served by	Purpose
`/`, `/de`	leader	Landing page, English and German (`/en` → 301 → `/`)
`/docs/`	frontend (nginx)	This MkDocs site, baked into the frontend image
`/robots.txt`	leader	Crawl policy; advertises both sitemaps
`/sitemap.xml`	leader	The app's public pages, with hreflang alternates
`/docs/sitemap.xml`	frontend	Generated by MkDocs from `nav`
`/llms.txt`	leader	Short brief for language models (llmstxt.org)
`/llms-full.txt`	leader	The documentation as one Markdown file
`/.well-known/security.txt`	leader	Vulnerability reporting contact (RFC 9116)
`/legal/*`	leader	Imprint, privacy, accessibility, Leichte Sprache
`/docs/blog/`	frontend	Notes - the project blog, with an RSS feed
`/docs/feed_rss_created.xml`	frontend	RSS for the blog
`/docs/compare/`	frontend	How SilentSwarm compares to the alternatives
`/api/v1/public/stats`	leader	Aggregate community numbers for the landing page

Everything else - the dashboard, pools, buckets, the admin pages, the whole API - is Disallowed in robots.txt and gated by route_policy.py. The Disallow list is not access control; it keeps pages out of search results that users should not reach through Google in the first place.

The docs are public¶

They used to sit behind a soft PIN cookie (docs_pin) on the frontend. That gated nothing sensitive - the source is published - but it answered every crawler with a 302, so the reference corpus was invisible to search engines and to the LLM crawlers that /llms.txt points at /docs/.

The gate is gone. /docs-gate.html now permanently redirects to /docs/ so links that were shared while it existed still work.

Canonical URLs are a build-time decision¶

MkDocs writes absolute URLs into every page's <link rel="canonical"> and into sitemap.xml, from site_url. That value is baked in when the frontend image is built, not when it starts:

# docker-compose.yml, service: frontend
build:
  args:
    SILENTSWARM_DOCS_URL: https://${SILENTSWARM_DOMAIN:-silentswarm.nexpatch.ai}/docs/

Set SILENTSWARM_DOMAIN in .env before building the frontend. An image built with the wrong domain publishes someone else's canonical links, which is the one SEO mistake that is hard to undo: search engines will happily drop your pages in favour of the URL you told them was authoritative.

docker compose build frontend      # re-bake after changing SILENTSWARM_DOMAIN

overrides/main.html adds OpenGraph and Twitter tags to every docs page, so a link pasted into Slack, Mastodon or a chat shows a title, a description and the project image rather than a bare URL. Material's social plugin does the same thing but pulls in Cairo and Pango, which the docs image and CI would both have to grow; the template costs nothing.

A page can override the description with front matter:

---
description: One sentence, 150-160 characters, describing this page.
---

Without it the site-wide site_description is used.

What a model reads¶

People increasingly ask a model before they ask a search engine, and a model summarising this project from the marketing page alone tends to invent the economics. Two files exist so it does not have to guess, both following the llmstxt.org convention. Their text lives in leader/web/llms_txt.py.

`/llms.txt` - the brief¶

A page of Markdown: what SilentSwarm is, where the important pages are, and an "Accurate as of this page" section that states plainly what the marketing copy could be read to imply:

contributing a machine does not pay money today;
the worker is not sandboxed;
the licence is free for noncommercial use and is not OSI open source;
the project is in open beta and accounts are approved by an admin.

That section is the reason the file exists. Keep it, and keep it first.

`/llms-full.txt` - the documentation, concatenated¶

Eleven documentation pages as one Markdown document (~80 KB), so a model that wants the detail reads it in one request instead of crawling and stitching twenty HTML pages. The honesty section is repeated at the top: a reader that gets only the first screen still has to come away with it.

Three things happen to each page on the way in:

its own # Title is dropped, because the section heading above it already says the same thing;
remaining headings drop one level, so the result is one document with one # rather than eleven competing ones;
relative links to other .md files become absolute documentation URLs - a model following ../03_operations/auth.md out of a concatenated file has nowhere to go.

The page list is curated, not globbed. PAGES in llms_txt.py names each file. docs/planning/ is internal, and a glob over docs/ would publish it the moment somebody built an image from a working tree - so docs/planning/ is in .dockerignore as well, and a test asserts every listed page exists and appears in the published MkDocs nav. Adding a page is a deliberate edit.

A deployment without the docs tree (a wheel-only install) serves a short body pointing at /docs/ rather than a 404 on a URL that /llms.txt advertises.

Crawl policy for AI agents¶

robots.txt names GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended and CCBot, and allows them. User-agent: * already permitted them, so nothing a crawler does changes - it is a statement that being read by a model is intended.

There is a trap in doing this by hand. A crawler applies the group that names it and ignores the wildcard entirely, so an agent named without repeating the Disallow lines is an agent invited into /admin/. Every group is therefore generated from one _ROBOTS_RULES list in app.py, and a test parses the served file and fails if any group has drifted from the wildcard's rules.

To refuse an AI crawler, give it its own group with Disallow: / - do not just remove it from the list, which only returns it to the permissive wildcard.

Registering with search engines¶

Sitemaps are discovered from robots.txt, but a property in Google Search Console and Bing Webmaster Tools is the only place the ranking data lives. Both accept a meta tag as proof of ownership; both are read from the environment, so claiming a property is a compose edit rather than a commit with a token in it:

SILENTSWARM_GOOGLE_SITE_VERIFICATION=<token from Search Console>
SILENTSWARM_BING_SITE_VERIFICATION=<token from Bing Webmaster Tools>

Restart the leader (not the frontend) and the tags appear in the landing page's <head>. DNS TXT verification works too and is preferable if you own the zone - it covers the whole domain rather than one page.

Submit both sitemaps once verified:

https://<domain>/sitemap.xml
https://<domain>/docs/sitemap.xml

What the landing page says to a search engine¶

The <title> and meta description are written for someone typing a problem into a search box; the OpenGraph and Twitter titles keep the softer community wording, because a link pasted into a chat is not a search result. Both are budgeted - about 60 characters for the title, 160 for the description - and a unit test fails when either overruns in either language. German runs roughly 15% longer than English, so the German versions are written to the budget rather than translated into it.

Structured data¶

The landing page carries four JSON-LD types:

Type	What it is for
`Organization`	NexPatch AI UG, with an `@id` the other nodes reference
`WebSite`	The site itself, in both languages
`SoftwareApplication`	What SilentSwarm is, and its licence terms
`FAQPage`	The questions the trust section answers

Two things about it are deliberate:

The SoftwareApplication description carries the vocabulary the page does not. The reader-facing copy avoids "pipeline stage", "activations", "NAT" and "VRAM" on purpose - a visitor should not need to know what a pipeline stage is, and a unit test enforces it. But a search engine that never sees the word "pipeline" cannot rank the page for it, and a model summarising the project needs the mechanism named. Structured data is where those words cost the reader nothing, and it says exactly what the docs say.

The FAQ entries are extracted from the template, not restated. Structured FAQ data that does not match the visible page is a lie told to a crawler, so _faq_entries() in leader/web/auth_ui.py reads the trust section's <dt>/<dd> pairs out of landing.html. A copy edit cannot leave the markup describing answers that are no longer on the page, and a test compares the two sets. The German rendering comes for free: the localiser rewrites the JSON block along with the prose, because both carry the same English source strings.

Docs pages carry BreadcrumbList instead (overrides/main.html), two levels deep - the docs home and the page. The nav's sections are headings rather than pages and have no URL of their own; inventing one by pointing at a section's first child would tell a search engine that a page is something it is not.

The audit in CI¶

scripts/frontend_smoke.mjs runs Lighthouse over / and /de when the lighthouse package is installed, and fails on an SEO score below 0.95. Performance and best-practices are printed, not gated:

performance on a shared CI runner varies by more than any regression worth catching;
best-practices counts the 401s that anonymous identity bootstrap legitimately logs to the console.

The audit runs before the smoke test logs in. With a session cookie, / redirects to the dashboard, and the audit would be scoring an app page that is deliberately not indexed at all - which reads as "page is blocked from indexing" and looks like a real failure.

Publishing¶

The blog¶

Material's blog plugin, with posts in docs/blog/posts/. A post needs front matter with a date, a category from the allowed list in mkdocs.yml, an author from docs/blog/.authors.yml, a description (it becomes the meta description and the RSS summary), and a  marker for the excerpt - post_excerpt: required, so a post without one fails the build rather than showing the whole article on the index.

The RSS feed comes from mkdocs-rss-plugin and takes its dates from front matter, not git. That is not a style preference - it is what makes the docs buildable in the frontend image at all:

the image builds from a context with no .git directory (see .dockerignore), so a git-derived date would silently become the build date for every post - hence use_git: false;
the plugin pulls in GitPython, which raises at import time when there is no git binary on the image, before any config is read. use_git: false alone does not help there, so the docs stage sets GIT_PYTHON_REFRESH=quiet, which is the documented escape hatch.

If a docs build starts failing with Bad git executable, that pair is what came undone.

Adding the plugin block to mkdocs.yml means search has to be listed explicitly - naming any plugin replaces MkDocs' default set, and leaving it out silently removes the docs search box.

Comparison and benchmarks¶

Two pages exist because they answer the two questions this audience arrives with, and both are written to be believed rather than to win:

How SilentSwarm compares names the cases where Petals, Exo, GPUStack, Ray or a plain single-node stack is the better answer, and carries a table of what is not built yet. It deliberately publishes no throughput comparison against other projects: we have not run them on identical hardware, and a table assembled from other people's published numbers is marketing.
Benchmarks is methodology with an explicitly empty results table. The rules on it - state the link, report distributions not means, publish the unflattering configuration, keep the quality cost next to the compression ratio - are the reason a future number there will mean something.

CHANGELOG.md records what shipped, including a "Known limitations" section, so the honest account is in the repository as well as on the site.

Performance notes¶

Compression is done at the edge, in docker/nginx/frontend.conf, not in the leader. The pages inline their CSS and JS, so there is a lot of compressible text in each response - about 104 KiB on the landing page. It is not GZipMiddleware on the leader because Starlette's implementation writes into a GzipFile without flushing per chunk: SSE frames would sit in the compressor's buffer and the training metrics stream would stop being live. text/event-stream is likewise absent from gzip_types for the same reason.

Two measurements worth recording, so they are not re-derived:

The hero canvas is not the LCP element. The largest contentful paint is the <p class="lead"> text. Deferring the rotating node-cloud animation until after load was tried and measured: LCP 3493 ms before, 3502 ms after, total blocking time 0 ms in both. The animation yields between frames and never creates a long task, so the change was reverted rather than shipped on a hypothesis. prefers-reduced-motion is honoured already, and the hero carries a visible pause control.
og-image.png is not in the render path. It is referenced only from <meta> tags, so it is fetched by social scrapers and never by a browser rendering the page. 104 KB of PNG costs a visitor nothing, and PNG is the format every scraper handles - converting it to WebP would trade support for bytes nobody waits on.

The fonts are self-hosted¶

Space Grotesk, Inter and JetBrains Mono are vendored by scripts/vendor_frontend.py - pinned, SHA-256-checked @fontsource tarballs, extracted into templates/vendor/fonts/ and served same-origin from /vendor/fonts/. They used to come from fonts.googleapis.com, which was wrong twice over:

the stylesheet blocked the first paint on every page (Lighthouse scored it as ~660 ms of render-blocking work; that audit now passes);
hotlinking it sent every visitor's IP address to Google, which a court has held to be an unlawful transfer for a German site (LG München I, 20.01.2022, 3 O 17493/20).

Only the weights the stylesheets ask for, and only the latin and latin-ext subsets - German needs nothing beyond latin (ä, ö, ü and ß all live under U+00C0-00FF), and latin-ext is there for names that do. Twenty files, ~390 KB on disk; a visitor downloads the two or three faces their page actually renders, because each @font-face carries the unicode-range from the upstream package.

The generated vendor/fonts.css is inlined with the rest of the CSS rather than linked - a linked stylesheet is one more render-blocking request for about three kilobytes. Two faces are preloaded in the <head>: the body face the largest paint is made of, and the display face the h1 is set in. Font preloads need crossorigin even same-origin, or the browser fetches the file twice.

Consequences worth knowing:

The CSP now has no external origin at all - font-src 'self' data:, style-src 'self' 'unsafe-inline', and a test fails if https:// appears in the policy.
scripts/frontend_smoke.mjs allows no third-party hosts. The guarantee that nothing leaves the origin is now literally true, and enforced per page.
The docs use the same files: theme.font is false in mkdocs.yml (so Material emits no stylesheet of its own) and docs/assets/extra.css declares the faces against /vendor/fonts/.... The docs are served under /docs/ on the same origin, so there is one copy and one cache. A standalone mkdocs serve has no /vendor/ and falls back to the system stack.
A fresh checkout that has not run the vendor script renders in system fonts. Every family stack ends in a real fallback, so that is a plainer page rather than a broken one - and a test asserts it.

The language a visitor gets¶

/ and /de are two URLs with their own canonical and hreflang links, so / redirects (302) a browser that asks for German rather than answering the same URL with different content - which would make both of those links a lie.

The signal is the browser's Accept-Language header, not a geo-IP lookup. Where someone is says much less about what they read than what they configured, and looking it up would mean processing an IP address to infer something about a person for a worse answer. An explicit choice always wins: negotiate_lang() reads the swarm_lang cookie before it reads the header, so the switcher works for a German browser instead of bouncing it back on every click.

Both language URLs send Vary: Accept-Language, Cookie. Without it a proxy can hand the English page to a German visitor, or hand everyone the redirect.

The cookies this site sets¶

Cookie	When	Lifetime	Purpose
`swarm_lang`	only when a visitor picks a language (`/lang/{code}`)	1 year	Remembers that choice
`swarm_csrf`	on pages that can submit a form	14 days	Double-submit CSRF token
`swarm_session`	on login	session TTL	Authentication (`HttpOnly`)
`swarm_pool`	when a signed-in member switches pool	1 year	Which pool the consoles act on

There is no analytics cookie and no third-party cookie. The landing-view counter is server-side and aggregate (see below), the fonts and UI libraries are served from this origin, and the Content-Security-Policy allows no external origin at all.

Consent under § 25 TDDDG (the German implementation of the ePrivacy Directive) is not required for storage that is strictly necessary to provide a service the user explicitly requested. Session, CSRF and active-pool are plainly that. A language preference is the textbook exempt case - the EDPB names "user-interface customisation" cookies explicitly - provided it is stored as a result of the user's own choice.

That proviso is why viewing a page now stores nothing. The landing page used to write swarm_lang on every render, before anyone had chosen anything, with a one-year lifetime; that was the single piece of storage here that would have been hard to defend, and it bought nothing Accept-Language does not already answer. A test asserts the whole public entry point - /, /de, the legal pages, robots.txt - sets no cookie at all.

Adding a banner anyway would be worse than useless: it would ask for consent that is not needed, train people to click through one, and lower the conversion of the page it covers. What a public deployment does need instead is a privacy notice (Art. 13 GDPR) and, in Germany, a provider identification (§ 5 DDG) - see SWARM_PRIVACY_FILE and SWARM_IMPRINT_FILE in Users and authentication. Those are served as placeholders until an operator configures them, and a placeholder is not a privacy policy.

Reading the funnel¶

The landing page loads no third-party script, so there is no analytics vendor to ask how many people saw it. The numbers are first-party and aggregate:

curl -H "Authorization: Bearer $ADMIN_TOKEN" \
  https://<domain>/api/v1/admin/funnel?days=30

{
  "days": 30,
  "landing_views": {"human": 4120, "bot": 980, "pages": [...], "window_days": 30},
  "waitlist_total": 63,
  "waitlist_by_source": {"landing": 58, "earnings": 5},
  "signups_by_intent": {"host": 21, "use": 34, "unknown": 2},
  "signups_total": 57,
  "nodes_registered": 9,
  "nodes_ready": 7,
  "daily": [{"day": "2026-09-01", "human": 210, "bot": 44}]
}

Four rungs, in order: saw the page → left an email → finished a registration → a machine actually joined. The last one is the only number that means the project works; everything above it is a ratio that tells you which step is leaking.

landing_views is windowed by days; every other total is lifetime. The field names say so because dividing a 30-day numerator by a lifetime denominator is the easiest way to talk yourself into a bad decision.

What is stored¶

One row per (UTC day, path, language) with two integers - human views and crawler views. No IP address, no cookie, no user-agent string, no session identifier. Nothing that identifies a visitor, so this needs no consent banner and adds nothing to a data-subject request. Counts are buffered in memory for up to 30 seconds and flushed on shutdown, so a launch peak is a write every few seconds rather than one per page render.

Crawler traffic is counted separately rather than dropped: during a launch, the bot share is the difference between "the post did nothing" and "the post worked and half of it was Bingbot". Classification is by User-Agent and deliberately errs towards calling things crawlers, so the human number is a floor.

Reporting a vulnerability¶

/.well-known/security.txt carries the contact, the policy link and an Expires date computed 180 days ahead on every request - a security.txt that expired last year reads as an abandoned project, which is the opposite of the signal it exists to send. The same content lives in SECURITY.md for people who found the repository rather than the deployment.

After a redeploy¶

Docs changes ship in the frontend image; app and landing changes ship in the leader image. Rebuilding only the leader after editing docs/ ships nothing. Check both containers are up - a stopped frontend surfaces as Bad Gateway even while the leader is perfectly healthy:

docker compose ps                                   # both must be running
curl -sI https://<domain>/docs/ | head -1           # expect 200, not 302
curl -s  https://<domain>/robots.txt | grep Sitemap # expect two lines
curl -s  https://<domain>/.well-known/security.txt | head -1