Skip to content

Inference — /ops/inference

Load a trained model onto the cluster and chat with it, or drive it over an OpenAI‑compatible API. Scoped to the pool selected in the sidebar.

Inference page

Model (left panel)

  • Model dropdown — every model trained and exported in this pool. Each shows its base architecture, compression method, and training loss. Incomplete models (missing a shard from one node) are greyed out.
  • Load / Unload — pick a model (and the node to serve it on) and Load; a progress bar tracks download → extract → load‑into‑VRAM. Unload frees the VRAM. The ↻ button refreshes the list.
  • Status line — under the buttons, once a load is under way or done: the overall state plus one entry per pipeline stage (Stage 0: gpu-a100-01 · L0-18 · 3100 MB), so you can see which machine holds which layers.

A load can be refused before it starts, with a message like stage 0 on gpu-1 needs 15.2 GB (9.0 GB weights + 4.8 GB KV cache at 8192 tokens), has 12.0 GB free. That is the leader budgeting the KV cache the stage will allocate on top of its weights - see Serving VRAM. Serving on a node with more free VRAM, or lowering SWARM_SERVE_CONTEXT_TOKENS, makes it fit.

Generation

  • Temperature — sampling randomness (0 = deterministic).
  • Max tokens — response length cap.

API

Everything you need to call the model programmatically:

  • Base URL — the OpenAI‑compatible endpoint.
  • Token picker / + Token — create or paste an API token. Tokens are stored only in your browser and included in the copyable curl.
  • Copy & run — a ready‑made curl against /v1/chat/completions.

Chat

The main panel is a chat transcript. Type a prompt and Send (Enter to send, Shift+Enter for a newline); the model streams tokens back. Responses render Markdown, code, diagrams, and math. Clear resets the conversation. Each reply notes which model answered.