Files
Dolon (Agent)andClaude Fable 5.1 2d3c548542 0.2.2: pin num_ctx on every forge-bound request and on the health probe
Ollama keeps one runner config per model and reloads the 64 GB model on
every request whose num_ctx differs from the loaded runner's. The router
now sets options.num_ctx = FORGEWAY_NUM_CTX (default 131072) on the Ollama
dialect and carries the same value in the probe (native /api/chat; the
OpenAI /v1 endpoint ignores options, verified). /status exposes the pin and
the observed runner context and warns on mismatch; check 23 asserts it.

Verified by behaviour: an 8192 caller, a 4096 caller, probes and 40k/100k-
token retrieval prompts in one window -> 0 reloads, 0 timeouts, HA turns
0.3-0.6 s; before, 3 rounds of an 8192 caller = 7 loads and two probe
timeouts. 23/23 acceptance on both clusters.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-18 19:43:13 +00:00

435 lines
23 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# forgeway
An LLM router in front of forge's Ollama `gpt-oss:120b`, speaking **both** the OpenAI
API and Ollama's native API on the same two endpoints. One instance runs **per
cluster**, not one central service.
> **Names (confirmed by Rick, 2026-09-18).** Explicit per-site names
> `forgeway.lounge.falkvinge.net` and `forgeway.serenity.falkvinge.net` (Chalet later:
> `forgeway.chalet.falkvinge.net`), plus the split-horizon local-first alias
> **`forgeway.services.falkvinge.net`**, which resolves to whichever instance is local to
> the querying client's site. `.services.` names are internal-only by convention: no
> entrance SAN, no external exposure, ever.
## The two endpoints
| Endpoint | Behaviour when forge is available | Behaviour when it is not |
|---|---|---|
| `/local-optimistic/v1` | serves from forge | falls back to OpenAI `gpt-5.4-mini` |
| `/local-strict/v1` | serves from forge | clean HTTP **503**, no fallback, by design |
Each endpoint speaks two dialects:
| Dialect | Paths | Typical caller |
|---|---|---|
| OpenAI | `/{endpoint}/v1/chat/completions`, `/{endpoint}/v1/models` | scripts, the OpenAI SDK, OpenClaw |
| Ollama (native) | `/{endpoint}/api/chat`, `/api/generate`, `/api/tags`, `/api/version`, `/api/show`, `/api/ps` | Home Assistant's Ollama integration, any Ollama client |
Streaming works in both: SSE for the OpenAI dialect, newline-delimited JSON for the
Ollama one (where `stream` defaults to **true**, as in Ollama itself).
Where they live:
| Site | Cluster | Use this | Raw LB IP (DNS-independent) |
|---|---|---|---|
| any | — | `https://forgeway.services.falkvinge.net` → the instance local to you | — |
| Lounge | Rocinante | `http://forgeway.lounge.falkvinge.net` | `http://172.19.10.90` |
| Serenity | Niðavellir | `http://forgeway.serenity.falkvinge.net` | `http://172.24.10.21` |
| Chalet | — | when the cluster exists | — |
The alias is the front door. Use the per-site names only when you deliberately want a
*specific* site's instance (for example, asking the Lounge one from Serenity). HTTPS with
a valid certificate is available on the alias at both sites; the per-site names are
`http://` only — the internal ingress certificates carry `*.services.falkvinge.net` but
not `*.lounge.` / `*.serenity.` (adding those SANs to the two *internal* ingress
certificates is a one-line cert-manager change if ever wanted; the entrance bundle is
not the place).
### Calling them
**`local-strict` — do not retry a 503 blindly.** A 503 means the local model is
unavailable *right now*: forge is off, Ollama is stopped, or the model is evicted for
a training run. None of those clear in the next second, and retrying just adds load
to something already struggling. Handle the error, or degrade to your own fallback.
```bash
# strict: one attempt, handle the 503 yourself
curl -sS -X POST https://forgeway.services.falkvinge.net/local-strict/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"gpt-oss:120b","messages":[{"role":"user","content":"hello"}]}' \
-w '\n%{http_code}\n'
```
```python
import httpx
# strict — no retries on 503; that status is an answer, not a hiccup
with httpx.Client(base_url="https://forgeway.services.falkvinge.net/local-strict/v1", timeout=600) as c:
r = c.post("/chat/completions", json={
"model": "gpt-oss:120b",
"messages": [{"role": "user", "content": "hello"}],
})
if r.status_code == 503:
... # local model is down: skip, queue, or use your own fallback
else:
r.raise_for_status()
print(r.json()["choices"][0]["message"]["content"])
# optimistic — fallback is already handled inside the router.
# Do NOT add your own retry loop: a failure here means forge AND the cloud both failed.
with httpx.Client(base_url="https://forgeway.services.falkvinge.net/local-optimistic/v1", timeout=600) as c:
r = c.post("/chat/completions", json={
"model": "gpt-oss:120b",
"messages": [{"role": "user", "content": "hello"}],
})
r.raise_for_status()
print(r.headers["x-forgeway-backend"]) # "forge" or "cloud"
```
With the OpenAI SDK, point `base_url` at one of the two prefixes and use any
non-empty api key (the router does not authenticate callers):
```python
from openai import OpenAI
client = OpenAI(base_url="https://forgeway.services.falkvinge.net/local-optimistic/v1", api_key="unused")
```
Every response carries `x-forgeway-backend: forge|cloud`, `x-forgeway-upstream`,
`x-forgeway-site` and `x-forgeway-mode`, so a caller can always tell what answered.
**Model selection.** The `model` field in a request is ignored — each endpoint routes
to its own backend. Send anything; `gpt-oss:120b` is conventional. `max_tokens` and
`max_completion_tokens` are interchangeable: the router translates between them,
because Ollama only accepts the first and `gpt-5.4-mini` rejects it outright.
### From OpenClaw agents
The same endpoints are registered as model providers (base URL = the `.services.`
alias over HTTPS, so agents get the local instance too), so agents and scripts use one
backend:
```
forgeway-strict/gpt-oss-120b # forge or an error
forgeway-optimistic/gpt-oss-120b # forge, else gpt-5.4-mini
```
## Home Assistant
HA's core *OpenAI Conversation* integration only talks to the official OpenAI endpoint,
so it cannot use forgeway. HA's core **Ollama** integration can: it takes an arbitrary
server URL, and its client keeps the URL path, so an endpoint prefix is a valid server.
**Lounge HA has both privacy flavours, configured and proven 2026-09-18.** Two separate
integration entries on the split-horizon alias, named so that picking one in a dropdown
reads as a data-residency choice rather than a mode name:
| What a dropdown shows (agent) | Entity | Integration entry | Server URL |
|---|---|---|---|
| **Local only — private (fails if forge is down)** | `conversation.forgeway_local_only` | *Forgeway local-strict @ forgeway.services.falkvinge.net — local only, private (fails if forge is down)* | `https://forgeway.services.falkvinge.net/local-strict` |
| **Local preferred — falls back to OpenAI cloud** | `conversation.forgeway_local_preferred` | *Forgeway local-optimistic @ forgeway.services.falkvinge.net — local preferred (falls back to OpenAI cloud)* | `https://forgeway.services.falkvinge.net/local-optimistic` |
Both: model `gpt-oss:120b`, think off, no API key, no device control (the *Control Home
Assistant* box is deliberately unticked — ticking it hands the LLM the Assist tools).
Neither is the default assistant; that is *Settings → Voice assistants → Assist →
Conversation agent*.
Proven by behaviour, through HA's own conversation API, not by "the integration loaded":
| | forge available | forge unavailable (dead-upstream instance, never by stopping Ollama) |
|---|---|---|
| Local only | real answer from forge | HA error, **no answer, no cloud** |
| Local preferred | real answer from forge | real answer in ~1 s, served by gpt-5.4-mini (router log: *falling back to cloud*) |
The unavailable column was produced by publishing a throwaway forgeway instance with a
dead upstream under a temporary `.services.` name, pointing two temporary HA entries at
it, asking through each, and removing all of it afterwards.
**Network prerequisites, verified from HA's side.** `.services.` names are internal-only
by convention: `forgeway.services.falkvinge.net` is NXDOMAIN on public DNS and answers
only on the internal `.201` resolvers, so an entry that *loads* proves HA resolved it
internally. HA lives on the sandbox VLAN, which cannot reach the k8s ingress by default;
**Rick added a pfSense rule** passing HA (`192.168.120.12`) to the ingress `172.19.10.11`
(HA's IP only — the neighbouring scripthost still cannot connect). Both entries loaded
and answered after that; the ingress access log shows HA's own client
(`ollama-python/0.6.2 (aarch64 linux)`) posting to both endpoints.
There is deliberately **no path through the entrance HAProxy any more.** A fenced vhost
(`forgeway.dmz.lounge.falkvinge.net`) existed for a few hours on 2026-09-18 while HA could
not reach the ingress; Rick chose the pfSense rule instead, and a public-edge listener in
front of an unauthenticated service must not survive by accident. The vhost, the DNS
record and the Ingress host are all removed, verified from outside (the hostname now falls
to the edge's default backend).
**What HA sends and how it is handled.** `list()` → `/api/tags` with a 5 s timeout, so
tags never wait on a model load (a cold `gpt-oss:120b` takes ~25–30 s); the list is
exactly one model, the local one. `chat(stream=True)` with `keep_alive: -1`,
`options: {num_ctx: 8192}`, `think: false`, and `tools` when the Assist API is enabled —
all passed to forge untouched. HA reads `message.content`, `message.thinking`,
`message.tool_calls`, `done`, `done_reason`; gpt-oss returns `content` and `thinking`
separately regardless of `think`, so a normal answer is never empty. HA shows the
thinking as a collapsible block.
Managing it headlessly (idempotent; every `setup` ends with a real conversation):
```bash
.venv/bin/python scripts/ha_ollama.py list
.venv/bin/python scripts/ha_ollama.py setup https://forgeway.services.falkvinge.net/local-strict \
--agent "Local only — private (fails if forge is down)" --title "…" --entity-id conversation.forgeway_local_only
.venv/bin/python scripts/ha_ollama.py ask conversation.forgeway_local_preferred "Is the kitchen light on?"
.venv/bin/python scripts/ha_ollama.py delete <url>
```
REST config flows create entries and subentries; the WebSocket API renames entries,
devices and entity ids. HA exposes neither `entry.data` (the URL) nor a subentry rename,
so entry titles carry the URL and mode, and the pre-existing optimistic subentry still
reads *Forge Conversation* in the integration's subentry list (rename it in the UI if it
bothers you; the agent name users see is the device/entity name above).
## One runner config per model: the `num_ctx` pin
Ollama keeps **one loaded runner configuration per model**, keyed on the request's
options. If one caller sends `options.num_ctx 8192` and the next sends nothing (→ the
model default, 131072 for `gpt-oss:120b`), Ollama evicts and fully reloads the 64 GB
model — 15–30 s — on *every* alternation. On 2026-09-18 that was Home Assistant (8192
by default) against the router's own health probes (default): four reloads in two
minutes, 33–66 s per HA turn, and one silent cloud fallback on `local-optimistic`
because a probe queued behind a reload exceeded its 45 s limit.
**forgeway therefore pins `num_ctx` on every forge-bound request** (`FORGEWAY_NUM_CTX`,
default **131072**), overriding whatever the caller sent, and the health probe carries
the same value via Ollama's native `/api/chat`. A caller's own `num_ctx` cannot
re-introduce the thrash; the router logs `pinning num_ctx 8192.0 -> 131072` when it
overrides one. `/status` reports `num_ctx_pin` and `runner_context` (what forge's
loaded runner was last seen with) and the router warns if they ever differ.
Why 131072 and not something smaller: it is the model's trained window and exactly
what an unpinned request loads anyway, so it costs no extra VRAM (KV ≈ 4.6 GB inside a
61 GB resident model on a 97 GB card); the retrieval workload that motivated the box
sends 40k–100k-token prompts; and a large *allocated* context does not slow short
turns — with the pin, HA turns measure 0.3–0.6 s end to end and a 1-token probe 60–100 ms.
One caveat, verified empirically: **Ollama's OpenAI-compatible `/v1` endpoint ignores
`options.num_ctx`**, so OpenAI-dialect callers (OpenClaw, scripts) always get the model
default. The pin must equal that default — it does today — and the `runner_context`
warning is the tripwire if an Ollama upgrade or an `OLLAMA_CONTEXT_LENGTH` setting on
forge ever changes it. Diagnose on forge in one line:
```bash
journalctl -u ollama --since "-20 min" | grep -oE "llama_context: n_ctx += *[0-9]+" | awk '{print $NF}' | tr '\n' ' '
# healthy: nothing, or a single value. thrash: 8192 131072 8192 131072 …
```
`scripts/verify_numctx_pin.py <router>` drives the exact mix that used to thrash (HA-style
turns, off-value scripted callers, probes, 40k and 100k-token needle-in-a-haystack
prompts) and prints latencies; pair it with the one-liner above for the window.
## The Ollama dialect, and the cloud fallback through it
forge *is* Ollama, so on the forge path the router is a raw proxy: the response —
including every streamed NDJSON frame — is forge's own, with only the `model` pinned.
Everything below is about the **cloud fallback** (`local-optimistic` only; `local-strict`
returns Ollama's error shape, `{"error": "..."}`, with status 503):
- The Ollama request is translated to an OpenAI chat completion and the answer is
translated back into Ollama's shape, non-streamed or as NDJSON frames byte-compatible
with Ollama's: `model`, `created_at` (nanosecond RFC 3339), `message.role/content`,
`done`, `done_reason` (`stop` / `length`), and the `*_duration` / `*_count` stats
(token counts from usage; durations that OpenAI does not report are 0).
- **Tool calls** round-trip. Ollama messages carry no tool-call ids and OpenAI insists on
them, so ids are synthesised in order and handed to the following `tool` messages in
order; OpenAI's JSON-string arguments become Ollama's objects. On the way out, OpenAI
streams tool calls as fragments; they are reassembled and emitted as one frame.
- `format: "json"` → `response_format: json_object`; a schema → `json_schema`.
- **`think` → `reasoning_effort`**: `false` → `low`, `true` → `medium`, a level → itself.
With **tools present it is forced to `none`**: gpt-5.4-mini rejects any other value
together with function tools on `/v1/chat/completions`, and HA's Assist API always
sends tools. The cloud model's reasoning text is not exposed by that API, so cloud
answers carry no `thinking`.
- `options.num_predict` → `max_completion_tokens`; `num_ctx` and `keep_alive` are
Ollama-only and dropped.
- `/api/generate` is served the same way (`prompt`/`system`/`images` become messages).
`/api/pull`, `/api/push`, `/api/create`, `/api/delete`, `/api/embed` return 400: this
endpoint serves exactly one model and manages none.
**The reasoning trap, decided.** A thinking model with a tiny output budget spends it all
thinking and returns empty `content` — that is what bit the OpenAI dialect at
`max_tokens: 10`. The router does not silently rewrite budgets; instead it makes the
failure visible: a non-streamed answer with thinking but no content gets the header
`x-forgeway-warning: content-starved-by-reasoning` and a log line. HA never sets a
budget, so it never hits this.
## DNS: how the alias finds the local instance
Every `.services.` name in the estate is a CNAME chain ending at a site's ingress-nginx,
and the authoritative `services.falkvinge.net` zone lives on the Lounge Technitium. So
at the Lounge nothing special is needed:
```
forgeway.services.falkvinge.net → ingress.dmz.lounge.falkvinge.net → 172.19.10.11
forgeway.lounge.falkvinge.net → ingress.dmz.lounge.falkvinge.net → 172.19.10.11
forgeway.serenity.falkvinge.net → forgeway.dmz.serenity.falkvinge.net → ingress.dmz.serenity.falkvinge.net → 172.24.10.11
```
Serenity's Pi-holes forward everything under `falkvinge.net` (except `serenity.`) to the
Lounge — so without an override, a Serenity client would silently get the **Lounge**
ingress and the remote instance, defeating the whole multi-site design. Both Serenity
Pi-holes therefore pin the alias to the Serenity ingress:
| Resolver | Where the override lives |
|---|---|
| `172.24.10.201` (k8s, two pods) | `serenity-cluster-config/apps/pihole/pihole.yaml` → `03-serenity-forwards.conf` |
| `172.24.1.201` (LXC) | Pi-hole v6 API: `dns.hosts` entry + `misc.dnsmasq_lines` |
Both carry `host-record=forgeway.services.falkvinge.net,172.24.10.11` **and**
`local=/forgeway.services.falkvinge.net/`. The `local=` line matters: without it an AAAA
query is still forwarded to the Lounge, and the day the Lounge ingress gains an IPv6
address every dual-stack Serenity client would prefer it and quietly reach the remote
instance — the same dual-stack bypass that defeated the Lounge overrides on 2026-08-24.
Each site's ingress serves `forgeway.services.falkvinge.net` for its own instance, so the
DNS answer alone decides which site you land on. Verified 2026-09-18 from real clients:
Lounge (OpenClaw VM, Lounge Kuma host) → `172.19.10.11`, `x-forgeway-site: lounge`;
Serenity (Niðavellir pod, both Serenity Pi-holes) → `172.24.10.11`,
`x-forgeway-site: serenity`. Never the reverse.
## Why one instance per cluster
A single central router would sit on one site. The cross-site link to forge is exactly
the kind of thing that goes down — and if the router lived on the far side of it, the
router *would be* the outage rather than the thing that reports it. So each cluster
runs its own, and each one answers for itself: when Serenity cannot reach forge,
Serenity's `local-optimistic` still serves from the cloud and its `local-strict` still
returns a clean 503, both without depending on anything at the Lounge.
It does **not** run on forge itself, for the same reason: a router that dies with its
upstream cannot report on it.
## Discovery — and what happens when silicon moves to Serenity
No instance is told which site forge is at. Each one holds an ordered candidate list
(`FORGEWAY_UPSTREAMS`) and finds forge at request time:
```
http://forge.dmz.falkvinge.net:11434 the stable name — DNS follows a host move
http://172.19.0.130:11434 forge at the Lounge (today)
http://172.24.0.130:11434 forge at Serenity (after the move)
```
Candidates are probed concurrently but awaited in priority order, so the happy path
costs one fast round trip and dead candidates never slow it down. The winner is sticky
and tried first next time.
**When silicon relocates to Serenity, nothing here needs rewriting.** Re-point
`forge.dmz.falkvinge.net` at its new address and both instances follow on their next
health refresh (seconds). If DNS is stale or broken, discovery still finds forge at the
Serenity literal already in the list. The only thing that changes materially is which
site pays the cross-site latency — and that flips automatically, because each instance
independently discovers the nearest working answer.
Two things do need a human when that happens:
1. **forge's ufw** — the allow rules are scoped to source ranges. Serenity-local
traffic will then arrive from `172.24.9.0/24` (already allowed); Lounge traffic
will arrive SNAT'd from the Serenity edge and will need its own rule.
2. **Kuma** — the monitors point at each local router by IP, so they keep working, but
the Lounge monitor will start reporting the slower cross-site path.
## Health and monitoring
`/healthz` performs a **real 1-token completion** against forge, never a ping. A ping
cannot tell the difference between a healthy forge and a forge that is up with Ollama
stopped — which was literally its state on the morning this was built.
| Result | Meaning |
|---|---|
| `200` | forge answered a real completion |
| `503` | forge did not |
| no response at all | the router itself is gone |
Verdicts are cached for a few seconds (`FORGEWAY_HEALTH_TTL`), so a burst of traffic
does not become a burst of probe completions against forge.
`/healthz/live` and `/healthz/ready` are **deliberately not gated on forge**. If
readiness were, Kubernetes would pull every pod out of the Service the moment forge
died — callers would get a connection failure instead of the 503 `local-strict`
promises, and Kuma could not distinguish "forge down" from "router gone".
Monitoring is **active HTTP probing, not push**: a push heartbeat cannot tell "router
dead" from "idle". One monitor per Kuma instance, each watching its own local router:
| Kuma | Monitor | Target |
|---|---|---|
| Lounge `192.168.81.98:3001` | #307 `forgeway — local LLM router (Lounge)` | `http://172.19.10.90/healthz` |
| Serenity `192.168.24.98:3001` | #57 `forgeway — local LLM router (Serenity)` | `http://172.24.10.21/healthz` |
Both: 60 s interval, 48 s timeout, 2 retries, accepted status `200-299`. The timeout is
wide because a **cold `gpt-oss:120b` load takes ~25–30 s** and is a legitimate success,
not a failure.
### The one operational conflict: monitoring vs. LoRA training
A 60 s probe interval keeps `gpt-oss:120b` (~67 GB) permanently resident on forge's
card 1. An ai-toolkit LoRA run needs ~88 GB on the same card, and the two do not fit —
`ollama stop gpt-oss:120b` is the documented pre-training step, but with monitoring
running, the next probe simply loads it again.
Before a training run, do one of:
- **pause the Kuma monitor** for that site (simplest), or
- set `FORGEWAY_HEALTH_COLD_LOAD=false` in the ConfigMap. The probe then checks
Ollama's `/api/ps` and reports DOWN instead of triggering a load. During training
that reading is *correct*: `local-strict` genuinely cannot serve, and
`local-optimistic` correctly falls through to the cloud.
## Configuration
Everything is environment-driven; one image serves every site. See
`apps/forgeway/manifests/configmap.yaml` in each cluster repo.
| Variable | Default | Notes |
|---|---|---|
| `FORGEWAY_NAME` | `forgeway` | the single place the service's name lives |
| `FORGEWAY_SITE` | `unknown` | `lounge` / `serenity` |
| `FORGEWAY_UPSTREAMS` | forge DNS name | comma-separated, priority order |
| `FORGEWAY_LOCAL_MODEL` | `gpt-oss:120b` | |
| `FORGEWAY_CLOUD_MODEL` | `gpt-5.4-mini` | optimistic only |
| `FORGEWAY_HEALTH_TTL` | `5` | cache for a healthy verdict |
| `FORGEWAY_HEALTH_FAIL_TTL` | `3` | cache for an unhealthy verdict |
| `FORGEWAY_HEALTH_TIMEOUT` | `45` | must exceed a ~25–30 s cold load |
| `FORGEWAY_DISCOVERY_TIMEOUT` | `4` | `6` at Serenity — the mesh hop |
| `FORGEWAY_HEALTH_COLD_LOAD` | `true` | see the training conflict above |
| `FORGEWAY_NUM_CTX` | `131072` | pinned on every forge-bound request and the probe; `0` disables |
| `FORGEWAY_REQUEST_TIMEOUT` | `600` | proxied completions |
| `OPENAI_API_KEY` | — | from the `forgeway-cloud` Secret, applied out of band |
The cloud credential is **never in git**. It is applied with
`scripts/apply_cloud_secret.py`, which reads the value from the process environment and
hands it to kubectl on stdin. The `secretKeyRef` is `optional: true`, so the Deployment
still runs without it — `local-strict` is unaffected and `local-optimistic` reports
that no fallback credential is configured.
## Deploying
```bash
./scripts/build-push.sh 0.1.1 # multi-arch; both clusters are arm64-majority
```
Then bump the image tag in each cluster repo and push — ArgoCD syncs:
- Lounge: `lounge-cluster-config/apps/forgeway/manifests/` + `base/apps/forgeway/`
- Serenity: `serenity-cluster-config/apps/forgeway/` + `base/apps/forgeway-app.yaml`
## Verifying
```bash
./scripts/local-smoke.sh # OpenAI dialect, against a real forge
./scripts/local-smoke-ollama.sh # Ollama dialect, incl. the exact client HA pins
./scripts/verify-cluster.sh ~/.kube/rocinante-admin.yaml lounge
./scripts/verify-cluster.sh ~/.kube/nidavellir.yaml serenity
```
`verify-cluster.sh` runs 23 checks from inside the cluster (13 OpenAI, 9 Ollama, 1 pin-vs-runner). It produces the
"forge unavailable" case with a throwaway Deployment pointed at a dead upstream —
**it never stops Ollama on forge.**