# forgeway — build report, 2026-09-18 Two OpenAI-compatible query endpoints over forge's Ollama `gpt-oss:120b`, deployed as one instance per cluster on Lounge (Rocinante) and Serenity (Niðavellir). Working, in GitOps, monitored, and reachable by IP. **Update, later the same day:** Rick confirmed the name; DNS, ingress and the split-horizon alias are live and verified — see the addendum at the end of this file. | | Lounge | Serenity | |---|---|---| | Address | `http://172.19.10.90` | `http://172.24.10.21` | | Replicas | 2/2 | 2/2 | | Kuma monitor | #307, `200 - OK` | #57, `200 - OK` | | Real completion latency | 54 ms warm | 383 ms warm (mesh hop) | | Acceptance checks | 13/13 | 13/13 | Image `git.falkvinge.net/rick/forgeway:0.1.1`, linux/amd64 + linux/arm64 (both clusters are arm64-majority: Lounge 12/15, Nidavellir 9/12), anonymously pullable from the Gitea registry, so no imagePullSecret is needed. --- ## Three blockers found on forge, and what was changed None of these were visible from the brief; all were found by probing. **1. Ollama was bound to `127.0.0.1` only.** Nothing off-box could reach it. Added a *new* systemd drop-in `/etc/systemd/system/ollama.service.d/20-listen.conf` with `OLLAMA_HOST=0.0.0.0:11434`. The existing `10-gpu-pinning.conf` was not touched, and the pinning was re-verified after the restart: `llama_prepare_model_devices: using device CUDA0 ... (0000:02:00.0)` — still card 1, as intended. **2. forge's ufw denied both clusters.** Default deny-incoming with a single allow for the OpenClaw VM. Added three port-scoped rules: ``` 11434/tcp ALLOW IN 172.19.9.0/24 Lounge k8s nodes 11434/tcp ALLOW IN 172.24.9.0/24 Nidavellir k8s nodes 11434/tcp ALLOW IN 172.19.0.1 Lounge pfSense SNAT, cross-site Serenity traffic ``` **3. Serenity traffic arrives SNAT'd.** Serenity pods reach forge with source `172.19.0.1` — the Lounge pfSense — not their real `172.24.9.x`. Confirmed in forge's `UFW BLOCK` log. The `172.24.9.0/24` rule alone was therefore never going to match. > ⚠️ **Security note, for Rick's judgement.** Rule 3 is wider than I would like: it > allows anything pfSense forwards out of the Tailscale mesh to reach Ollama, which has > no authentication. It is scoped to port 11434, but it is not scoped to Serenity. > The clean fix is a pfSense outbound-NAT exemption so tailscale→DMZ traffic keeps its > original source (this matches the documented `NoSNAT` intent already set on the > Serenity UDM), after which rule 3 can be deleted and `172.24.9.0/24` does the work. > That is an edge-firewall change on your core router, so I did not make it. --- ## Design decisions worth knowing **Discovery, not configuration.** Each instance holds an ordered candidate list — the stable DNS name `forge.dmz.falkvinge.net`, then the Lounge literal `172.19.0.130`, then the future Serenity literal `172.24.0.130`. Candidates are probed concurrently but awaited in priority order, so the happy path is one fast round trip and dead candidates never add latency. When silicon moves, re-pointing DNS is enough; if DNS is stale, the Serenity literal already in the list catches it. Neither instance assumes a site. **Readiness is not gated on forge.** This looks wrong at first glance and is the single most important choice here. If readiness depended on forge, Kubernetes would evict every pod from the Service during a forge outage — callers would get a connection refusal instead of the 503 `local-strict` promises, and Kuma would see "no response" (router gone) instead of 503 (forge down). Forge health is exposed separately on `/healthz`; liveness and readiness only answer for the router process. **Health is a real completion.** `/healthz` runs a 1-token completion, because forge can be up with Ollama stopped — its state on the morning of the build. Verdicts are cached 5 s (3 s when unhealthy) behind a single-flight lock, so concurrent traffic collapses into one probe. **A real bug the cloud path surfaced.** `gpt-5.4-mini` rejects `max_tokens` outright (`400 Unsupported parameter ... use max_completion_tokens`), while Ollama accepts only `max_tokens`. The first fallback attempt failed on exactly this. The router now translates between the two spellings per backend, so a caller can write either and both endpoints work. Verified against the live API: only `max_tokens` is affected — `temperature`, `top_p` and `stream` are all accepted. --- ## Verification `scripts/verify-cluster.sh` runs 13 checks from **inside** each cluster. All 13 passed on both. The "forge unavailable" cases were produced with a throwaway Deployment pointed at a dead upstream — **Ollama on forge was never stopped.** | # | Check | Lounge | Serenity | |---|---|---|---| | 1 | `/healthz`, forge up | 200 | 200 | | 2 | strict chat, forge up | 200, `backend=forge`, `PONG` | 200, `backend=forge`, `PONG` | | 3 | optimistic chat, forge up | 200, `backend=forge` | 200, `backend=forge` | | 4–5 | `/v1/models` both modes | 200 | 200 | | **6** | **strict chat, forge unavailable** | **503, clean error body** | **503, clean error body** | | **7** | **optimistic chat, forge unavailable** | **200, `backend=cloud`, `PONG`** | **200, `backend=cloud`, `PONG`** | | 8 | `/healthz`, forge unavailable | 503 | 503 | | 9–10 | liveness / readiness, forge unavailable | 200 / 200 | 200 / 200 | | 11 | streaming through forge | 200, 53 SSE chunks | 200, 46 SSE chunks | | 12–13 | bare `/v1` and unknown-mode guards | 404 / 404 | 404 / 404 | Check 1 on Lounge recorded `latency_ms: 28056` — a genuine cold model load, which is why the Kuma timeout is 48 s rather than something tighter. **Kuma DOWN path proven end-to-end**, not inferred: a temporary monitor pointed at a temporary 503-returning instance produced `status: 0, "Request failed with status code 503"`, while the real monitor simultaneously read `status: 1, "200 - OK"`. Both the temp monitor and the temp deployment were removed afterwards. **OpenClaw providers proven live:** ``` forgeway-strict/gpt-oss-120b -> PROVIDER-PROOF-OK (23.1 s) forgeway-optimistic/gpt-oss-120b -> PROVIDER-PROOF-OK (21.1 s) ``` --- ## Two corrections to the brief - **Serenity Kuma is `192.168.24.98`, not `172.24.24.98`.** The brief's address does not respond; the memory note was right, and the monitor was created on the correct host. - **The next free Lounge LB IP was `.90`, not `.89`** — `.89` is clipcascade, already in use. `.90` was taken; `docs/ip-allocations.md` is updated (11 of 19 in the band used). ## Open for Rick 1. ~~The DNS name.~~ **Resolved** — confirmed and built; see the addendum. 2. **The pfSense SNAT exemption** described above, which would let me tighten forge's firewall rule from "the mesh" to "Serenity's nodes". 3. **Monitoring vs. LoRA training.** A 60 s health probe keeps `gpt-oss:120b` resident on card 1, which collides with an ai-toolkit run. Levers are documented in the README (pause the monitor, or `FORGEWAY_HEALTH_COLD_LOAD=false`). If you would rather monitoring never hold the card, say so and I will change the default — the trade is detection latency and a cold load per probe. ## Where everything lives | | | |---|---| | Service source | `/home/rick/repos/forgeway` (this repo) | | Lounge manifests | `lounge-cluster-config/apps/forgeway/manifests/`, `base/apps/forgeway/` — commit `bba7898` | | Serenity manifests | `serenity-cluster-config/apps/forgeway/`, `base/apps/forgeway-app.yaml` — commit `4a32fde` | | Kuma ops files | `deploy/kuma/ops-{lounge,serenity}.json` | | OpenClaw provider config | `deploy/openclaw/providers.batch.json` | | Image | `git.falkvinge.net/rick/forgeway:0.1.1` | --- # Addendum, 2026-09-18 later — DNS and ingress (name confirmed by Rick) The open item is closed. Rick confirmed `forgeway.lounge.falkvinge.net`, `forgeway.serenity.falkvinge.net` (Chalet later) and the split-horizon local-first alias `forgeway.services.falkvinge.net`. All three are live at both sites and verified from real clients, and the OpenClaw providers now use the alias over HTTPS. ## What was built **Ingress at each site** (`apps/forgeway/.../ingress.yaml`, ingress-nginx, class `nginx`), serving `forgeway.services.falkvinge.net`, the per-site name, and the `.dmz..` CNAME hop. No SSL redirect — this is an API for scripts, and neither httpx nor the OpenAI SDK follow a 308 on POST by default. Buffering off and 600 s timeouts for SSE streams. HTTPS on the alias presents a valid certificate at both sites (`*.services.falkvinge.net` is on both internal wildcard certs); the per-site names are `http://` only, because neither internal cert carries `*.lounge.` / `*.serenity.`. Adding those SANs to the two *internal* cert-manager Certificates is a one-line change if ever wanted. The entrance bundle was not touched and must not be — `.services.` stays internal-only by convention. **DNS.** I followed the estate's existing `.services.` pattern exactly: every record in the `services.falkvinge.net` zone is a CNAME chain that ends at a site ingress, so: | Record | Zone (primary) | Points at | |---|---|---| | `forgeway.services` | `services.falkvinge.net` (Lounge Technitium) | `forgeway.dmz.lounge.falkvinge.net` | | `forgeway.dmz.lounge` | `dmz.lounge.falkvinge.net` (Lounge) | `ingress.dmz.lounge.falkvinge.net` → 172.19.10.11 | | `forgeway.lounge` | `lounge.falkvinge.net` (Lounge) | `forgeway.dmz.lounge.falkvinge.net` | | `forgeway.dmz.serenity` | `dmz.serenity.falkvinge.net` (Serenity Technitium) | `ingress.dmz.serenity.falkvinge.net` → 172.24.10.11 | | `forgeway.serenity` | `serenity.falkvinge.net` (Serenity) | `forgeway.dmz.serenity.falkvinge.net` | **The split-horizon override — the whole point.** Serenity's Pi-holes forward everything under `falkvinge.net` except `serenity.` to the Lounge Pi-holes, so without an override a Serenity client would silently receive the Lounge ingress and the remote instance. Both Serenity resolvers therefore pin the alias to the Serenity ingress `172.24.10.11`: - `172.24.10.201` (k8s, pods `pihole` + `pihole-b` behind one Service): in GitOps, `serenity-cluster-config/apps/pihole/pihole.yaml`. The ConfigMap is a subPath mount, so the two pods were restarted one at a time (no outage — they share the LB). - `172.24.1.201` (LXC): via the Pi-hole v6 API — a `dns.hosts` entry and one `misc.dnsmasq_lines` entry (that list was empty; nothing was overwritten). Both carry `host-record=forgeway.services.falkvinge.net,172.24.10.11` **and** `local=/forgeway.services.falkvinge.net/`. The second line closes a real hole: with only the host-record, AAAA queries were still forwarded to the Lounge. Today that returns no address, but the day the Lounge ingress gains an IPv6 record every dual-stack Serenity client would prefer it and quietly reach the *remote* instance — the same dual-stack bypass that defeated the Lounge overrides on 2026-08-24. AAAA is now NODATA on both. ## Interpretation I made, stated plainly The brief said each site's resolver should answer with "its own instance's MetalLB IP". I pointed the alias at each site's **ingress-nginx** (also a MetalLB IP) rather than at the forgeway Service IP directly, because that is how every other `.services.` name in the estate works and it is the only way the alias gets a valid certificate. Local-first is preserved: each site's ingress fronts only its own instance. The raw Service IPs (`172.19.10.90`, `172.24.10.21`) remain reachable as DNS-independent doors. ## Verification — the alias returns the LOCAL instance, never the remote one | Client | Site | `forgeway.services` → | `https://…/healthz` | `x-forgeway-site` | |---|---|---|---|---| | OpenClaw VM (resolvers `.1.201`/`.10.201`) | Lounge | `172.19.10.11` | 200, cert OK | `lounge` | | Lounge Kuma LXC | Lounge | `172.19.10.11` | 200, cert OK | `lounge` | | Niðavellir pod (CoreDNS → both Serenity Pi-holes) | Serenity | `172.24.10.11` | 200, cert OK | `serenity` | | Serenity Kuma LXC, `@172.24.10.201` and `@172.24.1.201` | Serenity | `172.24.10.11` | 200, cert OK | `serenity` | Per-site names, from both sites: `forgeway.lounge` → `172.19.10.11` / `site=lounge`; `forgeway.serenity` → `172.24.10.11` / `site=serenity`. A real `local-strict` completion through `https://forgeway.services.falkvinge.net` from the Lounge answered from forge. OpenClaw providers `forgeway-strict/gpt-oss-120b` and `forgeway-optimistic/gpt-oss-120b` now use the alias over HTTPS; both returned `PROVIDER-PROOF-OK` (22 s each, real completions). One wrinkle: OpenClaw exempts a plain-IP `http://` base URL from needing a key but demands one for an `https://` hostname (`missing-provider-auth`), so both carry the placeholder `apiKey: forgeway-no-auth`. It is not a credential; the router ignores the header. Side effect: `openclaw doctor --lint` now lists those two `apiKey` paths among its "plaintext secret-bearing config fields" — a false positive for these two, alongside the pre-existing entries it already flagged. ## Findings on the way (none caused by this work) 1. **Both LXC Pi-hole v6 APIs have no password set** — `172.19.1.201` and `172.24.1.201` both answer `/api/auth` with `"no password set"`. Any host that can reach them can rewrite their DNS config without credentials. It made today's LXC edit trivial; it is worth a password. The k8s Pi-holes require auth. 2. **The Serenity Kuma LXC's default resolver is dead for internal names.** It resolves via the UDM (`192.168.24.1`), which times out on every `falkvinge.net` name, while both Serenity Pi-holes answer it directly and instantly. Its monitors work because they use IPs. Same class as the Lounge Kuma resolv.conf note. 3. **Lounge `.1.202` lags Serenity zone changes by up to 15 minutes.** It is a tier-two secondary (Serenity primary → `.10.202` → `.1.202`) and `.10.202` sends no NOTIFY, so `.1.202` only refreshes on its 900 s SOA timer, and the Lounge Pi-holes round-robin both Technitiums. For a quarter of an hour after any Serenity record is created, Lounge clients get it about half the time — then negative-cache the miss for another 900 s. I resynced `.1.202` by API for today; the log confirms it. Optional hygiene, your call: `notify: SpecifiedNameServers → 172.19.1.202` on `.10.202`'s two Serenity secondaries. I initially misread this last one as a month-old broken replica; the serials were simply the primary's last-change dates. Corrected here and in memory. ## Where everything lives (additions) | | | |---|---| | Lounge ingress | `lounge-cluster-config/apps/forgeway/manifests/ingress.yaml` — commits `674d09e`, `15b226e` | | Serenity ingress + Pi-hole override | `serenity-cluster-config/apps/forgeway/ingress.yaml`, `apps/pihole/pihole.yaml` — `82fef33`, `f44dbb8`, `532ba68` | | Technitium records | live on `172.19.1.202` and `172.24.1.202` (zone data is not in git by design) | | LXC Pi-hole override | live on `172.24.1.201` via API (`dns.hosts` + `misc.dnsmasq_lines`) | | Technitium helper | `scripts/technitium.py` (zones / records / add / resync / logtail) | --- # Addendum 2, 2026-09-18 evening — native Ollama dialect + Home Assistant **Done and proven from Home Assistant itself.** forgeway 0.2.1 serves Ollama's native API on both modes beside the OpenAI one. The Lounge HA (2026.9.2) now has an Ollama integration entry pointed at forgeway, a conversation agent `conversation.forgeway_optimistic` on `gpt-oss:120b`, and a real question asked through HA's own conversation API was answered by forge: > Q: In one short sentence, what is Home Assistant? > A: *Home Assistant is an open-source platform for local home automation and smart-device control.* (64 s including a cold model load; the pod logs show no cloud fallback in the window.) | | Lounge | Serenity | |---|---|---| | Image | `0.2.1` | `0.2.1` | | Acceptance | **22/22** (13 OpenAI + 9 Ollama) | **22/22** | | Ollama strict-503 / optimistic cloud fallback / cloud NDJSON stream | 503 / 200 `backend=cloud` / 3 frames, `done:true` | same | ## What HA actually calls — from its source, not guessed HA 2026.9.2 pins `ollama==0.6.2`. The integration calls **`list()` → `GET /api/tags`** (setup, config-flow validation, model picker; **5 s timeout**) and **`chat(stream=True)` → `POST /api/chat`** with `keep_alive: -1`, `options: {num_ctx: 8192}`, `think: false` (default) and `tools` when the Assist API is on. It calls `pull()` only for a model absent from tags, which cannot happen (tags list exactly `gpt-oss:120b`). It reads `message.content`, `message.thinking`, `message.tool_calls`, `done`, `done_reason`. The client keeps the URL path, so `https:///local-optimistic` is a valid server URL — verified with the pinned client against the live service. ## The blocker that changed the URL: HA cannot reach the ingress Rick asked for `https://forgeway.services.falkvinge.net/local-optimistic` pasted into HA. From HA's VLAN (`192.168.120.0/22`) the alias resolves correctly to `172.19.10.11` and then **fails to connect** — that VLAN reaches nothing in `172.19.10.x` and not forge; it does reach the entrance HAProxy and the internet. The magpie jump to pfSense is down right now, so I could neither read the rule set nor add a rule. What works today, with a valid certificate and existing firewall openings: **`https://forgeway.dmz.lounge.falkvinge.net/local-optimistic`** → entrance HAProxy → `172.19.10.90:80` (the band opened for exactly this). The entrance cert covers `*.dmz.lounge.falkvinge.net`; it does **not** cover `*.lounge.falkvinge.net`, which corrects an assumption from this morning. Changes made: - `lounge-haproxy-config` (`08843be`, `27ce8e4`): vhost `forgeway.dmz.lounge.falkvinge.net` → backend `172.19.10.90:80`, `timeout server 600s`, and — because the entrance is a public edge and the router has no auth — `http-request deny` for non-RFC1918 sources. Deployed by the autopull. An intermediate commit (`596d68b`) failed `haproxy -c` (ACL used before declared); the autopull rolled it back as designed and the edge never stopped serving. Rick can confirm the applied revision in `/var/log/haproxy-autopull.log`. - Lounge Technitium: `forgeway.services` and `forgeway.lounge` now CNAME straight to the ingress; `forgeway.dmz.lounge` CNAMEs to `entrance.dmz.falkvinge.net`. The alias and per-site name are unchanged in effect; only the `.dmz.lounge.` hop moved. > **If you want HA on the `.services.` alias as originally asked:** pfSense, pass > `192.168.120.12` (or the sandbox net) → `172.19.10.11` TCP 443. Then > `scripts/ha_setup_ollama.py … https://forgeway.services.falkvinge.net/local-optimistic`. > I could not verify the source fence from the public side; it relies on pfSense not > rewriting inbound sources, which is the default. ## Design points in the dialect - **forge path = raw proxy.** forge is Ollama; frames are forge's own, model pinned. - **Cloud fallback translated both ways**, streaming as NDJSON with Ollama's compact byte shape; tool calls round-trip with synthesised ids; `format` mapped; usage → counts. - **`think` → `reasoning_effort`**, and **forced to `none` when tools are present**: the smoke test caught OpenAI rejecting `reasoning_effort` with function tools on gpt-5.4-mini — precisely the shape HA sends with the Assist API. Fixed before shipping. - **Reasoning trap**: gpt-oss returns `content` and `thinking` separately whatever `think` is, and HA sets no output budget, so answers are never empty. Where a caller does starve content with a tiny `num_predict`, the router flags it (`x-forgeway-warning: content-starved-by-reasoning`) rather than rewriting budgets. - **Tags never block on forge** (2.5 s cap, 60 s cache, synthetic entry if unreachable) — HA's 5 s list timeout would otherwise trip on every cold load. - **OpenAI dialect untouched**: checks 1–13 identical to this morning. ## For Rick to do (only if wanted) 1. Make it the default assistant: *Settings → Voice assistants → Assist → Conversation agent → "forgeway optimistic"*. Left as-is on purpose. 2. Let it control the house: on the Ollama integration's "forgeway optimistic" subentry, tick *Control Home Assistant*. Left off on purpose — that hands device control to an LLM. 3. Optional: the pfSense rule above, to use the alias instead of the entrance. ## Where (additions) | | | |---|---| | Dialect | `app/ollama_api.py`; shared bits in `app/common.py` | | HA setup + proof | `scripts/ha_setup_ollama.py` (idempotent; ends with a real conversation) | | Local smoke | `scripts/local-smoke-ollama.sh` (uses `ollama==0.6.2`, the version HA pins) | | Edge vhost | `lounge-haproxy-config` `27ce8e4` | | Cluster repos | `lounge-cluster-config` `c2c2c3e`, `serenity-cluster-config` `f35dc0a` | --- # Addendum 3, 2026-09-18 late — both privacy flavours in Home Assistant, on the alias Rick opened `192.168.120.12 → 172.19.10.11` on pfSense. Both HA Ollama entries now sit on the split-horizon alias, named so a dropdown reads as a data-residency choice: | Agent name (what a dropdown shows) | Entity | Server URL | |---|---|---| | **Local only — private (fails if forge is down)** | `conversation.forgeway_local_only` | `https://forgeway.services.falkvinge.net/local-strict` | | **Local preferred — falls back to OpenAI cloud** | `conversation.forgeway_local_preferred` | `https://forgeway.services.falkvinge.net/local-optimistic` | Entry titles: *Forgeway local-strict @ … — local only, private (fails if forge is down)* and *Forgeway local-optimistic @ … — local preferred (falls back to OpenAI cloud)*. Model `gpt-oss:120b` on both, no API key, think off, no device control, neither made the default assistant. The optimistic entry already existed when I started (created on Rick's side after the pfSense change, agent "Forge Conversation"); I reused it and renamed entry, device and entity rather than duplicating it. HA has no API to rename a subentry, so its subentry list still says "Forge Conversation" — cosmetic, UI-renameable. My earlier `.dmz.lounge` entry was already gone. **Created programmatically**, no hands needed: REST config flows for entries and subentries, WebSocket for renames. One trap found and fixed in the tooling: renaming an entry triggers a reload, and a subentry flow started during it aborts with `entry_not_loaded` — wait for `state: loaded` first. ## Proof by behaviour, through HA's own conversation API | | forge available | forge unavailable | |---|---|---| | Local only (strict) | *"Home Assistant is an open-source platform that lets you automate and control smart home devices."* (33 s, from forge) | **no answer, no cloud** — HA raised its Ollama error (REST surface: HTTP 500 from the conversation API; the integration logs *Unexpected error talking to Ollama server* with the 503 body per its source — I could not read HA's log to quote it, `/api/error_log` is gone in 2026.9) | | Local preferred (optimistic) | *"Home Assistant is an open-source home automation platform that lets you control and automate smart devices locally."* (66 s, from forge; pod log shows no fallback) | *"Home Assistant is an open-source home automation platform …"* in **1.3 s**, router log: *falling back to cloud gpt-5.4-mini* | The "unavailable" column: a throwaway forgeway instance with upstream `127.0.0.1:1` published under a temporary `forgeway-deadproof.services.falkvinge.net` (Ingress + a temporary Technitium CNAME), two temporary HA entries pointed at it, one question through each, then everything removed — entries, Ingress, Service, Deployment, DNS record. Ollama on forge was never touched. ## Route and DNS, verified from HA's side rather than assumed - The alias is **NXDOMAIN on 1.1.1.1 and 9.9.9.9** and answers only on `172.19.1.201` / `172.19.10.201` (→ `ingress.dmz.lounge.falkvinge.net` → `172.19.10.11`). HA's entries *loading* (which requires `/api/tags` over TLS with the `*.services.` cert) is therefore proof that HA resolves the name through the internal resolvers. HA's Supervisor API (which would show the resolver list directly) refuses the long-lived token (401), so this behavioural proof is the strongest available. - The pfSense rule covers HA's IP only: the scripthost on the same VLAN still cannot connect to `172.19.10.11`. Intended, and a useful negative control. - The entrance-HAProxy path via `forgeway.dmz.lounge.falkvinge.net` stays as a documented, fenced alternative; nothing uses it now. Say the word and I remove the vhost and record. ## Nothing left for Rick unless wanted - Default assistant: *Settings → Voice assistants → Assist → Conversation agent*. - Device control: tick *Control Home Assistant* on a subentry (deliberately off). - Cosmetic: rename the optimistic subentry "Forge Conversation" in the UI. --- # Addendum 4, 2026-09-18 night — converged on the approved design Rick approved: pfSense rule for HA, drop the entrance vhost. State was re-established with evidence before acting, then converged. **Who did what:** | | Done by | Evidence | |---|---|---| | pfSense rule `192.168.120.12 → 172.19.10.11` | **Rick** (his statement; I could not add or even read it — the magpie jump to pfSense is down, `ssh -p 2222` refused) | both HA entries on the alias load and answer; the scripthost on the same VLAN still cannot connect (rule is HA-IP-scoped) | | Entrance HAProxy vhost | **me**: added earlier today, **removed now** — `lounge-haproxy-config` `73f4389`, applied by the autopull at 19:05:07 UTC | before: outside vantage (Serenity's public egress → Lounge WAN) got the 403 fence; after: nginx 404 from the default backend; the `git` control vhost answered 200 on every 10 s tick through the change — no blip | | `forgeway.dmz.lounge.falkvinge.net` (DNS + Ingress host) | **me**: created for the edge path, **retired now** (`lounge-cluster-config` `b77c0d1`) | authoritative answer empty; live Ingress hosts = `forgeway.services` + `forgeway.lounge` | | HA entries on the alias | optimistic entry created on **Rick's side** (I renamed entry/device/entity); strict entry created by **me** | see below | ## HA end state, proven after the vhost was gone | Agent (dropdown) | Entity | Server URL | Real conversation after 19:05:07 | |---|---|---|---| | **Local only — private (fails if forge is down)** | `conversation.forgeway_local_only` | `https://forgeway.services.falkvinge.net/local-strict` | *"Home Assistant is an open-source platform that lets you automate and control smart-home devices from a single interface."* (73 s) | | **Local preferred — falls back to OpenAI cloud** | `conversation.forgeway_local_preferred` | `https://forgeway.services.falkvinge.net/local-optimistic` | *"Home Assistant is a home automation platform that lets you control and automate smart devices locally."* (25 s) | The ingress-nginx access log attributes these to HA by its client signature `ollama-python/0.6.2 (aarch64 linux) Python/3.14.6` (HA Yellow; every test of mine is curl or x86_64), posting to `/local-strict/api/chat` and `/local-optimistic/api/chat` against upstream service `forgeway-forgeway-80`: ``` 19:06:25 UTC POST /local-strict/api/chat -> 200 (73.3 s) conversation.forgeway_local_only 19:06:50 UTC POST /local-optimistic/api/chat -> 200 (24.6 s) conversation.forgeway_local_preferred ``` (A third HA-originated optimistic conversation completed at 19:05:27; not one I started.) With the edge vhost gone, the ingress is the only route a `.services.` name has. **Strict genuinely fails when forge is unavailable — re-proven through HA** with a throwaway dead-upstream instance published under a temporary `.services.` name: the strict temp agent produced no answer and no cloud call (HA raised its Ollama error; REST surface HTTP 500), the optimistic temp agent answered in 1.3 s with the router logging *falling back to cloud gpt-5.4-mini*. Temporaries (HA entries, Ingress, Service, Deployment, DNS record) removed; the namespace holds only forgeway. Ollama on forge was never touched. Nothing is left for Rick unless he wants the optional bits from Addendum 3 (default assistant, device control, cosmetic subentry rename). --- # Addendum 5, 2026-09-18 — sequencing check after the fact, and the resolution paths A sequencing constraint arrived after Addendum 4 had executed: *prove the pfSense rule from HA first, move the entries to the alias second, remove the vhost only then, and if the rule cannot be proven, leave the vhost.* It was worth re-establishing the state rather than arguing, so here is the timeline from HA's own `created_at` values and the ingress access log (all UTC, 2026-09-18): | Time | Event | Evidence | |---|---|---| | 18:26 | My first HA entry, on `forgeway.dmz.lounge` via the entrance vhost | Addendum 2 | | ≈18:35 | **Rick** adds the pfSense rule `192.168.120.12 → 172.19.10.11` and removes that entry | his statement; the entry was gone by 18:42 | | **18:39:18** | HA entry `01M2TX1P` created on **`forgeway.services…/local-optimistic`** (Rick's side) | HA `created_at`; loading requires `GET /api/tags` over TLS to `172.19.10.11`, i.e. **the rule works, proven from HA itself** | | **18:44:34** | HA entry `01M2TXBA` created on **`forgeway.services…/local-strict`** (me) | HA `created_at`, same proof | | ≈18:50–18:58 | Real conversations through both alias agents, and the dead-upstream difference test through HA | Addendum 3 | | **19:05:07** | Entrance vhost removed by the autopull | edge watch, Addendum 4 | | 19:06:25 / 19:06:50 | Both alias agents answer again | ingress log, HA's client UA | | 19:12:06 / 19:12:08 | Both alias agents answer again (this check) | ingress log; forgeway pod log shows no fallback, and strict cannot fall back by design | So the order 1 → 2 → 3 held: the rule was proven from HA at 18:39, both entries were on the alias by 18:44, and the vhost went at 19:05. **No HA entry referenced `forgeway.dmz.lounge` at any point after ≈18:35**, so the vhost was not "the only thing making HA's agent work" when it was removed — nothing depended on it. The one thing I would do differently: run a fresh conversation *immediately* before the removal rather than relying on the ones from fifteen minutes earlier. Nothing had changed in between, but a proof that is minutes old is cheaper than an argument. ## The resolution paths, recorded so nobody re-derives them From the Lounge client resolvers `172.19.1.201` / `172.19.10.201`, now: ``` forgeway.services.falkvinge.net → ingress.dmz.lounge.falkvinge.net → 172.19.10.11 (k8s ingress-nginx; local-first alias) forgeway.lounge.falkvinge.net → ingress.dmz.lounge.falkvinge.net → 172.19.10.11 (explicit Lounge instance; http only) forgeway.serenity.falkvinge.net → forgeway.dmz.serenity… → ingress.dmz.serenity… → 172.24.10.11 (explicit Serenity instance) forgeway.dmz.lounge.falkvinge.net → NXDOMAIN (retired; .10.201 held a cached answer for ≤5 min) ``` Until 19:03 the retired name resolved `→ entrance.dmz.falkvinge.net → demuxer.dmz.falkvinge.net → 172.20.0.101`, the entrance HAProxy — a genuinely different path from the alias. That is why, earlier today, "HA's VLAN can't reach `172.19.10.x`, so use the `.dmz.lounge.` name" was a coherent instruction at all: the two names went to different places. The claim itself was verified at the time from the scripthost on HA's VLAN (`172.19.10.11` unreachable, entrance reachable — Addendum 2), not from HA itself; Rick was right to ask for the resolution paths to be stated rather than implied. Today the alias goes to the ingress from HA because of Rick's pfSense rule, which is scoped to HA's IP: the scripthost on the same VLAN still gets no connection to `172.19.10.11:443` (checked again now). The entrance path is gone in config (`lounge-haproxy-config` `73f4389`), from inside, and from outside (Serenity's public egress → the Lounge WAN address → the edge's default backend, 404). ## Found while closing: one spurious cloud fallback at 19:10:53 — forge was alive While confirming the 19:12 answers came from forge, the Lounge pod log showed one earlier line: `19:10:53 falling back to cloud gpt-5.4-mini for /api/chat (completion-failed: ReadTimeout)`. Forge was not down. Its journal for the minute before shows **four full reloads of the 64 GB model in two minutes**, each 15–30 s, alternating between two context sizes: ``` 19:09:20 runner loaded n_ctx = 8192 (HA chat; HA sends options.num_ctx 8192) 19:09:51 reloaded n_ctx = 131072 (router health probes: default context) 19:10:23 reloaded n_ctx = 8192 (HA strict chat — answered 19:10:25, 62 s end to end) 19:10:53 reloaded n_ctx = 131072 (probe) — the roci-03 pod's probe hit its 45 s limit at that exact second → "unavailable" → optimistic request answered by the cloud (HA saw it 46 s later) 19:11:19 reloaded n_ctx = 131072 again (the interrupted load was torn down) ``` Three router instances (two Lounge pods, one Serenity) each probe forge every minute, all at the default context, while HA asks for 8192 — so every alternation is a reload, and a probe queued behind a live generation plus a reload can exceed 45 s. Effects: a **"Local preferred" answer silently came from the cloud while forge was working**, and a "Local only" answer took a minute. The Serenity instance's probe, arriving a second later, succeeded in 7 s. This is the runner thrash already noted in memory (`reference_ollama_num_ctx_runner_thrash`); what is new is that it trips the router's availability verdict, which is privacy-relevant on the optimistic endpoint. **Not changed by me.** The fix I recommend: make the router pin `options.num_ctx` on every forge-bound request (probe, OpenAI dialect, Ollama dialect) to one configured value, default 131072 — the size the model already runs at, so no extra VRAM — which ends the reloads entirely; HA's 8192 is a runner-size hint, not a prompt limit (HA trims history by message count). Router-only, one config knob, a 0.2.2 image. The alternate is setting `num_ctx` to 131072 in each HA subentry; it has the same effect but has to be repeated per agent and is invisible in git. I stopped short of either because it overrides a user-visible HA setting and the main session is tracking the thrash. --- # Addendum 6, 2026-09-18 — `num_ctx` pinned at the router (0.2.2), verified by behaviour **Done and shipped to both clusters.** Every forge-bound request from forgeway now carries `options.num_ctx = 131072` (`FORGEWAY_NUM_CTX`), overriding the caller's value, and the health probe carries the same value via Ollama's native `/api/chat`. `/status` exposes `num_ctx_pin` and `runner_context` and the router warns if they differ. Vikunja card #190 (), moved to Done. ## Value: 131072, and why it is not a compromise | Consideration | Finding | |---|---| | Retrieval workload | needs 40k–100k-token prompts; gpt-oss's trained window is 131072 (`gptoss.context_length`) | | VRAM | an *unpinned* request already loads at 131072 (forge has no `OLLAMA_CONTEXT_LENGTH`; Ollama 0.34 uses the model's trained length), so the pin adds nothing: runner 61,466 MiB resident, 66.7 GB used of 97.9 GB on card 1 | | Short turns | a large *allocated* context does not slow them: 1-token probes 60–100 ms, HA turns 0.3–0.6 s end to end, scripted short turns 0.5 s | | OpenAI dialect | Ollama's `/v1` endpoint **ignores `options.num_ctx`** (tested on qwen2.5vl: asked 6144, loaded 128000; native `/api/chat` asked 5120, loaded 5120), so OpenAI callers get the model default — the pin equals it, and `runner_context` is the tripwire | One value serves both workloads; no alternative needed. ## Before (0.2.1, 19:29–19:33 UTC) — an 8192-sending caller alternating with the probe Forge's journal for three rounds: `n_ctx` sequence **`8192 131072 8192 131072 8192 131072`**, **7 model loads**, and forge's own request log: ``` 19:30:09 200 32.6 s /api/chat (8192 caller: reload + answer) 19:30:30 499 47.0 s /v1/chat/completions (probe timed out during the reload back) 19:30:33 499 46.9 s /v1/chat/completions (Serenity's probe, same) 19:31:33 200 27.9 s /api/chat 19:32:41 200 28.8 s /api/chat 19:33:16 200 33.2 s /v1/chat/completions (probe) ``` Every alternation cost a reload; two probes hit the 45 s limit — the spurious "unavailable" that produced the 19:10:53 cloud fallback. (HA itself was already on 131072 since the main session's 19:21 subentry fix; its turns measured 0.3–0.6 s before and after — the router pin makes that independent of what any caller sends.) ## After (0.2.2, 19:36:59–19:38:28 UTC) — the mix that used to thrash, one window ``` HA-style turn (num_ctx 8192.0), strict 0.50 s health probe 0.13 s retrieval ~40k tokens, OpenAI dialect 7.61 s prompt_tokens=38484 needle FOUND HA-style turn (num_ctx 8192.0), optimistic 2.03 s retrieval ~100k tokens, OpenAI dialect 23.01 s prompt_tokens=96175 needle FOUND health probe 0.01 s HA-style turn (num_ctx 8192.0), strict 0.50 s scripted caller num_ctx 4096, generate 0.58 s HA-style turn (num_ctx 8192.0), optimistic 1.08 s real Home Assistant turns ×5 0.6 / 0.5 / 0.4 / 0.3 / 0.4 s ``` Forge's journal for that window: **`n_ctx` (re)loads: none; model loads: 0; timeouts: 0**; `ollama ps`: `gpt-oss:120b`, context 131072, 61,466 MiB. The model was never reloaded across 8192-, 4096-, default- and 100k-token callers. (Traffic from `127.0.0.1` on forge in the same window is the engine-evaluation lane, not mine; left alone.) | | before | after | |---|---|---| | reloads per 8192↔default alternation | 1 (15–30 s) | 0 | | turn with an off-value `num_ctx` | 27–33 s | 0.5 s | | probe during contention | 27–47 s, two timeouts | 0.01–0.13 s | | HA turn end to end | 0.3–0.6 s (already fixed on HA's side) | 0.3–0.6 s, now caller-independent | ## Acceptance 23/23 on Lounge and 23/23 on Serenity (13 OpenAI unchanged, 9 Ollama, plus check 23: `num_ctx_pin == runner_context`). Both instances report `pin=131072 runner=131072`. Ollama on forge was not stopped; ComfyUI on card 0 untouched (559 MiB, idle). ## Follow-up (not done, from the main session's note) The hot-path gate still runs a real completion before every proxied request (cached 5 s). With the pin it no longer thrashes, but a probe that needs a cold load still cannot fit in 45 s under contention and fails the optimistic request over to the cloud. The cleaner design — treat "model resident per `/api/ps`" as available on the hot path and keep the real completion for `/healthz` only — is a semantic change to what "available" means and stays Rick's call.