Files
Dolon (Agent)andClaude Fable 5.1 2d3c548542 0.2.2: pin num_ctx on every forge-bound request and on the health probe
Ollama keeps one runner config per model and reloads the 64 GB model on
every request whose num_ctx differs from the loaded runner's. The router
now sets options.num_ctx = FORGEWAY_NUM_CTX (default 131072) on the Ollama
dialect and carries the same value in the probe (native /api/chat; the
OpenAI /v1 endpoint ignores options, verified). /status exposes the pin and
the observed runner context and warns on mismatch; check 23 asserts it.

Verified by behaviour: an 8192 caller, a 4096 caller, probes and 40k/100k-
token retrieval prompts in one window -> 0 reloads, 0 timeouts, HA turns
0.3-0.6 s; before, 3 rounds of an 8192 caller = 7 loads and two probe
timeouts. 23/23 acceptance on both clusters.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-18 19:43:13 +00:00

634 lines
38 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# forgeway — build report, 2026-09-18
Two OpenAI-compatible query endpoints over forge's Ollama `gpt-oss:120b`, deployed as
one instance per cluster on Lounge (Rocinante) and Serenity (Niðavellir). Working, in
GitOps, monitored, and reachable by IP. **Update, later the same day:** Rick confirmed the
name; DNS, ingress and the split-horizon alias are live and verified — see the addendum at
the end of this file.
| | Lounge | Serenity |
|---|---|---|
| Address | `http://172.19.10.90` | `http://172.24.10.21` |
| Replicas | 2/2 | 2/2 |
| Kuma monitor | #307, `200 - OK` | #57, `200 - OK` |
| Real completion latency | 54 ms warm | 383 ms warm (mesh hop) |
| Acceptance checks | 13/13 | 13/13 |
Image `git.falkvinge.net/rick/forgeway:0.1.1`, linux/amd64 + linux/arm64 (both clusters
are arm64-majority: Lounge 12/15, Nidavellir 9/12), anonymously pullable from the Gitea
registry, so no imagePullSecret is needed.
---
## Three blockers found on forge, and what was changed
None of these were visible from the brief; all were found by probing.
**1. Ollama was bound to `127.0.0.1` only.** Nothing off-box could reach it. Added a
*new* systemd drop-in `/etc/systemd/system/ollama.service.d/20-listen.conf` with
`OLLAMA_HOST=0.0.0.0:11434`. The existing `10-gpu-pinning.conf` was not touched, and the
pinning was re-verified after the restart: `llama_prepare_model_devices: using device
CUDA0 ... (0000:02:00.0)` — still card 1, as intended.
**2. forge's ufw denied both clusters.** Default deny-incoming with a single allow for
the OpenClaw VM. Added three port-scoped rules:
```
11434/tcp ALLOW IN 172.19.9.0/24 Lounge k8s nodes
11434/tcp ALLOW IN 172.24.9.0/24 Nidavellir k8s nodes
11434/tcp ALLOW IN 172.19.0.1 Lounge pfSense SNAT, cross-site Serenity traffic
```
**3. Serenity traffic arrives SNAT'd.** Serenity pods reach forge with source
`172.19.0.1` — the Lounge pfSense — not their real `172.24.9.x`. Confirmed in forge's
`UFW BLOCK` log. The `172.24.9.0/24` rule alone was therefore never going to match.
> ⚠️ **Security note, for Rick's judgement.** Rule 3 is wider than I would like: it
> allows anything pfSense forwards out of the Tailscale mesh to reach Ollama, which has
> no authentication. It is scoped to port 11434, but it is not scoped to Serenity.
> The clean fix is a pfSense outbound-NAT exemption so tailscale→DMZ traffic keeps its
> original source (this matches the documented `NoSNAT` intent already set on the
> Serenity UDM), after which rule 3 can be deleted and `172.24.9.0/24` does the work.
> That is an edge-firewall change on your core router, so I did not make it.
---
## Design decisions worth knowing
**Discovery, not configuration.** Each instance holds an ordered candidate list — the
stable DNS name `forge.dmz.falkvinge.net`, then the Lounge literal `172.19.0.130`, then
the future Serenity literal `172.24.0.130`. Candidates are probed concurrently but
awaited in priority order, so the happy path is one fast round trip and dead candidates
never add latency. When silicon moves, re-pointing DNS is enough; if DNS is stale, the
Serenity literal already in the list catches it. Neither instance assumes a site.
**Readiness is not gated on forge.** This looks wrong at first glance and is the single
most important choice here. If readiness depended on forge, Kubernetes would evict every
pod from the Service during a forge outage — callers would get a connection refusal
instead of the 503 `local-strict` promises, and Kuma would see "no response" (router
gone) instead of 503 (forge down). Forge health is exposed separately on `/healthz`;
liveness and readiness only answer for the router process.
**Health is a real completion.** `/healthz` runs a 1-token completion, because forge can
be up with Ollama stopped — its state on the morning of the build. Verdicts are cached
5 s (3 s when unhealthy) behind a single-flight lock, so concurrent traffic collapses
into one probe.
**A real bug the cloud path surfaced.** `gpt-5.4-mini` rejects `max_tokens` outright
(`400 Unsupported parameter ... use max_completion_tokens`), while Ollama accepts only
`max_tokens`. The first fallback attempt failed on exactly this. The router now
translates between the two spellings per backend, so a caller can write either and both
endpoints work. Verified against the live API: only `max_tokens` is affected —
`temperature`, `top_p` and `stream` are all accepted.
---
## Verification
`scripts/verify-cluster.sh` runs 13 checks from **inside** each cluster. All 13 passed
on both. The "forge unavailable" cases were produced with a throwaway Deployment pointed
at a dead upstream — **Ollama on forge was never stopped.**
| # | Check | Lounge | Serenity |
|---|---|---|---|
| 1 | `/healthz`, forge up | 200 | 200 |
| 2 | strict chat, forge up | 200, `backend=forge`, `PONG` | 200, `backend=forge`, `PONG` |
| 3 | optimistic chat, forge up | 200, `backend=forge` | 200, `backend=forge` |
| 4–5 | `/v1/models` both modes | 200 | 200 |
| **6** | **strict chat, forge unavailable** | **503, clean error body** | **503, clean error body** |
| **7** | **optimistic chat, forge unavailable** | **200, `backend=cloud`, `PONG`** | **200, `backend=cloud`, `PONG`** |
| 8 | `/healthz`, forge unavailable | 503 | 503 |
| 9–10 | liveness / readiness, forge unavailable | 200 / 200 | 200 / 200 |
| 11 | streaming through forge | 200, 53 SSE chunks | 200, 46 SSE chunks |
| 12–13 | bare `/v1` and unknown-mode guards | 404 / 404 | 404 / 404 |
Check 1 on Lounge recorded `latency_ms: 28056` — a genuine cold model load, which is why
the Kuma timeout is 48 s rather than something tighter.
**Kuma DOWN path proven end-to-end**, not inferred: a temporary monitor pointed at a
temporary 503-returning instance produced `status: 0, "Request failed with status code
503"`, while the real monitor simultaneously read `status: 1, "200 - OK"`. Both the temp
monitor and the temp deployment were removed afterwards.
**OpenClaw providers proven live:**
```
forgeway-strict/gpt-oss-120b -> PROVIDER-PROOF-OK (23.1 s)
forgeway-optimistic/gpt-oss-120b -> PROVIDER-PROOF-OK (21.1 s)
```
---
## Two corrections to the brief
- **Serenity Kuma is `192.168.24.98`, not `172.24.24.98`.** The brief's address does not
respond; the memory note was right, and the monitor was created on the correct host.
- **The next free Lounge LB IP was `.90`, not `.89`** — `.89` is clipcascade, already in
use. `.90` was taken; `docs/ip-allocations.md` is updated (11 of 19 in the band used).
## Open for Rick
1. ~~The DNS name.~~ **Resolved** — confirmed and built; see the addendum.
2. **The pfSense SNAT exemption** described above, which would let me tighten forge's
firewall rule from "the mesh" to "Serenity's nodes".
3. **Monitoring vs. LoRA training.** A 60 s health probe keeps `gpt-oss:120b` resident on
card 1, which collides with an ai-toolkit run. Levers are documented in the README
(pause the monitor, or `FORGEWAY_HEALTH_COLD_LOAD=false`). If you would rather
monitoring never hold the card, say so and I will change the default — the trade is
detection latency and a cold load per probe.
## Where everything lives
| | |
|---|---|
| Service source | `/home/rick/repos/forgeway` (this repo) |
| Lounge manifests | `lounge-cluster-config/apps/forgeway/manifests/`, `base/apps/forgeway/` — commit `bba7898` |
| Serenity manifests | `serenity-cluster-config/apps/forgeway/`, `base/apps/forgeway-app.yaml` — commit `4a32fde` |
| Kuma ops files | `deploy/kuma/ops-{lounge,serenity}.json` |
| OpenClaw provider config | `deploy/openclaw/providers.batch.json` |
| Image | `git.falkvinge.net/rick/forgeway:0.1.1` |
---
# Addendum, 2026-09-18 later — DNS and ingress (name confirmed by Rick)
The open item is closed. Rick confirmed `forgeway.lounge.falkvinge.net`,
`forgeway.serenity.falkvinge.net` (Chalet later) and the split-horizon local-first alias
`forgeway.services.falkvinge.net`. All three are live at both sites and verified from real
clients, and the OpenClaw providers now use the alias over HTTPS.
## What was built
**Ingress at each site** (`apps/forgeway/.../ingress.yaml`, ingress-nginx, class `nginx`),
serving `forgeway.services.falkvinge.net`, the per-site name, and the `.dmz.<site>.` CNAME
hop. No SSL redirect — this is an API for scripts, and neither httpx nor the OpenAI SDK
follow a 308 on POST by default. Buffering off and 600 s timeouts for SSE streams. HTTPS on
the alias presents a valid certificate at both sites (`*.services.falkvinge.net` is on both
internal wildcard certs); the per-site names are `http://` only, because neither internal
cert carries `*.lounge.` / `*.serenity.`. Adding those SANs to the two *internal*
cert-manager Certificates is a one-line change if ever wanted. The entrance bundle was not
touched and must not be — `.services.` stays internal-only by convention.
**DNS.** I followed the estate's existing `.services.` pattern exactly: every record in
the `services.falkvinge.net` zone is a CNAME chain that ends at a site ingress, so:
| Record | Zone (primary) | Points at |
|---|---|---|
| `forgeway.services` | `services.falkvinge.net` (Lounge Technitium) | `forgeway.dmz.lounge.falkvinge.net` |
| `forgeway.dmz.lounge` | `dmz.lounge.falkvinge.net` (Lounge) | `ingress.dmz.lounge.falkvinge.net` → 172.19.10.11 |
| `forgeway.lounge` | `lounge.falkvinge.net` (Lounge) | `forgeway.dmz.lounge.falkvinge.net` |
| `forgeway.dmz.serenity` | `dmz.serenity.falkvinge.net` (Serenity Technitium) | `ingress.dmz.serenity.falkvinge.net` → 172.24.10.11 |
| `forgeway.serenity` | `serenity.falkvinge.net` (Serenity) | `forgeway.dmz.serenity.falkvinge.net` |
**The split-horizon override — the whole point.** Serenity's Pi-holes forward everything
under `falkvinge.net` except `serenity.` to the Lounge Pi-holes, so without an override a
Serenity client would silently receive the Lounge ingress and the remote instance. Both
Serenity resolvers therefore pin the alias to the Serenity ingress `172.24.10.11`:
- `172.24.10.201` (k8s, pods `pihole` + `pihole-b` behind one Service): in GitOps,
`serenity-cluster-config/apps/pihole/pihole.yaml`. The ConfigMap is a subPath mount, so
the two pods were restarted one at a time (no outage — they share the LB).
- `172.24.1.201` (LXC): via the Pi-hole v6 API — a `dns.hosts` entry and one
`misc.dnsmasq_lines` entry (that list was empty; nothing was overwritten).
Both carry `host-record=forgeway.services.falkvinge.net,172.24.10.11` **and**
`local=/forgeway.services.falkvinge.net/`. The second line closes a real hole: with only
the host-record, AAAA queries were still forwarded to the Lounge. Today that returns no
address, but the day the Lounge ingress gains an IPv6 record every dual-stack Serenity
client would prefer it and quietly reach the *remote* instance — the same dual-stack
bypass that defeated the Lounge overrides on 2026-08-24. AAAA is now NODATA on both.
## Interpretation I made, stated plainly
The brief said each site's resolver should answer with "its own instance's MetalLB IP".
I pointed the alias at each site's **ingress-nginx** (also a MetalLB IP) rather than at the
forgeway Service IP directly, because that is how every other `.services.` name in the
estate works and it is the only way the alias gets a valid certificate. Local-first is
preserved: each site's ingress fronts only its own instance. The raw Service IPs
(`172.19.10.90`, `172.24.10.21`) remain reachable as DNS-independent doors.
## Verification — the alias returns the LOCAL instance, never the remote one
| Client | Site | `forgeway.services` → | `https://…/healthz` | `x-forgeway-site` |
|---|---|---|---|---|
| OpenClaw VM (resolvers `.1.201`/`.10.201`) | Lounge | `172.19.10.11` | 200, cert OK | `lounge` |
| Lounge Kuma LXC | Lounge | `172.19.10.11` | 200, cert OK | `lounge` |
| Niðavellir pod (CoreDNS → both Serenity Pi-holes) | Serenity | `172.24.10.11` | 200, cert OK | `serenity` |
| Serenity Kuma LXC, `@172.24.10.201` and `@172.24.1.201` | Serenity | `172.24.10.11` | 200, cert OK | `serenity` |
Per-site names, from both sites: `forgeway.lounge` → `172.19.10.11` / `site=lounge`;
`forgeway.serenity` → `172.24.10.11` / `site=serenity`. A real `local-strict` completion
through `https://forgeway.services.falkvinge.net` from the Lounge answered from forge.
OpenClaw providers `forgeway-strict/gpt-oss-120b` and `forgeway-optimistic/gpt-oss-120b`
now use the alias over HTTPS; both returned `PROVIDER-PROOF-OK` (22 s each, real
completions). One wrinkle: OpenClaw exempts a plain-IP `http://` base URL from needing a
key but demands one for an `https://` hostname (`missing-provider-auth`), so both carry
the placeholder `apiKey: forgeway-no-auth`. It is not a credential; the router ignores
the header. Side effect: `openclaw doctor --lint` now lists those two `apiKey` paths
among its "plaintext secret-bearing config fields" — a false positive for these two,
alongside the pre-existing entries it already flagged.
## Findings on the way (none caused by this work)
1. **Both LXC Pi-hole v6 APIs have no password set** — `172.19.1.201` and `172.24.1.201`
both answer `/api/auth` with `"no password set"`. Any host that can reach them can
rewrite their DNS config without credentials. It made today's LXC edit trivial; it is
worth a password. The k8s Pi-holes require auth.
2. **The Serenity Kuma LXC's default resolver is dead for internal names.** It resolves via
the UDM (`192.168.24.1`), which times out on every `falkvinge.net` name, while both
Serenity Pi-holes answer it directly and instantly. Its monitors work because they use
IPs. Same class as the Lounge Kuma resolv.conf note.
3. **Lounge `.1.202` lags Serenity zone changes by up to 15 minutes.** It is a tier-two
secondary (Serenity primary → `.10.202` → `.1.202`) and `.10.202` sends no NOTIFY, so
`.1.202` only refreshes on its 900 s SOA timer, and the Lounge Pi-holes round-robin
both Technitiums. For a quarter of an hour after any Serenity record is created, Lounge
clients get it about half the time — then negative-cache the miss for another 900 s.
I resynced `.1.202` by API for today; the log confirms it. Optional hygiene, your call:
`notify: SpecifiedNameServers → 172.19.1.202` on `.10.202`'s two Serenity secondaries.
I initially misread this last one as a month-old broken replica; the serials were simply
the primary's last-change dates. Corrected here and in memory.
## Where everything lives (additions)
| | |
|---|---|
| Lounge ingress | `lounge-cluster-config/apps/forgeway/manifests/ingress.yaml` — commits `674d09e`, `15b226e` |
| Serenity ingress + Pi-hole override | `serenity-cluster-config/apps/forgeway/ingress.yaml`, `apps/pihole/pihole.yaml` — `82fef33`, `f44dbb8`, `532ba68` |
| Technitium records | live on `172.19.1.202` and `172.24.1.202` (zone data is not in git by design) |
| LXC Pi-hole override | live on `172.24.1.201` via API (`dns.hosts` + `misc.dnsmasq_lines`) |
| Technitium helper | `scripts/technitium.py` (zones / records / add / resync / logtail) |
---
# Addendum 2, 2026-09-18 evening — native Ollama dialect + Home Assistant
**Done and proven from Home Assistant itself.** forgeway 0.2.1 serves Ollama's native API
on both modes beside the OpenAI one. The Lounge HA (2026.9.2) now has an Ollama
integration entry pointed at forgeway, a conversation agent
`conversation.forgeway_optimistic` on `gpt-oss:120b`, and a real question asked through
HA's own conversation API was answered by forge:
> Q: In one short sentence, what is Home Assistant?
> A: *Home Assistant is an open-source platform for local home automation and smart-device control.*
(64 s including a cold model load; the pod logs show no cloud fallback in the window.)
| | Lounge | Serenity |
|---|---|---|
| Image | `0.2.1` | `0.2.1` |
| Acceptance | **22/22** (13 OpenAI + 9 Ollama) | **22/22** |
| Ollama strict-503 / optimistic cloud fallback / cloud NDJSON stream | 503 / 200 `backend=cloud` / 3 frames, `done:true` | same |
## What HA actually calls — from its source, not guessed
HA 2026.9.2 pins `ollama==0.6.2`. The integration calls **`list()` → `GET /api/tags`**
(setup, config-flow validation, model picker; **5 s timeout**) and
**`chat(stream=True)` → `POST /api/chat`** with `keep_alive: -1`, `options: {num_ctx: 8192}`,
`think: false` (default) and `tools` when the Assist API is on. It calls `pull()` only
for a model absent from tags, which cannot happen (tags list exactly `gpt-oss:120b`). It
reads `message.content`, `message.thinking`, `message.tool_calls`, `done`,
`done_reason`. The client keeps the URL path, so `https://<host>/local-optimistic` is a
valid server URL — verified with the pinned client against the live service.
## The blocker that changed the URL: HA cannot reach the ingress
Rick asked for `https://forgeway.services.falkvinge.net/local-optimistic` pasted into HA.
From HA's VLAN (`192.168.120.0/22`) the alias resolves correctly to `172.19.10.11` and
then **fails to connect** — that VLAN reaches nothing in `172.19.10.x` and not forge;
it does reach the entrance HAProxy and the internet. The magpie jump to pfSense is down
right now, so I could neither read the rule set nor add a rule.
What works today, with a valid certificate and existing firewall openings:
**`https://forgeway.dmz.lounge.falkvinge.net/local-optimistic`** → entrance HAProxy →
`172.19.10.90:80` (the band opened for exactly this). The entrance cert covers
`*.dmz.lounge.falkvinge.net`; it does **not** cover `*.lounge.falkvinge.net`, which
corrects an assumption from this morning. Changes made:
- `lounge-haproxy-config` (`08843be`, `27ce8e4`): vhost `forgeway.dmz.lounge.falkvinge.net`
→ backend `172.19.10.90:80`, `timeout server 600s`, and — because the entrance is a
public edge and the router has no auth — `http-request deny` for non-RFC1918 sources.
Deployed by the autopull. An intermediate commit (`596d68b`) failed `haproxy -c` (ACL
used before declared); the autopull rolled it back as designed and the edge never
stopped serving. Rick can confirm the applied revision in `/var/log/haproxy-autopull.log`.
- Lounge Technitium: `forgeway.services` and `forgeway.lounge` now CNAME straight to the
ingress; `forgeway.dmz.lounge` CNAMEs to `entrance.dmz.falkvinge.net`. The alias and
per-site name are unchanged in effect; only the `.dmz.lounge.` hop moved.
> **If you want HA on the `.services.` alias as originally asked:** pfSense, pass
> `192.168.120.12` (or the sandbox net) → `172.19.10.11` TCP 443. Then
> `scripts/ha_setup_ollama.py … https://forgeway.services.falkvinge.net/local-optimistic`.
> I could not verify the source fence from the public side; it relies on pfSense not
> rewriting inbound sources, which is the default.
## Design points in the dialect
- **forge path = raw proxy.** forge is Ollama; frames are forge's own, model pinned.
- **Cloud fallback translated both ways**, streaming as NDJSON with Ollama's compact
byte shape; tool calls round-trip with synthesised ids; `format` mapped; usage → counts.
- **`think` → `reasoning_effort`**, and **forced to `none` when tools are present**: the
smoke test caught OpenAI rejecting `reasoning_effort` with function tools on
gpt-5.4-mini — precisely the shape HA sends with the Assist API. Fixed before shipping.
- **Reasoning trap**: gpt-oss returns `content` and `thinking` separately whatever
`think` is, and HA sets no output budget, so answers are never empty. Where a caller
does starve content with a tiny `num_predict`, the router flags it
(`x-forgeway-warning: content-starved-by-reasoning`) rather than rewriting budgets.
- **Tags never block on forge** (2.5 s cap, 60 s cache, synthetic entry if unreachable) —
HA's 5 s list timeout would otherwise trip on every cold load.
- **OpenAI dialect untouched**: checks 1–13 identical to this morning.
## For Rick to do (only if wanted)
1. Make it the default assistant: *Settings → Voice assistants → Assist → Conversation
agent → "forgeway optimistic"*. Left as-is on purpose.
2. Let it control the house: on the Ollama integration's "forgeway optimistic" subentry,
tick *Control Home Assistant*. Left off on purpose — that hands device control to an
LLM.
3. Optional: the pfSense rule above, to use the alias instead of the entrance.
## Where (additions)
| | |
|---|---|
| Dialect | `app/ollama_api.py`; shared bits in `app/common.py` |
| HA setup + proof | `scripts/ha_setup_ollama.py` (idempotent; ends with a real conversation) |
| Local smoke | `scripts/local-smoke-ollama.sh` (uses `ollama==0.6.2`, the version HA pins) |
| Edge vhost | `lounge-haproxy-config` `27ce8e4` |
| Cluster repos | `lounge-cluster-config` `c2c2c3e`, `serenity-cluster-config` `f35dc0a` |
---
# Addendum 3, 2026-09-18 late — both privacy flavours in Home Assistant, on the alias
Rick opened `192.168.120.12 → 172.19.10.11` on pfSense. Both HA Ollama entries now sit on
the split-horizon alias, named so a dropdown reads as a data-residency choice:
| Agent name (what a dropdown shows) | Entity | Server URL |
|---|---|---|
| **Local only — private (fails if forge is down)** | `conversation.forgeway_local_only` | `https://forgeway.services.falkvinge.net/local-strict` |
| **Local preferred — falls back to OpenAI cloud** | `conversation.forgeway_local_preferred` | `https://forgeway.services.falkvinge.net/local-optimistic` |
Entry titles: *Forgeway local-strict @ … — local only, private (fails if forge is down)*
and *Forgeway local-optimistic @ … — local preferred (falls back to OpenAI cloud)*. Model
`gpt-oss:120b` on both, no API key, think off, no device control, neither made the
default assistant. The optimistic entry already existed when I started (created on
Rick's side after the pfSense change, agent "Forge Conversation"); I reused it and
renamed entry, device and entity rather than duplicating it. HA has no API to rename a
subentry, so its subentry list still says "Forge Conversation" — cosmetic, UI-renameable.
My earlier `.dmz.lounge` entry was already gone.
**Created programmatically**, no hands needed: REST config flows for entries and
subentries, WebSocket for renames. One trap found and fixed in the tooling: renaming an
entry triggers a reload, and a subentry flow started during it aborts with
`entry_not_loaded` — wait for `state: loaded` first.
## Proof by behaviour, through HA's own conversation API
| | forge available | forge unavailable |
|---|---|---|
| Local only (strict) | *"Home Assistant is an open-source platform that lets you automate and control smart home devices."* (33 s, from forge) | **no answer, no cloud** — HA raised its Ollama error (REST surface: HTTP 500 from the conversation API; the integration logs *Unexpected error talking to Ollama server* with the 503 body per its source — I could not read HA's log to quote it, `/api/error_log` is gone in 2026.9) |
| Local preferred (optimistic) | *"Home Assistant is an open-source home automation platform that lets you control and automate smart devices locally."* (66 s, from forge; pod log shows no fallback) | *"Home Assistant is an open-source home automation platform …"* in **1.3 s**, router log: *falling back to cloud gpt-5.4-mini* |
The "unavailable" column: a throwaway forgeway instance with upstream `127.0.0.1:1`
published under a temporary `forgeway-deadproof.services.falkvinge.net` (Ingress + a
temporary Technitium CNAME), two temporary HA entries pointed at it, one question through
each, then everything removed — entries, Ingress, Service, Deployment, DNS record. Ollama
on forge was never touched.
## Route and DNS, verified from HA's side rather than assumed
- The alias is **NXDOMAIN on 1.1.1.1 and 9.9.9.9** and answers only on `172.19.1.201` /
`172.19.10.201` (→ `ingress.dmz.lounge.falkvinge.net` → `172.19.10.11`). HA's entries
*loading* (which requires `/api/tags` over TLS with the `*.services.` cert) is therefore
proof that HA resolves the name through the internal resolvers. HA's Supervisor API
(which would show the resolver list directly) refuses the long-lived token (401), so
this behavioural proof is the strongest available.
- The pfSense rule covers HA's IP only: the scripthost on the same VLAN still cannot
connect to `172.19.10.11`. Intended, and a useful negative control.
- The entrance-HAProxy path via `forgeway.dmz.lounge.falkvinge.net` stays as a documented,
fenced alternative; nothing uses it now. Say the word and I remove the vhost and record.
## Nothing left for Rick unless wanted
- Default assistant: *Settings → Voice assistants → Assist → Conversation agent*.
- Device control: tick *Control Home Assistant* on a subentry (deliberately off).
- Cosmetic: rename the optimistic subentry "Forge Conversation" in the UI.
---
# Addendum 4, 2026-09-18 night — converged on the approved design
Rick approved: pfSense rule for HA, drop the entrance vhost. State was re-established
with evidence before acting, then converged. **Who did what:**
| | Done by | Evidence |
|---|---|---|
| pfSense rule `192.168.120.12 → 172.19.10.11` | **Rick** (his statement; I could not add or even read it — the magpie jump to pfSense is down, `ssh -p 2222` refused) | both HA entries on the alias load and answer; the scripthost on the same VLAN still cannot connect (rule is HA-IP-scoped) |
| Entrance HAProxy vhost | **me**: added earlier today, **removed now** — `lounge-haproxy-config` `73f4389`, applied by the autopull at 19:05:07 UTC | before: outside vantage (Serenity's public egress → Lounge WAN) got the 403 fence; after: nginx 404 from the default backend; the `git` control vhost answered 200 on every 10 s tick through the change — no blip |
| `forgeway.dmz.lounge.falkvinge.net` (DNS + Ingress host) | **me**: created for the edge path, **retired now** (`lounge-cluster-config` `b77c0d1`) | authoritative answer empty; live Ingress hosts = `forgeway.services` + `forgeway.lounge` |
| HA entries on the alias | optimistic entry created on **Rick's side** (I renamed entry/device/entity); strict entry created by **me** | see below |
## HA end state, proven after the vhost was gone
| Agent (dropdown) | Entity | Server URL | Real conversation after 19:05:07 |
|---|---|---|---|
| **Local only — private (fails if forge is down)** | `conversation.forgeway_local_only` | `https://forgeway.services.falkvinge.net/local-strict` | *"Home Assistant is an open-source platform that lets you automate and control smart-home devices from a single interface."* (73 s) |
| **Local preferred — falls back to OpenAI cloud** | `conversation.forgeway_local_preferred` | `https://forgeway.services.falkvinge.net/local-optimistic` | *"Home Assistant is a home automation platform that lets you control and automate smart devices locally."* (25 s) |
The ingress-nginx access log attributes these to HA by its client signature
`ollama-python/0.6.2 (aarch64 linux) Python/3.14.6` (HA Yellow; every test of mine is
curl or x86_64), posting to `/local-strict/api/chat` and `/local-optimistic/api/chat`
against upstream service `forgeway-forgeway-80`:
```
19:06:25 UTC POST /local-strict/api/chat -> 200 (73.3 s) conversation.forgeway_local_only
19:06:50 UTC POST /local-optimistic/api/chat -> 200 (24.6 s) conversation.forgeway_local_preferred
```
(A third HA-originated optimistic conversation completed at 19:05:27; not one I
started.) With the edge vhost gone, the ingress is the only route a `.services.` name has.
**Strict genuinely fails when forge is unavailable — re-proven through HA** with a
throwaway dead-upstream instance published under a temporary `.services.` name: the
strict temp agent produced no answer and no cloud call (HA raised its Ollama error; REST
surface HTTP 500), the optimistic temp agent answered in 1.3 s with the router logging
*falling back to cloud gpt-5.4-mini*. Temporaries (HA entries, Ingress, Service,
Deployment, DNS record) removed; the namespace holds only forgeway. Ollama on forge
was never touched.
Nothing is left for Rick unless he wants the optional bits from Addendum 3 (default
assistant, device control, cosmetic subentry rename).
---
# Addendum 5, 2026-09-18 — sequencing check after the fact, and the resolution paths
A sequencing constraint arrived after Addendum 4 had executed: *prove the pfSense rule
from HA first, move the entries to the alias second, remove the vhost only then, and if
the rule cannot be proven, leave the vhost.* It was worth re-establishing the state
rather than arguing, so here is the timeline from HA's own `created_at` values and the
ingress access log (all UTC, 2026-09-18):
| Time | Event | Evidence |
|---|---|---|
| 18:26 | My first HA entry, on `forgeway.dmz.lounge` via the entrance vhost | Addendum 2 |
| ≈18:35 | **Rick** adds the pfSense rule `192.168.120.12 → 172.19.10.11` and removes that entry | his statement; the entry was gone by 18:42 |
| **18:39:18** | HA entry `01M2TX1P` created on **`forgeway.services…/local-optimistic`** (Rick's side) | HA `created_at`; loading requires `GET /api/tags` over TLS to `172.19.10.11`, i.e. **the rule works, proven from HA itself** |
| **18:44:34** | HA entry `01M2TXBA` created on **`forgeway.services…/local-strict`** (me) | HA `created_at`, same proof |
| ≈18:50–18:58 | Real conversations through both alias agents, and the dead-upstream difference test through HA | Addendum 3 |
| **19:05:07** | Entrance vhost removed by the autopull | edge watch, Addendum 4 |
| 19:06:25 / 19:06:50 | Both alias agents answer again | ingress log, HA's client UA |
| 19:12:06 / 19:12:08 | Both alias agents answer again (this check) | ingress log; forgeway pod log shows no fallback, and strict cannot fall back by design |
So the order 1 → 2 → 3 held: the rule was proven from HA at 18:39, both entries were on
the alias by 18:44, and the vhost went at 19:05. **No HA entry referenced
`forgeway.dmz.lounge` at any point after ≈18:35**, so the vhost was not "the only thing
making HA's agent work" when it was removed — nothing depended on it. The one thing I
would do differently: run a fresh conversation *immediately* before the removal rather
than relying on the ones from fifteen minutes earlier. Nothing had changed in between,
but a proof that is minutes old is cheaper than an argument.
## The resolution paths, recorded so nobody re-derives them
From the Lounge client resolvers `172.19.1.201` / `172.19.10.201`, now:
```
forgeway.services.falkvinge.net → ingress.dmz.lounge.falkvinge.net → 172.19.10.11 (k8s ingress-nginx; local-first alias)
forgeway.lounge.falkvinge.net → ingress.dmz.lounge.falkvinge.net → 172.19.10.11 (explicit Lounge instance; http only)
forgeway.serenity.falkvinge.net → forgeway.dmz.serenity… → ingress.dmz.serenity… → 172.24.10.11 (explicit Serenity instance)
forgeway.dmz.lounge.falkvinge.net → NXDOMAIN (retired; .10.201 held a cached answer for ≤5 min)
```
Until 19:03 the retired name resolved `→ entrance.dmz.falkvinge.net → demuxer.dmz.falkvinge.net → 172.20.0.101`,
the entrance HAProxy — a genuinely different path from the alias. That is why, earlier
today, "HA's VLAN can't reach `172.19.10.x`, so use the `.dmz.lounge.` name" was a
coherent instruction at all: the two names went to different places. The claim itself
was verified at the time from the scripthost on HA's VLAN (`172.19.10.11` unreachable,
entrance reachable — Addendum 2), not from HA itself; Rick was right to ask for the
resolution paths to be stated rather than implied.
Today the alias goes to the ingress from HA because of Rick's pfSense rule, which is
scoped to HA's IP: the scripthost on the same VLAN still gets no connection to
`172.19.10.11:443` (checked again now). The entrance path is gone in config
(`lounge-haproxy-config` `73f4389`), from inside, and from outside (Serenity's public
egress → the Lounge WAN address → the edge's default backend, 404).
## Found while closing: one spurious cloud fallback at 19:10:53 — forge was alive
While confirming the 19:12 answers came from forge, the Lounge pod log showed one
earlier line: `19:10:53 falling back to cloud gpt-5.4-mini for /api/chat
(completion-failed: ReadTimeout)`. Forge was not down. Its journal for the minute
before shows **four full reloads of the 64 GB model in two minutes**, each 15–30 s,
alternating between two context sizes:
```
19:09:20 runner loaded n_ctx = 8192 (HA chat; HA sends options.num_ctx 8192)
19:09:51 reloaded n_ctx = 131072 (router health probes: default context)
19:10:23 reloaded n_ctx = 8192 (HA strict chat — answered 19:10:25, 62 s end to end)
19:10:53 reloaded n_ctx = 131072 (probe) — the roci-03 pod's probe hit its 45 s limit
at that exact second → "unavailable" → optimistic
request answered by the cloud (HA saw it 46 s later)
19:11:19 reloaded n_ctx = 131072 again (the interrupted load was torn down)
```
Three router instances (two Lounge pods, one Serenity) each probe forge every minute,
all at the default context, while HA asks for 8192 — so every alternation is a reload,
and a probe queued behind a live generation plus a reload can exceed 45 s. Effects:
a **"Local preferred" answer silently came from the cloud while forge was working**,
and a "Local only" answer took a minute. The Serenity instance's probe, arriving a
second later, succeeded in 7 s. This is the runner thrash already noted in memory
(`reference_ollama_num_ctx_runner_thrash`); what is new is that it trips the router's
availability verdict, which is privacy-relevant on the optimistic endpoint.
**Not changed by me.** The fix I recommend: make the router pin `options.num_ctx` on
every forge-bound request (probe, OpenAI dialect, Ollama dialect) to one configured
value, default 131072 — the size the model already runs at, so no extra VRAM — which
ends the reloads entirely; HA's 8192 is a runner-size hint, not a prompt limit (HA trims
history by message count). Router-only, one config knob, a 0.2.2 image. The alternate
is setting `num_ctx` to 131072 in each HA subentry; it has the same effect but has to
be repeated per agent and is invisible in git. I stopped short of either because it
overrides a user-visible HA setting and the main session is tracking the thrash.
---
# Addendum 6, 2026-09-18 — `num_ctx` pinned at the router (0.2.2), verified by behaviour
**Done and shipped to both clusters.** Every forge-bound request from forgeway now carries
`options.num_ctx = 131072` (`FORGEWAY_NUM_CTX`), overriding the caller's value, and the
health probe carries the same value via Ollama's native `/api/chat`. `/status` exposes
`num_ctx_pin` and `runner_context` and the router warns if they differ. Vikunja card
#190 (<https://progress.falkvinge.net/tasks/190>), moved to Done.
## Value: 131072, and why it is not a compromise
| Consideration | Finding |
|---|---|
| Retrieval workload | needs 40k–100k-token prompts; gpt-oss's trained window is 131072 (`gptoss.context_length`) |
| VRAM | an *unpinned* request already loads at 131072 (forge has no `OLLAMA_CONTEXT_LENGTH`; Ollama 0.34 uses the model's trained length), so the pin adds nothing: runner 61,466 MiB resident, 66.7 GB used of 97.9 GB on card 1 |
| Short turns | a large *allocated* context does not slow them: 1-token probes 60–100 ms, HA turns 0.3–0.6 s end to end, scripted short turns 0.5 s |
| OpenAI dialect | Ollama's `/v1` endpoint **ignores `options.num_ctx`** (tested on qwen2.5vl: asked 6144, loaded 128000; native `/api/chat` asked 5120, loaded 5120), so OpenAI callers get the model default — the pin equals it, and `runner_context` is the tripwire |
One value serves both workloads; no alternative needed.
## Before (0.2.1, 19:29–19:33 UTC) — an 8192-sending caller alternating with the probe
Forge's journal for three rounds: `n_ctx` sequence **`8192 131072 8192 131072 8192 131072`**,
**7 model loads**, and forge's own request log:
```
19:30:09 200 32.6 s /api/chat (8192 caller: reload + answer)
19:30:30 499 47.0 s /v1/chat/completions (probe timed out during the reload back)
19:30:33 499 46.9 s /v1/chat/completions (Serenity's probe, same)
19:31:33 200 27.9 s /api/chat
19:32:41 200 28.8 s /api/chat
19:33:16 200 33.2 s /v1/chat/completions (probe)
```
Every alternation cost a reload; two probes hit the 45 s limit — the spurious
"unavailable" that produced the 19:10:53 cloud fallback. (HA itself was already on
131072 since the main session's 19:21 subentry fix; its turns measured 0.3–0.6 s
before and after — the router pin makes that independent of what any caller sends.)
## After (0.2.2, 19:36:59–19:38:28 UTC) — the mix that used to thrash, one window
```
HA-style turn (num_ctx 8192.0), strict 0.50 s
health probe 0.13 s
retrieval ~40k tokens, OpenAI dialect 7.61 s prompt_tokens=38484 needle FOUND
HA-style turn (num_ctx 8192.0), optimistic 2.03 s
retrieval ~100k tokens, OpenAI dialect 23.01 s prompt_tokens=96175 needle FOUND
health probe 0.01 s
HA-style turn (num_ctx 8192.0), strict 0.50 s
scripted caller num_ctx 4096, generate 0.58 s
HA-style turn (num_ctx 8192.0), optimistic 1.08 s
real Home Assistant turns ×5 0.6 / 0.5 / 0.4 / 0.3 / 0.4 s
```
Forge's journal for that window: **`n_ctx` (re)loads: none; model loads: 0; timeouts: 0**;
`ollama ps`: `gpt-oss:120b`, context 131072, 61,466 MiB. The model was never reloaded
across 8192-, 4096-, default- and 100k-token callers. (Traffic from `127.0.0.1` on forge in
the same window is the engine-evaluation lane, not mine; left alone.)
| | before | after |
|---|---|---|
| reloads per 8192↔default alternation | 1 (15–30 s) | 0 |
| turn with an off-value `num_ctx` | 27–33 s | 0.5 s |
| probe during contention | 27–47 s, two timeouts | 0.01–0.13 s |
| HA turn end to end | 0.3–0.6 s (already fixed on HA's side) | 0.3–0.6 s, now caller-independent |
## Acceptance
23/23 on Lounge and 23/23 on Serenity (13 OpenAI unchanged, 9 Ollama, plus check 23:
`num_ctx_pin == runner_context`). Both instances report `pin=131072 runner=131072`.
Ollama on forge was not stopped; ComfyUI on card 0 untouched (559 MiB, idle).
## Follow-up (not done, from the main session's note)
The hot-path gate still runs a real completion before every proxied request (cached 5 s).
With the pin it no longer thrashes, but a probe that needs a cold load still cannot fit
in 45 s under contention and fails the optimistic request over to the cloud. The cleaner
design — treat "model resident per `/api/ps`" as available on the hot path and keep the
real completion for `/healthz` only — is a semantic change to what "available" means and
stays Rick's call.