Ollama keeps one runner config per model and reloads the 64 GB model on every request whose num_ctx differs from the loaded runner's. The router now sets options.num_ctx = FORGEWAY_NUM_CTX (default 131072) on the Ollama dialect and carries the same value in the probe (native /api/chat; the OpenAI /v1 endpoint ignores options, verified). /status exposes the pin and the observed runner context and warns on mismatch; check 23 asserts it. Verified by behaviour: an 8192 caller, a 4096 caller, probes and 40k/100k- token retrieval prompts in one window -> 0 reloads, 0 timeouts, HA turns 0.3-0.6 s; before, 3 rounds of an 8192 caller = 7 loads and two probe timeouts. 23/23 acceptance on both clusters. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
634 lines
38 KiB
Markdown
634 lines
38 KiB
Markdown
# forgeway — build report, 2026-09-18
|
||
|
||
Two OpenAI-compatible query endpoints over forge's Ollama `gpt-oss:120b`, deployed as
|
||
one instance per cluster on Lounge (Rocinante) and Serenity (Niðavellir). Working, in
|
||
GitOps, monitored, and reachable by IP. **Update, later the same day:** Rick confirmed the
|
||
name; DNS, ingress and the split-horizon alias are live and verified — see the addendum at
|
||
the end of this file.
|
||
|
||
| | Lounge | Serenity |
|
||
|---|---|---|
|
||
| Address | `http://172.19.10.90` | `http://172.24.10.21` |
|
||
| Replicas | 2/2 | 2/2 |
|
||
| Kuma monitor | #307, `200 - OK` | #57, `200 - OK` |
|
||
| Real completion latency | 54 ms warm | 383 ms warm (mesh hop) |
|
||
| Acceptance checks | 13/13 | 13/13 |
|
||
|
||
Image `git.falkvinge.net/rick/forgeway:0.1.1`, linux/amd64 + linux/arm64 (both clusters
|
||
are arm64-majority: Lounge 12/15, Nidavellir 9/12), anonymously pullable from the Gitea
|
||
registry, so no imagePullSecret is needed.
|
||
|
||
---
|
||
|
||
## Three blockers found on forge, and what was changed
|
||
|
||
None of these were visible from the brief; all were found by probing.
|
||
|
||
**1. Ollama was bound to `127.0.0.1` only.** Nothing off-box could reach it. Added a
|
||
*new* systemd drop-in `/etc/systemd/system/ollama.service.d/20-listen.conf` with
|
||
`OLLAMA_HOST=0.0.0.0:11434`. The existing `10-gpu-pinning.conf` was not touched, and the
|
||
pinning was re-verified after the restart: `llama_prepare_model_devices: using device
|
||
CUDA0 ... (0000:02:00.0)` — still card 1, as intended.
|
||
|
||
**2. forge's ufw denied both clusters.** Default deny-incoming with a single allow for
|
||
the OpenClaw VM. Added three port-scoped rules:
|
||
|
||
```
|
||
11434/tcp ALLOW IN 172.19.9.0/24 Lounge k8s nodes
|
||
11434/tcp ALLOW IN 172.24.9.0/24 Nidavellir k8s nodes
|
||
11434/tcp ALLOW IN 172.19.0.1 Lounge pfSense SNAT, cross-site Serenity traffic
|
||
```
|
||
|
||
**3. Serenity traffic arrives SNAT'd.** Serenity pods reach forge with source
|
||
`172.19.0.1` — the Lounge pfSense — not their real `172.24.9.x`. Confirmed in forge's
|
||
`UFW BLOCK` log. The `172.24.9.0/24` rule alone was therefore never going to match.
|
||
|
||
> ⚠️ **Security note, for Rick's judgement.** Rule 3 is wider than I would like: it
|
||
> allows anything pfSense forwards out of the Tailscale mesh to reach Ollama, which has
|
||
> no authentication. It is scoped to port 11434, but it is not scoped to Serenity.
|
||
> The clean fix is a pfSense outbound-NAT exemption so tailscale→DMZ traffic keeps its
|
||
> original source (this matches the documented `NoSNAT` intent already set on the
|
||
> Serenity UDM), after which rule 3 can be deleted and `172.24.9.0/24` does the work.
|
||
> That is an edge-firewall change on your core router, so I did not make it.
|
||
|
||
---
|
||
|
||
## Design decisions worth knowing
|
||
|
||
**Discovery, not configuration.** Each instance holds an ordered candidate list — the
|
||
stable DNS name `forge.dmz.falkvinge.net`, then the Lounge literal `172.19.0.130`, then
|
||
the future Serenity literal `172.24.0.130`. Candidates are probed concurrently but
|
||
awaited in priority order, so the happy path is one fast round trip and dead candidates
|
||
never add latency. When silicon moves, re-pointing DNS is enough; if DNS is stale, the
|
||
Serenity literal already in the list catches it. Neither instance assumes a site.
|
||
|
||
**Readiness is not gated on forge.** This looks wrong at first glance and is the single
|
||
most important choice here. If readiness depended on forge, Kubernetes would evict every
|
||
pod from the Service during a forge outage — callers would get a connection refusal
|
||
instead of the 503 `local-strict` promises, and Kuma would see "no response" (router
|
||
gone) instead of 503 (forge down). Forge health is exposed separately on `/healthz`;
|
||
liveness and readiness only answer for the router process.
|
||
|
||
**Health is a real completion.** `/healthz` runs a 1-token completion, because forge can
|
||
be up with Ollama stopped — its state on the morning of the build. Verdicts are cached
|
||
5 s (3 s when unhealthy) behind a single-flight lock, so concurrent traffic collapses
|
||
into one probe.
|
||
|
||
**A real bug the cloud path surfaced.** `gpt-5.4-mini` rejects `max_tokens` outright
|
||
(`400 Unsupported parameter ... use max_completion_tokens`), while Ollama accepts only
|
||
`max_tokens`. The first fallback attempt failed on exactly this. The router now
|
||
translates between the two spellings per backend, so a caller can write either and both
|
||
endpoints work. Verified against the live API: only `max_tokens` is affected —
|
||
`temperature`, `top_p` and `stream` are all accepted.
|
||
|
||
---
|
||
|
||
## Verification
|
||
|
||
`scripts/verify-cluster.sh` runs 13 checks from **inside** each cluster. All 13 passed
|
||
on both. The "forge unavailable" cases were produced with a throwaway Deployment pointed
|
||
at a dead upstream — **Ollama on forge was never stopped.**
|
||
|
||
| # | Check | Lounge | Serenity |
|
||
|---|---|---|---|
|
||
| 1 | `/healthz`, forge up | 200 | 200 |
|
||
| 2 | strict chat, forge up | 200, `backend=forge`, `PONG` | 200, `backend=forge`, `PONG` |
|
||
| 3 | optimistic chat, forge up | 200, `backend=forge` | 200, `backend=forge` |
|
||
| 4–5 | `/v1/models` both modes | 200 | 200 |
|
||
| **6** | **strict chat, forge unavailable** | **503, clean error body** | **503, clean error body** |
|
||
| **7** | **optimistic chat, forge unavailable** | **200, `backend=cloud`, `PONG`** | **200, `backend=cloud`, `PONG`** |
|
||
| 8 | `/healthz`, forge unavailable | 503 | 503 |
|
||
| 9–10 | liveness / readiness, forge unavailable | 200 / 200 | 200 / 200 |
|
||
| 11 | streaming through forge | 200, 53 SSE chunks | 200, 46 SSE chunks |
|
||
| 12–13 | bare `/v1` and unknown-mode guards | 404 / 404 | 404 / 404 |
|
||
|
||
Check 1 on Lounge recorded `latency_ms: 28056` — a genuine cold model load, which is why
|
||
the Kuma timeout is 48 s rather than something tighter.
|
||
|
||
**Kuma DOWN path proven end-to-end**, not inferred: a temporary monitor pointed at a
|
||
temporary 503-returning instance produced `status: 0, "Request failed with status code
|
||
503"`, while the real monitor simultaneously read `status: 1, "200 - OK"`. Both the temp
|
||
monitor and the temp deployment were removed afterwards.
|
||
|
||
**OpenClaw providers proven live:**
|
||
|
||
```
|
||
forgeway-strict/gpt-oss-120b -> PROVIDER-PROOF-OK (23.1 s)
|
||
forgeway-optimistic/gpt-oss-120b -> PROVIDER-PROOF-OK (21.1 s)
|
||
```
|
||
|
||
---
|
||
|
||
## Two corrections to the brief
|
||
|
||
- **Serenity Kuma is `192.168.24.98`, not `172.24.24.98`.** The brief's address does not
|
||
respond; the memory note was right, and the monitor was created on the correct host.
|
||
- **The next free Lounge LB IP was `.90`, not `.89`** — `.89` is clipcascade, already in
|
||
use. `.90` was taken; `docs/ip-allocations.md` is updated (11 of 19 in the band used).
|
||
|
||
## Open for Rick
|
||
|
||
1. ~~The DNS name.~~ **Resolved** — confirmed and built; see the addendum.
|
||
2. **The pfSense SNAT exemption** described above, which would let me tighten forge's
|
||
firewall rule from "the mesh" to "Serenity's nodes".
|
||
3. **Monitoring vs. LoRA training.** A 60 s health probe keeps `gpt-oss:120b` resident on
|
||
card 1, which collides with an ai-toolkit run. Levers are documented in the README
|
||
(pause the monitor, or `FORGEWAY_HEALTH_COLD_LOAD=false`). If you would rather
|
||
monitoring never hold the card, say so and I will change the default — the trade is
|
||
detection latency and a cold load per probe.
|
||
|
||
## Where everything lives
|
||
|
||
| | |
|
||
|---|---|
|
||
| Service source | `/home/rick/repos/forgeway` (this repo) |
|
||
| Lounge manifests | `lounge-cluster-config/apps/forgeway/manifests/`, `base/apps/forgeway/` — commit `bba7898` |
|
||
| Serenity manifests | `serenity-cluster-config/apps/forgeway/`, `base/apps/forgeway-app.yaml` — commit `4a32fde` |
|
||
| Kuma ops files | `deploy/kuma/ops-{lounge,serenity}.json` |
|
||
| OpenClaw provider config | `deploy/openclaw/providers.batch.json` |
|
||
| Image | `git.falkvinge.net/rick/forgeway:0.1.1` |
|
||
|
||
---
|
||
|
||
# Addendum, 2026-09-18 later — DNS and ingress (name confirmed by Rick)
|
||
|
||
The open item is closed. Rick confirmed `forgeway.lounge.falkvinge.net`,
|
||
`forgeway.serenity.falkvinge.net` (Chalet later) and the split-horizon local-first alias
|
||
`forgeway.services.falkvinge.net`. All three are live at both sites and verified from real
|
||
clients, and the OpenClaw providers now use the alias over HTTPS.
|
||
|
||
## What was built
|
||
|
||
**Ingress at each site** (`apps/forgeway/.../ingress.yaml`, ingress-nginx, class `nginx`),
|
||
serving `forgeway.services.falkvinge.net`, the per-site name, and the `.dmz.<site>.` CNAME
|
||
hop. No SSL redirect — this is an API for scripts, and neither httpx nor the OpenAI SDK
|
||
follow a 308 on POST by default. Buffering off and 600 s timeouts for SSE streams. HTTPS on
|
||
the alias presents a valid certificate at both sites (`*.services.falkvinge.net` is on both
|
||
internal wildcard certs); the per-site names are `http://` only, because neither internal
|
||
cert carries `*.lounge.` / `*.serenity.`. Adding those SANs to the two *internal*
|
||
cert-manager Certificates is a one-line change if ever wanted. The entrance bundle was not
|
||
touched and must not be — `.services.` stays internal-only by convention.
|
||
|
||
**DNS.** I followed the estate's existing `.services.` pattern exactly: every record in
|
||
the `services.falkvinge.net` zone is a CNAME chain that ends at a site ingress, so:
|
||
|
||
| Record | Zone (primary) | Points at |
|
||
|---|---|---|
|
||
| `forgeway.services` | `services.falkvinge.net` (Lounge Technitium) | `forgeway.dmz.lounge.falkvinge.net` |
|
||
| `forgeway.dmz.lounge` | `dmz.lounge.falkvinge.net` (Lounge) | `ingress.dmz.lounge.falkvinge.net` → 172.19.10.11 |
|
||
| `forgeway.lounge` | `lounge.falkvinge.net` (Lounge) | `forgeway.dmz.lounge.falkvinge.net` |
|
||
| `forgeway.dmz.serenity` | `dmz.serenity.falkvinge.net` (Serenity Technitium) | `ingress.dmz.serenity.falkvinge.net` → 172.24.10.11 |
|
||
| `forgeway.serenity` | `serenity.falkvinge.net` (Serenity) | `forgeway.dmz.serenity.falkvinge.net` |
|
||
|
||
**The split-horizon override — the whole point.** Serenity's Pi-holes forward everything
|
||
under `falkvinge.net` except `serenity.` to the Lounge Pi-holes, so without an override a
|
||
Serenity client would silently receive the Lounge ingress and the remote instance. Both
|
||
Serenity resolvers therefore pin the alias to the Serenity ingress `172.24.10.11`:
|
||
|
||
- `172.24.10.201` (k8s, pods `pihole` + `pihole-b` behind one Service): in GitOps,
|
||
`serenity-cluster-config/apps/pihole/pihole.yaml`. The ConfigMap is a subPath mount, so
|
||
the two pods were restarted one at a time (no outage — they share the LB).
|
||
- `172.24.1.201` (LXC): via the Pi-hole v6 API — a `dns.hosts` entry and one
|
||
`misc.dnsmasq_lines` entry (that list was empty; nothing was overwritten).
|
||
|
||
Both carry `host-record=forgeway.services.falkvinge.net,172.24.10.11` **and**
|
||
`local=/forgeway.services.falkvinge.net/`. The second line closes a real hole: with only
|
||
the host-record, AAAA queries were still forwarded to the Lounge. Today that returns no
|
||
address, but the day the Lounge ingress gains an IPv6 record every dual-stack Serenity
|
||
client would prefer it and quietly reach the *remote* instance — the same dual-stack
|
||
bypass that defeated the Lounge overrides on 2026-08-24. AAAA is now NODATA on both.
|
||
|
||
## Interpretation I made, stated plainly
|
||
|
||
The brief said each site's resolver should answer with "its own instance's MetalLB IP".
|
||
I pointed the alias at each site's **ingress-nginx** (also a MetalLB IP) rather than at the
|
||
forgeway Service IP directly, because that is how every other `.services.` name in the
|
||
estate works and it is the only way the alias gets a valid certificate. Local-first is
|
||
preserved: each site's ingress fronts only its own instance. The raw Service IPs
|
||
(`172.19.10.90`, `172.24.10.21`) remain reachable as DNS-independent doors.
|
||
|
||
## Verification — the alias returns the LOCAL instance, never the remote one
|
||
|
||
| Client | Site | `forgeway.services` → | `https://…/healthz` | `x-forgeway-site` |
|
||
|---|---|---|---|---|
|
||
| OpenClaw VM (resolvers `.1.201`/`.10.201`) | Lounge | `172.19.10.11` | 200, cert OK | `lounge` |
|
||
| Lounge Kuma LXC | Lounge | `172.19.10.11` | 200, cert OK | `lounge` |
|
||
| Niðavellir pod (CoreDNS → both Serenity Pi-holes) | Serenity | `172.24.10.11` | 200, cert OK | `serenity` |
|
||
| Serenity Kuma LXC, `@172.24.10.201` and `@172.24.1.201` | Serenity | `172.24.10.11` | 200, cert OK | `serenity` |
|
||
|
||
Per-site names, from both sites: `forgeway.lounge` → `172.19.10.11` / `site=lounge`;
|
||
`forgeway.serenity` → `172.24.10.11` / `site=serenity`. A real `local-strict` completion
|
||
through `https://forgeway.services.falkvinge.net` from the Lounge answered from forge.
|
||
|
||
OpenClaw providers `forgeway-strict/gpt-oss-120b` and `forgeway-optimistic/gpt-oss-120b`
|
||
now use the alias over HTTPS; both returned `PROVIDER-PROOF-OK` (22 s each, real
|
||
completions). One wrinkle: OpenClaw exempts a plain-IP `http://` base URL from needing a
|
||
key but demands one for an `https://` hostname (`missing-provider-auth`), so both carry
|
||
the placeholder `apiKey: forgeway-no-auth`. It is not a credential; the router ignores
|
||
the header. Side effect: `openclaw doctor --lint` now lists those two `apiKey` paths
|
||
among its "plaintext secret-bearing config fields" — a false positive for these two,
|
||
alongside the pre-existing entries it already flagged.
|
||
|
||
## Findings on the way (none caused by this work)
|
||
|
||
1. **Both LXC Pi-hole v6 APIs have no password set** — `172.19.1.201` and `172.24.1.201`
|
||
both answer `/api/auth` with `"no password set"`. Any host that can reach them can
|
||
rewrite their DNS config without credentials. It made today's LXC edit trivial; it is
|
||
worth a password. The k8s Pi-holes require auth.
|
||
2. **The Serenity Kuma LXC's default resolver is dead for internal names.** It resolves via
|
||
the UDM (`192.168.24.1`), which times out on every `falkvinge.net` name, while both
|
||
Serenity Pi-holes answer it directly and instantly. Its monitors work because they use
|
||
IPs. Same class as the Lounge Kuma resolv.conf note.
|
||
3. **Lounge `.1.202` lags Serenity zone changes by up to 15 minutes.** It is a tier-two
|
||
secondary (Serenity primary → `.10.202` → `.1.202`) and `.10.202` sends no NOTIFY, so
|
||
`.1.202` only refreshes on its 900 s SOA timer, and the Lounge Pi-holes round-robin
|
||
both Technitiums. For a quarter of an hour after any Serenity record is created, Lounge
|
||
clients get it about half the time — then negative-cache the miss for another 900 s.
|
||
I resynced `.1.202` by API for today; the log confirms it. Optional hygiene, your call:
|
||
`notify: SpecifiedNameServers → 172.19.1.202` on `.10.202`'s two Serenity secondaries.
|
||
|
||
I initially misread this last one as a month-old broken replica; the serials were simply
|
||
the primary's last-change dates. Corrected here and in memory.
|
||
|
||
## Where everything lives (additions)
|
||
|
||
| | |
|
||
|---|---|
|
||
| Lounge ingress | `lounge-cluster-config/apps/forgeway/manifests/ingress.yaml` — commits `674d09e`, `15b226e` |
|
||
| Serenity ingress + Pi-hole override | `serenity-cluster-config/apps/forgeway/ingress.yaml`, `apps/pihole/pihole.yaml` — `82fef33`, `f44dbb8`, `532ba68` |
|
||
| Technitium records | live on `172.19.1.202` and `172.24.1.202` (zone data is not in git by design) |
|
||
| LXC Pi-hole override | live on `172.24.1.201` via API (`dns.hosts` + `misc.dnsmasq_lines`) |
|
||
| Technitium helper | `scripts/technitium.py` (zones / records / add / resync / logtail) |
|
||
|
||
---
|
||
|
||
# Addendum 2, 2026-09-18 evening — native Ollama dialect + Home Assistant
|
||
|
||
**Done and proven from Home Assistant itself.** forgeway 0.2.1 serves Ollama's native API
|
||
on both modes beside the OpenAI one. The Lounge HA (2026.9.2) now has an Ollama
|
||
integration entry pointed at forgeway, a conversation agent
|
||
`conversation.forgeway_optimistic` on `gpt-oss:120b`, and a real question asked through
|
||
HA's own conversation API was answered by forge:
|
||
|
||
> Q: In one short sentence, what is Home Assistant?
|
||
> A: *Home Assistant is an open-source platform for local home automation and smart-device control.*
|
||
|
||
(64 s including a cold model load; the pod logs show no cloud fallback in the window.)
|
||
|
||
| | Lounge | Serenity |
|
||
|---|---|---|
|
||
| Image | `0.2.1` | `0.2.1` |
|
||
| Acceptance | **22/22** (13 OpenAI + 9 Ollama) | **22/22** |
|
||
| Ollama strict-503 / optimistic cloud fallback / cloud NDJSON stream | 503 / 200 `backend=cloud` / 3 frames, `done:true` | same |
|
||
|
||
## What HA actually calls — from its source, not guessed
|
||
|
||
HA 2026.9.2 pins `ollama==0.6.2`. The integration calls **`list()` → `GET /api/tags`**
|
||
(setup, config-flow validation, model picker; **5 s timeout**) and
|
||
**`chat(stream=True)` → `POST /api/chat`** with `keep_alive: -1`, `options: {num_ctx: 8192}`,
|
||
`think: false` (default) and `tools` when the Assist API is on. It calls `pull()` only
|
||
for a model absent from tags, which cannot happen (tags list exactly `gpt-oss:120b`). It
|
||
reads `message.content`, `message.thinking`, `message.tool_calls`, `done`,
|
||
`done_reason`. The client keeps the URL path, so `https://<host>/local-optimistic` is a
|
||
valid server URL — verified with the pinned client against the live service.
|
||
|
||
## The blocker that changed the URL: HA cannot reach the ingress
|
||
|
||
Rick asked for `https://forgeway.services.falkvinge.net/local-optimistic` pasted into HA.
|
||
From HA's VLAN (`192.168.120.0/22`) the alias resolves correctly to `172.19.10.11` and
|
||
then **fails to connect** — that VLAN reaches nothing in `172.19.10.x` and not forge;
|
||
it does reach the entrance HAProxy and the internet. The magpie jump to pfSense is down
|
||
right now, so I could neither read the rule set nor add a rule.
|
||
|
||
What works today, with a valid certificate and existing firewall openings:
|
||
**`https://forgeway.dmz.lounge.falkvinge.net/local-optimistic`** → entrance HAProxy →
|
||
`172.19.10.90:80` (the band opened for exactly this). The entrance cert covers
|
||
`*.dmz.lounge.falkvinge.net`; it does **not** cover `*.lounge.falkvinge.net`, which
|
||
corrects an assumption from this morning. Changes made:
|
||
|
||
- `lounge-haproxy-config` (`08843be`, `27ce8e4`): vhost `forgeway.dmz.lounge.falkvinge.net`
|
||
→ backend `172.19.10.90:80`, `timeout server 600s`, and — because the entrance is a
|
||
public edge and the router has no auth — `http-request deny` for non-RFC1918 sources.
|
||
Deployed by the autopull. An intermediate commit (`596d68b`) failed `haproxy -c` (ACL
|
||
used before declared); the autopull rolled it back as designed and the edge never
|
||
stopped serving. Rick can confirm the applied revision in `/var/log/haproxy-autopull.log`.
|
||
- Lounge Technitium: `forgeway.services` and `forgeway.lounge` now CNAME straight to the
|
||
ingress; `forgeway.dmz.lounge` CNAMEs to `entrance.dmz.falkvinge.net`. The alias and
|
||
per-site name are unchanged in effect; only the `.dmz.lounge.` hop moved.
|
||
|
||
> **If you want HA on the `.services.` alias as originally asked:** pfSense, pass
|
||
> `192.168.120.12` (or the sandbox net) → `172.19.10.11` TCP 443. Then
|
||
> `scripts/ha_setup_ollama.py … https://forgeway.services.falkvinge.net/local-optimistic`.
|
||
> I could not verify the source fence from the public side; it relies on pfSense not
|
||
> rewriting inbound sources, which is the default.
|
||
|
||
## Design points in the dialect
|
||
|
||
- **forge path = raw proxy.** forge is Ollama; frames are forge's own, model pinned.
|
||
- **Cloud fallback translated both ways**, streaming as NDJSON with Ollama's compact
|
||
byte shape; tool calls round-trip with synthesised ids; `format` mapped; usage → counts.
|
||
- **`think` → `reasoning_effort`**, and **forced to `none` when tools are present**: the
|
||
smoke test caught OpenAI rejecting `reasoning_effort` with function tools on
|
||
gpt-5.4-mini — precisely the shape HA sends with the Assist API. Fixed before shipping.
|
||
- **Reasoning trap**: gpt-oss returns `content` and `thinking` separately whatever
|
||
`think` is, and HA sets no output budget, so answers are never empty. Where a caller
|
||
does starve content with a tiny `num_predict`, the router flags it
|
||
(`x-forgeway-warning: content-starved-by-reasoning`) rather than rewriting budgets.
|
||
- **Tags never block on forge** (2.5 s cap, 60 s cache, synthetic entry if unreachable) —
|
||
HA's 5 s list timeout would otherwise trip on every cold load.
|
||
- **OpenAI dialect untouched**: checks 1–13 identical to this morning.
|
||
|
||
## For Rick to do (only if wanted)
|
||
|
||
1. Make it the default assistant: *Settings → Voice assistants → Assist → Conversation
|
||
agent → "forgeway optimistic"*. Left as-is on purpose.
|
||
2. Let it control the house: on the Ollama integration's "forgeway optimistic" subentry,
|
||
tick *Control Home Assistant*. Left off on purpose — that hands device control to an
|
||
LLM.
|
||
3. Optional: the pfSense rule above, to use the alias instead of the entrance.
|
||
|
||
## Where (additions)
|
||
|
||
| | |
|
||
|---|---|
|
||
| Dialect | `app/ollama_api.py`; shared bits in `app/common.py` |
|
||
| HA setup + proof | `scripts/ha_setup_ollama.py` (idempotent; ends with a real conversation) |
|
||
| Local smoke | `scripts/local-smoke-ollama.sh` (uses `ollama==0.6.2`, the version HA pins) |
|
||
| Edge vhost | `lounge-haproxy-config` `27ce8e4` |
|
||
| Cluster repos | `lounge-cluster-config` `c2c2c3e`, `serenity-cluster-config` `f35dc0a` |
|
||
|
||
---
|
||
|
||
# Addendum 3, 2026-09-18 late — both privacy flavours in Home Assistant, on the alias
|
||
|
||
Rick opened `192.168.120.12 → 172.19.10.11` on pfSense. Both HA Ollama entries now sit on
|
||
the split-horizon alias, named so a dropdown reads as a data-residency choice:
|
||
|
||
| Agent name (what a dropdown shows) | Entity | Server URL |
|
||
|---|---|---|
|
||
| **Local only — private (fails if forge is down)** | `conversation.forgeway_local_only` | `https://forgeway.services.falkvinge.net/local-strict` |
|
||
| **Local preferred — falls back to OpenAI cloud** | `conversation.forgeway_local_preferred` | `https://forgeway.services.falkvinge.net/local-optimistic` |
|
||
|
||
Entry titles: *Forgeway local-strict @ … — local only, private (fails if forge is down)*
|
||
and *Forgeway local-optimistic @ … — local preferred (falls back to OpenAI cloud)*. Model
|
||
`gpt-oss:120b` on both, no API key, think off, no device control, neither made the
|
||
default assistant. The optimistic entry already existed when I started (created on
|
||
Rick's side after the pfSense change, agent "Forge Conversation"); I reused it and
|
||
renamed entry, device and entity rather than duplicating it. HA has no API to rename a
|
||
subentry, so its subentry list still says "Forge Conversation" — cosmetic, UI-renameable.
|
||
My earlier `.dmz.lounge` entry was already gone.
|
||
|
||
**Created programmatically**, no hands needed: REST config flows for entries and
|
||
subentries, WebSocket for renames. One trap found and fixed in the tooling: renaming an
|
||
entry triggers a reload, and a subentry flow started during it aborts with
|
||
`entry_not_loaded` — wait for `state: loaded` first.
|
||
|
||
## Proof by behaviour, through HA's own conversation API
|
||
|
||
| | forge available | forge unavailable |
|
||
|---|---|---|
|
||
| Local only (strict) | *"Home Assistant is an open-source platform that lets you automate and control smart home devices."* (33 s, from forge) | **no answer, no cloud** — HA raised its Ollama error (REST surface: HTTP 500 from the conversation API; the integration logs *Unexpected error talking to Ollama server* with the 503 body per its source — I could not read HA's log to quote it, `/api/error_log` is gone in 2026.9) |
|
||
| Local preferred (optimistic) | *"Home Assistant is an open-source home automation platform that lets you control and automate smart devices locally."* (66 s, from forge; pod log shows no fallback) | *"Home Assistant is an open-source home automation platform …"* in **1.3 s**, router log: *falling back to cloud gpt-5.4-mini* |
|
||
|
||
The "unavailable" column: a throwaway forgeway instance with upstream `127.0.0.1:1`
|
||
published under a temporary `forgeway-deadproof.services.falkvinge.net` (Ingress + a
|
||
temporary Technitium CNAME), two temporary HA entries pointed at it, one question through
|
||
each, then everything removed — entries, Ingress, Service, Deployment, DNS record. Ollama
|
||
on forge was never touched.
|
||
|
||
## Route and DNS, verified from HA's side rather than assumed
|
||
|
||
- The alias is **NXDOMAIN on 1.1.1.1 and 9.9.9.9** and answers only on `172.19.1.201` /
|
||
`172.19.10.201` (→ `ingress.dmz.lounge.falkvinge.net` → `172.19.10.11`). HA's entries
|
||
*loading* (which requires `/api/tags` over TLS with the `*.services.` cert) is therefore
|
||
proof that HA resolves the name through the internal resolvers. HA's Supervisor API
|
||
(which would show the resolver list directly) refuses the long-lived token (401), so
|
||
this behavioural proof is the strongest available.
|
||
- The pfSense rule covers HA's IP only: the scripthost on the same VLAN still cannot
|
||
connect to `172.19.10.11`. Intended, and a useful negative control.
|
||
- The entrance-HAProxy path via `forgeway.dmz.lounge.falkvinge.net` stays as a documented,
|
||
fenced alternative; nothing uses it now. Say the word and I remove the vhost and record.
|
||
|
||
## Nothing left for Rick unless wanted
|
||
|
||
- Default assistant: *Settings → Voice assistants → Assist → Conversation agent*.
|
||
- Device control: tick *Control Home Assistant* on a subentry (deliberately off).
|
||
- Cosmetic: rename the optimistic subentry "Forge Conversation" in the UI.
|
||
|
||
---
|
||
|
||
# Addendum 4, 2026-09-18 night — converged on the approved design
|
||
|
||
Rick approved: pfSense rule for HA, drop the entrance vhost. State was re-established
|
||
with evidence before acting, then converged. **Who did what:**
|
||
|
||
| | Done by | Evidence |
|
||
|---|---|---|
|
||
| pfSense rule `192.168.120.12 → 172.19.10.11` | **Rick** (his statement; I could not add or even read it — the magpie jump to pfSense is down, `ssh -p 2222` refused) | both HA entries on the alias load and answer; the scripthost on the same VLAN still cannot connect (rule is HA-IP-scoped) |
|
||
| Entrance HAProxy vhost | **me**: added earlier today, **removed now** — `lounge-haproxy-config` `73f4389`, applied by the autopull at 19:05:07 UTC | before: outside vantage (Serenity's public egress → Lounge WAN) got the 403 fence; after: nginx 404 from the default backend; the `git` control vhost answered 200 on every 10 s tick through the change — no blip |
|
||
| `forgeway.dmz.lounge.falkvinge.net` (DNS + Ingress host) | **me**: created for the edge path, **retired now** (`lounge-cluster-config` `b77c0d1`) | authoritative answer empty; live Ingress hosts = `forgeway.services` + `forgeway.lounge` |
|
||
| HA entries on the alias | optimistic entry created on **Rick's side** (I renamed entry/device/entity); strict entry created by **me** | see below |
|
||
|
||
## HA end state, proven after the vhost was gone
|
||
|
||
| Agent (dropdown) | Entity | Server URL | Real conversation after 19:05:07 |
|
||
|---|---|---|---|
|
||
| **Local only — private (fails if forge is down)** | `conversation.forgeway_local_only` | `https://forgeway.services.falkvinge.net/local-strict` | *"Home Assistant is an open-source platform that lets you automate and control smart-home devices from a single interface."* (73 s) |
|
||
| **Local preferred — falls back to OpenAI cloud** | `conversation.forgeway_local_preferred` | `https://forgeway.services.falkvinge.net/local-optimistic` | *"Home Assistant is a home automation platform that lets you control and automate smart devices locally."* (25 s) |
|
||
|
||
The ingress-nginx access log attributes these to HA by its client signature
|
||
`ollama-python/0.6.2 (aarch64 linux) Python/3.14.6` (HA Yellow; every test of mine is
|
||
curl or x86_64), posting to `/local-strict/api/chat` and `/local-optimistic/api/chat`
|
||
against upstream service `forgeway-forgeway-80`:
|
||
|
||
```
|
||
19:06:25 UTC POST /local-strict/api/chat -> 200 (73.3 s) conversation.forgeway_local_only
|
||
19:06:50 UTC POST /local-optimistic/api/chat -> 200 (24.6 s) conversation.forgeway_local_preferred
|
||
```
|
||
|
||
(A third HA-originated optimistic conversation completed at 19:05:27; not one I
|
||
started.) With the edge vhost gone, the ingress is the only route a `.services.` name has.
|
||
|
||
**Strict genuinely fails when forge is unavailable — re-proven through HA** with a
|
||
throwaway dead-upstream instance published under a temporary `.services.` name: the
|
||
strict temp agent produced no answer and no cloud call (HA raised its Ollama error; REST
|
||
surface HTTP 500), the optimistic temp agent answered in 1.3 s with the router logging
|
||
*falling back to cloud gpt-5.4-mini*. Temporaries (HA entries, Ingress, Service,
|
||
Deployment, DNS record) removed; the namespace holds only forgeway. Ollama on forge
|
||
was never touched.
|
||
|
||
Nothing is left for Rick unless he wants the optional bits from Addendum 3 (default
|
||
assistant, device control, cosmetic subentry rename).
|
||
|
||
---
|
||
|
||
# Addendum 5, 2026-09-18 — sequencing check after the fact, and the resolution paths
|
||
|
||
A sequencing constraint arrived after Addendum 4 had executed: *prove the pfSense rule
|
||
from HA first, move the entries to the alias second, remove the vhost only then, and if
|
||
the rule cannot be proven, leave the vhost.* It was worth re-establishing the state
|
||
rather than arguing, so here is the timeline from HA's own `created_at` values and the
|
||
ingress access log (all UTC, 2026-09-18):
|
||
|
||
| Time | Event | Evidence |
|
||
|---|---|---|
|
||
| 18:26 | My first HA entry, on `forgeway.dmz.lounge` via the entrance vhost | Addendum 2 |
|
||
| ≈18:35 | **Rick** adds the pfSense rule `192.168.120.12 → 172.19.10.11` and removes that entry | his statement; the entry was gone by 18:42 |
|
||
| **18:39:18** | HA entry `01M2TX1P` created on **`forgeway.services…/local-optimistic`** (Rick's side) | HA `created_at`; loading requires `GET /api/tags` over TLS to `172.19.10.11`, i.e. **the rule works, proven from HA itself** |
|
||
| **18:44:34** | HA entry `01M2TXBA` created on **`forgeway.services…/local-strict`** (me) | HA `created_at`, same proof |
|
||
| ≈18:50–18:58 | Real conversations through both alias agents, and the dead-upstream difference test through HA | Addendum 3 |
|
||
| **19:05:07** | Entrance vhost removed by the autopull | edge watch, Addendum 4 |
|
||
| 19:06:25 / 19:06:50 | Both alias agents answer again | ingress log, HA's client UA |
|
||
| 19:12:06 / 19:12:08 | Both alias agents answer again (this check) | ingress log; forgeway pod log shows no fallback, and strict cannot fall back by design |
|
||
|
||
So the order 1 → 2 → 3 held: the rule was proven from HA at 18:39, both entries were on
|
||
the alias by 18:44, and the vhost went at 19:05. **No HA entry referenced
|
||
`forgeway.dmz.lounge` at any point after ≈18:35**, so the vhost was not "the only thing
|
||
making HA's agent work" when it was removed — nothing depended on it. The one thing I
|
||
would do differently: run a fresh conversation *immediately* before the removal rather
|
||
than relying on the ones from fifteen minutes earlier. Nothing had changed in between,
|
||
but a proof that is minutes old is cheaper than an argument.
|
||
|
||
## The resolution paths, recorded so nobody re-derives them
|
||
|
||
From the Lounge client resolvers `172.19.1.201` / `172.19.10.201`, now:
|
||
|
||
```
|
||
forgeway.services.falkvinge.net → ingress.dmz.lounge.falkvinge.net → 172.19.10.11 (k8s ingress-nginx; local-first alias)
|
||
forgeway.lounge.falkvinge.net → ingress.dmz.lounge.falkvinge.net → 172.19.10.11 (explicit Lounge instance; http only)
|
||
forgeway.serenity.falkvinge.net → forgeway.dmz.serenity… → ingress.dmz.serenity… → 172.24.10.11 (explicit Serenity instance)
|
||
forgeway.dmz.lounge.falkvinge.net → NXDOMAIN (retired; .10.201 held a cached answer for ≤5 min)
|
||
```
|
||
|
||
Until 19:03 the retired name resolved `→ entrance.dmz.falkvinge.net → demuxer.dmz.falkvinge.net → 172.20.0.101`,
|
||
the entrance HAProxy — a genuinely different path from the alias. That is why, earlier
|
||
today, "HA's VLAN can't reach `172.19.10.x`, so use the `.dmz.lounge.` name" was a
|
||
coherent instruction at all: the two names went to different places. The claim itself
|
||
was verified at the time from the scripthost on HA's VLAN (`172.19.10.11` unreachable,
|
||
entrance reachable — Addendum 2), not from HA itself; Rick was right to ask for the
|
||
resolution paths to be stated rather than implied.
|
||
|
||
Today the alias goes to the ingress from HA because of Rick's pfSense rule, which is
|
||
scoped to HA's IP: the scripthost on the same VLAN still gets no connection to
|
||
`172.19.10.11:443` (checked again now). The entrance path is gone in config
|
||
(`lounge-haproxy-config` `73f4389`), from inside, and from outside (Serenity's public
|
||
egress → the Lounge WAN address → the edge's default backend, 404).
|
||
|
||
## Found while closing: one spurious cloud fallback at 19:10:53 — forge was alive
|
||
|
||
While confirming the 19:12 answers came from forge, the Lounge pod log showed one
|
||
earlier line: `19:10:53 falling back to cloud gpt-5.4-mini for /api/chat
|
||
(completion-failed: ReadTimeout)`. Forge was not down. Its journal for the minute
|
||
before shows **four full reloads of the 64 GB model in two minutes**, each 15–30 s,
|
||
alternating between two context sizes:
|
||
|
||
```
|
||
19:09:20 runner loaded n_ctx = 8192 (HA chat; HA sends options.num_ctx 8192)
|
||
19:09:51 reloaded n_ctx = 131072 (router health probes: default context)
|
||
19:10:23 reloaded n_ctx = 8192 (HA strict chat — answered 19:10:25, 62 s end to end)
|
||
19:10:53 reloaded n_ctx = 131072 (probe) — the roci-03 pod's probe hit its 45 s limit
|
||
at that exact second → "unavailable" → optimistic
|
||
request answered by the cloud (HA saw it 46 s later)
|
||
19:11:19 reloaded n_ctx = 131072 again (the interrupted load was torn down)
|
||
```
|
||
|
||
Three router instances (two Lounge pods, one Serenity) each probe forge every minute,
|
||
all at the default context, while HA asks for 8192 — so every alternation is a reload,
|
||
and a probe queued behind a live generation plus a reload can exceed 45 s. Effects:
|
||
a **"Local preferred" answer silently came from the cloud while forge was working**,
|
||
and a "Local only" answer took a minute. The Serenity instance's probe, arriving a
|
||
second later, succeeded in 7 s. This is the runner thrash already noted in memory
|
||
(`reference_ollama_num_ctx_runner_thrash`); what is new is that it trips the router's
|
||
availability verdict, which is privacy-relevant on the optimistic endpoint.
|
||
|
||
**Not changed by me.** The fix I recommend: make the router pin `options.num_ctx` on
|
||
every forge-bound request (probe, OpenAI dialect, Ollama dialect) to one configured
|
||
value, default 131072 — the size the model already runs at, so no extra VRAM — which
|
||
ends the reloads entirely; HA's 8192 is a runner-size hint, not a prompt limit (HA trims
|
||
history by message count). Router-only, one config knob, a 0.2.2 image. The alternate
|
||
is setting `num_ctx` to 131072 in each HA subentry; it has the same effect but has to
|
||
be repeated per agent and is invisible in git. I stopped short of either because it
|
||
overrides a user-visible HA setting and the main session is tracking the thrash.
|
||
|
||
---
|
||
|
||
# Addendum 6, 2026-09-18 — `num_ctx` pinned at the router (0.2.2), verified by behaviour
|
||
|
||
**Done and shipped to both clusters.** Every forge-bound request from forgeway now carries
|
||
`options.num_ctx = 131072` (`FORGEWAY_NUM_CTX`), overriding the caller's value, and the
|
||
health probe carries the same value via Ollama's native `/api/chat`. `/status` exposes
|
||
`num_ctx_pin` and `runner_context` and the router warns if they differ. Vikunja card
|
||
#190 (<https://progress.falkvinge.net/tasks/190>), moved to Done.
|
||
|
||
## Value: 131072, and why it is not a compromise
|
||
|
||
| Consideration | Finding |
|
||
|---|---|
|
||
| Retrieval workload | needs 40k–100k-token prompts; gpt-oss's trained window is 131072 (`gptoss.context_length`) |
|
||
| VRAM | an *unpinned* request already loads at 131072 (forge has no `OLLAMA_CONTEXT_LENGTH`; Ollama 0.34 uses the model's trained length), so the pin adds nothing: runner 61,466 MiB resident, 66.7 GB used of 97.9 GB on card 1 |
|
||
| Short turns | a large *allocated* context does not slow them: 1-token probes 60–100 ms, HA turns 0.3–0.6 s end to end, scripted short turns 0.5 s |
|
||
| OpenAI dialect | Ollama's `/v1` endpoint **ignores `options.num_ctx`** (tested on qwen2.5vl: asked 6144, loaded 128000; native `/api/chat` asked 5120, loaded 5120), so OpenAI callers get the model default — the pin equals it, and `runner_context` is the tripwire |
|
||
|
||
One value serves both workloads; no alternative needed.
|
||
|
||
## Before (0.2.1, 19:29–19:33 UTC) — an 8192-sending caller alternating with the probe
|
||
|
||
Forge's journal for three rounds: `n_ctx` sequence **`8192 131072 8192 131072 8192 131072`**,
|
||
**7 model loads**, and forge's own request log:
|
||
|
||
```
|
||
19:30:09 200 32.6 s /api/chat (8192 caller: reload + answer)
|
||
19:30:30 499 47.0 s /v1/chat/completions (probe timed out during the reload back)
|
||
19:30:33 499 46.9 s /v1/chat/completions (Serenity's probe, same)
|
||
19:31:33 200 27.9 s /api/chat
|
||
19:32:41 200 28.8 s /api/chat
|
||
19:33:16 200 33.2 s /v1/chat/completions (probe)
|
||
```
|
||
|
||
Every alternation cost a reload; two probes hit the 45 s limit — the spurious
|
||
"unavailable" that produced the 19:10:53 cloud fallback. (HA itself was already on
|
||
131072 since the main session's 19:21 subentry fix; its turns measured 0.3–0.6 s
|
||
before and after — the router pin makes that independent of what any caller sends.)
|
||
|
||
## After (0.2.2, 19:36:59–19:38:28 UTC) — the mix that used to thrash, one window
|
||
|
||
```
|
||
HA-style turn (num_ctx 8192.0), strict 0.50 s
|
||
health probe 0.13 s
|
||
retrieval ~40k tokens, OpenAI dialect 7.61 s prompt_tokens=38484 needle FOUND
|
||
HA-style turn (num_ctx 8192.0), optimistic 2.03 s
|
||
retrieval ~100k tokens, OpenAI dialect 23.01 s prompt_tokens=96175 needle FOUND
|
||
health probe 0.01 s
|
||
HA-style turn (num_ctx 8192.0), strict 0.50 s
|
||
scripted caller num_ctx 4096, generate 0.58 s
|
||
HA-style turn (num_ctx 8192.0), optimistic 1.08 s
|
||
real Home Assistant turns ×5 0.6 / 0.5 / 0.4 / 0.3 / 0.4 s
|
||
```
|
||
|
||
Forge's journal for that window: **`n_ctx` (re)loads: none; model loads: 0; timeouts: 0**;
|
||
`ollama ps`: `gpt-oss:120b`, context 131072, 61,466 MiB. The model was never reloaded
|
||
across 8192-, 4096-, default- and 100k-token callers. (Traffic from `127.0.0.1` on forge in
|
||
the same window is the engine-evaluation lane, not mine; left alone.)
|
||
|
||
| | before | after |
|
||
|---|---|---|
|
||
| reloads per 8192↔default alternation | 1 (15–30 s) | 0 |
|
||
| turn with an off-value `num_ctx` | 27–33 s | 0.5 s |
|
||
| probe during contention | 27–47 s, two timeouts | 0.01–0.13 s |
|
||
| HA turn end to end | 0.3–0.6 s (already fixed on HA's side) | 0.3–0.6 s, now caller-independent |
|
||
|
||
## Acceptance
|
||
|
||
23/23 on Lounge and 23/23 on Serenity (13 OpenAI unchanged, 9 Ollama, plus check 23:
|
||
`num_ctx_pin == runner_context`). Both instances report `pin=131072 runner=131072`.
|
||
Ollama on forge was not stopped; ComfyUI on card 0 untouched (559 MiB, idle).
|
||
|
||
## Follow-up (not done, from the main session's note)
|
||
|
||
The hot-path gate still runs a real completion before every proxied request (cached 5 s).
|
||
With the pin it no longer thrashes, but a probe that needs a cold load still cannot fit
|
||
in 45 s under contention and fails the optimistic request over to the cloud. The cleaner
|
||
design — treat "model resident per `/api/ps`" as available on the hot path and keep the
|
||
real completion for `/healthz` only — is a semantic change to what "available" means and
|
||
stays Rick's call.
|