Ollama keeps one runner config per model and reloads the 64 GB model on every request whose num_ctx differs from the loaded runner's. The router now sets options.num_ctx = FORGEWAY_NUM_CTX (default 131072) on the Ollama dialect and carries the same value in the probe (native /api/chat; the OpenAI /v1 endpoint ignores options, verified). /status exposes the pin and the observed runner context and warns on mismatch; check 23 asserts it. Verified by behaviour: an 8192 caller, a 4096 caller, probes and 40k/100k- token retrieval prompts in one window -> 0 reloads, 0 timeouts, HA turns 0.3-0.6 s; before, 3 rounds of an 8192 caller = 7 loads and two probe timeouts. 23/23 acceptance on both clusters. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
435 lines
23 KiB
Markdown
435 lines
23 KiB
Markdown
# forgeway
|
||
|
||
An LLM router in front of forge's Ollama `gpt-oss:120b`, speaking **both** the OpenAI
|
||
API and Ollama's native API on the same two endpoints. One instance runs **per
|
||
cluster**, not one central service.
|
||
|
||
> **Names (confirmed by Rick, 2026-09-18).** Explicit per-site names
|
||
> `forgeway.lounge.falkvinge.net` and `forgeway.serenity.falkvinge.net` (Chalet later:
|
||
> `forgeway.chalet.falkvinge.net`), plus the split-horizon local-first alias
|
||
> **`forgeway.services.falkvinge.net`**, which resolves to whichever instance is local to
|
||
> the querying client's site. `.services.` names are internal-only by convention: no
|
||
> entrance SAN, no external exposure, ever.
|
||
|
||
## The two endpoints
|
||
|
||
| Endpoint | Behaviour when forge is available | Behaviour when it is not |
|
||
|---|---|---|
|
||
| `/local-optimistic/v1` | serves from forge | falls back to OpenAI `gpt-5.4-mini` |
|
||
| `/local-strict/v1` | serves from forge | clean HTTP **503**, no fallback, by design |
|
||
|
||
Each endpoint speaks two dialects:
|
||
|
||
| Dialect | Paths | Typical caller |
|
||
|---|---|---|
|
||
| OpenAI | `/{endpoint}/v1/chat/completions`, `/{endpoint}/v1/models` | scripts, the OpenAI SDK, OpenClaw |
|
||
| Ollama (native) | `/{endpoint}/api/chat`, `/api/generate`, `/api/tags`, `/api/version`, `/api/show`, `/api/ps` | Home Assistant's Ollama integration, any Ollama client |
|
||
|
||
Streaming works in both: SSE for the OpenAI dialect, newline-delimited JSON for the
|
||
Ollama one (where `stream` defaults to **true**, as in Ollama itself).
|
||
|
||
Where they live:
|
||
|
||
| Site | Cluster | Use this | Raw LB IP (DNS-independent) |
|
||
|---|---|---|---|
|
||
| any | — | `https://forgeway.services.falkvinge.net` → the instance local to you | — |
|
||
| Lounge | Rocinante | `http://forgeway.lounge.falkvinge.net` | `http://172.19.10.90` |
|
||
| Serenity | Niðavellir | `http://forgeway.serenity.falkvinge.net` | `http://172.24.10.21` |
|
||
| Chalet | — | when the cluster exists | — |
|
||
|
||
The alias is the front door. Use the per-site names only when you deliberately want a
|
||
*specific* site's instance (for example, asking the Lounge one from Serenity). HTTPS with
|
||
a valid certificate is available on the alias at both sites; the per-site names are
|
||
`http://` only — the internal ingress certificates carry `*.services.falkvinge.net` but
|
||
not `*.lounge.` / `*.serenity.` (adding those SANs to the two *internal* ingress
|
||
certificates is a one-line cert-manager change if ever wanted; the entrance bundle is
|
||
not the place).
|
||
|
||
### Calling them
|
||
|
||
**`local-strict` — do not retry a 503 blindly.** A 503 means the local model is
|
||
unavailable *right now*: forge is off, Ollama is stopped, or the model is evicted for
|
||
a training run. None of those clear in the next second, and retrying just adds load
|
||
to something already struggling. Handle the error, or degrade to your own fallback.
|
||
|
||
```bash
|
||
# strict: one attempt, handle the 503 yourself
|
||
curl -sS -X POST https://forgeway.services.falkvinge.net/local-strict/v1/chat/completions \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{"model":"gpt-oss:120b","messages":[{"role":"user","content":"hello"}]}' \
|
||
-w '\n%{http_code}\n'
|
||
```
|
||
|
||
```python
|
||
import httpx
|
||
|
||
# strict — no retries on 503; that status is an answer, not a hiccup
|
||
with httpx.Client(base_url="https://forgeway.services.falkvinge.net/local-strict/v1", timeout=600) as c:
|
||
r = c.post("/chat/completions", json={
|
||
"model": "gpt-oss:120b",
|
||
"messages": [{"role": "user", "content": "hello"}],
|
||
})
|
||
if r.status_code == 503:
|
||
... # local model is down: skip, queue, or use your own fallback
|
||
else:
|
||
r.raise_for_status()
|
||
print(r.json()["choices"][0]["message"]["content"])
|
||
|
||
# optimistic — fallback is already handled inside the router.
|
||
# Do NOT add your own retry loop: a failure here means forge AND the cloud both failed.
|
||
with httpx.Client(base_url="https://forgeway.services.falkvinge.net/local-optimistic/v1", timeout=600) as c:
|
||
r = c.post("/chat/completions", json={
|
||
"model": "gpt-oss:120b",
|
||
"messages": [{"role": "user", "content": "hello"}],
|
||
})
|
||
r.raise_for_status()
|
||
print(r.headers["x-forgeway-backend"]) # "forge" or "cloud"
|
||
```
|
||
|
||
With the OpenAI SDK, point `base_url` at one of the two prefixes and use any
|
||
non-empty api key (the router does not authenticate callers):
|
||
|
||
```python
|
||
from openai import OpenAI
|
||
client = OpenAI(base_url="https://forgeway.services.falkvinge.net/local-optimistic/v1", api_key="unused")
|
||
```
|
||
|
||
Every response carries `x-forgeway-backend: forge|cloud`, `x-forgeway-upstream`,
|
||
`x-forgeway-site` and `x-forgeway-mode`, so a caller can always tell what answered.
|
||
|
||
**Model selection.** The `model` field in a request is ignored — each endpoint routes
|
||
to its own backend. Send anything; `gpt-oss:120b` is conventional. `max_tokens` and
|
||
`max_completion_tokens` are interchangeable: the router translates between them,
|
||
because Ollama only accepts the first and `gpt-5.4-mini` rejects it outright.
|
||
|
||
### From OpenClaw agents
|
||
|
||
The same endpoints are registered as model providers (base URL = the `.services.`
|
||
alias over HTTPS, so agents get the local instance too), so agents and scripts use one
|
||
backend:
|
||
|
||
```
|
||
forgeway-strict/gpt-oss-120b # forge or an error
|
||
forgeway-optimistic/gpt-oss-120b # forge, else gpt-5.4-mini
|
||
```
|
||
|
||
|
||
## Home Assistant
|
||
|
||
HA's core *OpenAI Conversation* integration only talks to the official OpenAI endpoint,
|
||
so it cannot use forgeway. HA's core **Ollama** integration can: it takes an arbitrary
|
||
server URL, and its client keeps the URL path, so an endpoint prefix is a valid server.
|
||
|
||
**Lounge HA has both privacy flavours, configured and proven 2026-09-18.** Two separate
|
||
integration entries on the split-horizon alias, named so that picking one in a dropdown
|
||
reads as a data-residency choice rather than a mode name:
|
||
|
||
| What a dropdown shows (agent) | Entity | Integration entry | Server URL |
|
||
|---|---|---|---|
|
||
| **Local only — private (fails if forge is down)** | `conversation.forgeway_local_only` | *Forgeway local-strict @ forgeway.services.falkvinge.net — local only, private (fails if forge is down)* | `https://forgeway.services.falkvinge.net/local-strict` |
|
||
| **Local preferred — falls back to OpenAI cloud** | `conversation.forgeway_local_preferred` | *Forgeway local-optimistic @ forgeway.services.falkvinge.net — local preferred (falls back to OpenAI cloud)* | `https://forgeway.services.falkvinge.net/local-optimistic` |
|
||
|
||
Both: model `gpt-oss:120b`, think off, no API key, no device control (the *Control Home
|
||
Assistant* box is deliberately unticked — ticking it hands the LLM the Assist tools).
|
||
Neither is the default assistant; that is *Settings → Voice assistants → Assist →
|
||
Conversation agent*.
|
||
|
||
Proven by behaviour, through HA's own conversation API, not by "the integration loaded":
|
||
|
||
| | forge available | forge unavailable (dead-upstream instance, never by stopping Ollama) |
|
||
|---|---|---|
|
||
| Local only | real answer from forge | HA error, **no answer, no cloud** |
|
||
| Local preferred | real answer from forge | real answer in ~1 s, served by gpt-5.4-mini (router log: *falling back to cloud*) |
|
||
|
||
The unavailable column was produced by publishing a throwaway forgeway instance with a
|
||
dead upstream under a temporary `.services.` name, pointing two temporary HA entries at
|
||
it, asking through each, and removing all of it afterwards.
|
||
|
||
**Network prerequisites, verified from HA's side.** `.services.` names are internal-only
|
||
by convention: `forgeway.services.falkvinge.net` is NXDOMAIN on public DNS and answers
|
||
only on the internal `.201` resolvers, so an entry that *loads* proves HA resolved it
|
||
internally. HA lives on the sandbox VLAN, which cannot reach the k8s ingress by default;
|
||
**Rick added a pfSense rule** passing HA (`192.168.120.12`) to the ingress `172.19.10.11`
|
||
(HA's IP only — the neighbouring scripthost still cannot connect). Both entries loaded
|
||
and answered after that; the ingress access log shows HA's own client
|
||
(`ollama-python/0.6.2 (aarch64 linux)`) posting to both endpoints.
|
||
|
||
There is deliberately **no path through the entrance HAProxy any more.** A fenced vhost
|
||
(`forgeway.dmz.lounge.falkvinge.net`) existed for a few hours on 2026-09-18 while HA could
|
||
not reach the ingress; Rick chose the pfSense rule instead, and a public-edge listener in
|
||
front of an unauthenticated service must not survive by accident. The vhost, the DNS
|
||
record and the Ingress host are all removed, verified from outside (the hostname now falls
|
||
to the edge's default backend).
|
||
|
||
**What HA sends and how it is handled.** `list()` → `/api/tags` with a 5 s timeout, so
|
||
tags never wait on a model load (a cold `gpt-oss:120b` takes ~25–30 s); the list is
|
||
exactly one model, the local one. `chat(stream=True)` with `keep_alive: -1`,
|
||
`options: {num_ctx: 8192}`, `think: false`, and `tools` when the Assist API is enabled —
|
||
all passed to forge untouched. HA reads `message.content`, `message.thinking`,
|
||
`message.tool_calls`, `done`, `done_reason`; gpt-oss returns `content` and `thinking`
|
||
separately regardless of `think`, so a normal answer is never empty. HA shows the
|
||
thinking as a collapsible block.
|
||
|
||
Managing it headlessly (idempotent; every `setup` ends with a real conversation):
|
||
|
||
```bash
|
||
.venv/bin/python scripts/ha_ollama.py list
|
||
.venv/bin/python scripts/ha_ollama.py setup https://forgeway.services.falkvinge.net/local-strict \
|
||
--agent "Local only — private (fails if forge is down)" --title "…" --entity-id conversation.forgeway_local_only
|
||
.venv/bin/python scripts/ha_ollama.py ask conversation.forgeway_local_preferred "Is the kitchen light on?"
|
||
.venv/bin/python scripts/ha_ollama.py delete <url>
|
||
```
|
||
|
||
REST config flows create entries and subentries; the WebSocket API renames entries,
|
||
devices and entity ids. HA exposes neither `entry.data` (the URL) nor a subentry rename,
|
||
so entry titles carry the URL and mode, and the pre-existing optimistic subentry still
|
||
reads *Forge Conversation* in the integration's subentry list (rename it in the UI if it
|
||
bothers you; the agent name users see is the device/entity name above).
|
||
|
||
## One runner config per model: the `num_ctx` pin
|
||
|
||
Ollama keeps **one loaded runner configuration per model**, keyed on the request's
|
||
options. If one caller sends `options.num_ctx 8192` and the next sends nothing (→ the
|
||
model default, 131072 for `gpt-oss:120b`), Ollama evicts and fully reloads the 64 GB
|
||
model — 15–30 s — on *every* alternation. On 2026-09-18 that was Home Assistant (8192
|
||
by default) against the router's own health probes (default): four reloads in two
|
||
minutes, 33–66 s per HA turn, and one silent cloud fallback on `local-optimistic`
|
||
because a probe queued behind a reload exceeded its 45 s limit.
|
||
|
||
**forgeway therefore pins `num_ctx` on every forge-bound request** (`FORGEWAY_NUM_CTX`,
|
||
default **131072**), overriding whatever the caller sent, and the health probe carries
|
||
the same value via Ollama's native `/api/chat`. A caller's own `num_ctx` cannot
|
||
re-introduce the thrash; the router logs `pinning num_ctx 8192.0 -> 131072` when it
|
||
overrides one. `/status` reports `num_ctx_pin` and `runner_context` (what forge's
|
||
loaded runner was last seen with) and the router warns if they ever differ.
|
||
|
||
Why 131072 and not something smaller: it is the model's trained window and exactly
|
||
what an unpinned request loads anyway, so it costs no extra VRAM (KV ≈ 4.6 GB inside a
|
||
61 GB resident model on a 97 GB card); the retrieval workload that motivated the box
|
||
sends 40k–100k-token prompts; and a large *allocated* context does not slow short
|
||
turns — with the pin, HA turns measure 0.3–0.6 s end to end and a 1-token probe 60–100 ms.
|
||
|
||
One caveat, verified empirically: **Ollama's OpenAI-compatible `/v1` endpoint ignores
|
||
`options.num_ctx`**, so OpenAI-dialect callers (OpenClaw, scripts) always get the model
|
||
default. The pin must equal that default — it does today — and the `runner_context`
|
||
warning is the tripwire if an Ollama upgrade or an `OLLAMA_CONTEXT_LENGTH` setting on
|
||
forge ever changes it. Diagnose on forge in one line:
|
||
|
||
```bash
|
||
journalctl -u ollama --since "-20 min" | grep -oE "llama_context: n_ctx += *[0-9]+" | awk '{print $NF}' | tr '\n' ' '
|
||
# healthy: nothing, or a single value. thrash: 8192 131072 8192 131072 …
|
||
```
|
||
|
||
`scripts/verify_numctx_pin.py <router>` drives the exact mix that used to thrash (HA-style
|
||
turns, off-value scripted callers, probes, 40k and 100k-token needle-in-a-haystack
|
||
prompts) and prints latencies; pair it with the one-liner above for the window.
|
||
|
||
## The Ollama dialect, and the cloud fallback through it
|
||
|
||
forge *is* Ollama, so on the forge path the router is a raw proxy: the response —
|
||
including every streamed NDJSON frame — is forge's own, with only the `model` pinned.
|
||
Everything below is about the **cloud fallback** (`local-optimistic` only; `local-strict`
|
||
returns Ollama's error shape, `{"error": "..."}`, with status 503):
|
||
|
||
- The Ollama request is translated to an OpenAI chat completion and the answer is
|
||
translated back into Ollama's shape, non-streamed or as NDJSON frames byte-compatible
|
||
with Ollama's: `model`, `created_at` (nanosecond RFC 3339), `message.role/content`,
|
||
`done`, `done_reason` (`stop` / `length`), and the `*_duration` / `*_count` stats
|
||
(token counts from usage; durations that OpenAI does not report are 0).
|
||
- **Tool calls** round-trip. Ollama messages carry no tool-call ids and OpenAI insists on
|
||
them, so ids are synthesised in order and handed to the following `tool` messages in
|
||
order; OpenAI's JSON-string arguments become Ollama's objects. On the way out, OpenAI
|
||
streams tool calls as fragments; they are reassembled and emitted as one frame.
|
||
- `format: "json"` → `response_format: json_object`; a schema → `json_schema`.
|
||
- **`think` → `reasoning_effort`**: `false` → `low`, `true` → `medium`, a level → itself.
|
||
With **tools present it is forced to `none`**: gpt-5.4-mini rejects any other value
|
||
together with function tools on `/v1/chat/completions`, and HA's Assist API always
|
||
sends tools. The cloud model's reasoning text is not exposed by that API, so cloud
|
||
answers carry no `thinking`.
|
||
- `options.num_predict` → `max_completion_tokens`; `num_ctx` and `keep_alive` are
|
||
Ollama-only and dropped.
|
||
- `/api/generate` is served the same way (`prompt`/`system`/`images` become messages).
|
||
`/api/pull`, `/api/push`, `/api/create`, `/api/delete`, `/api/embed` return 400: this
|
||
endpoint serves exactly one model and manages none.
|
||
|
||
**The reasoning trap, decided.** A thinking model with a tiny output budget spends it all
|
||
thinking and returns empty `content` — that is what bit the OpenAI dialect at
|
||
`max_tokens: 10`. The router does not silently rewrite budgets; instead it makes the
|
||
failure visible: a non-streamed answer with thinking but no content gets the header
|
||
`x-forgeway-warning: content-starved-by-reasoning` and a log line. HA never sets a
|
||
budget, so it never hits this.
|
||
|
||
## DNS: how the alias finds the local instance
|
||
|
||
Every `.services.` name in the estate is a CNAME chain ending at a site's ingress-nginx,
|
||
and the authoritative `services.falkvinge.net` zone lives on the Lounge Technitium. So
|
||
at the Lounge nothing special is needed:
|
||
|
||
```
|
||
forgeway.services.falkvinge.net → ingress.dmz.lounge.falkvinge.net → 172.19.10.11
|
||
forgeway.lounge.falkvinge.net → ingress.dmz.lounge.falkvinge.net → 172.19.10.11
|
||
forgeway.serenity.falkvinge.net → forgeway.dmz.serenity.falkvinge.net → ingress.dmz.serenity.falkvinge.net → 172.24.10.11
|
||
```
|
||
|
||
Serenity's Pi-holes forward everything under `falkvinge.net` (except `serenity.`) to the
|
||
Lounge — so without an override, a Serenity client would silently get the **Lounge**
|
||
ingress and the remote instance, defeating the whole multi-site design. Both Serenity
|
||
Pi-holes therefore pin the alias to the Serenity ingress:
|
||
|
||
| Resolver | Where the override lives |
|
||
|---|---|
|
||
| `172.24.10.201` (k8s, two pods) | `serenity-cluster-config/apps/pihole/pihole.yaml` → `03-serenity-forwards.conf` |
|
||
| `172.24.1.201` (LXC) | Pi-hole v6 API: `dns.hosts` entry + `misc.dnsmasq_lines` |
|
||
|
||
Both carry `host-record=forgeway.services.falkvinge.net,172.24.10.11` **and**
|
||
`local=/forgeway.services.falkvinge.net/`. The `local=` line matters: without it an AAAA
|
||
query is still forwarded to the Lounge, and the day the Lounge ingress gains an IPv6
|
||
address every dual-stack Serenity client would prefer it and quietly reach the remote
|
||
instance — the same dual-stack bypass that defeated the Lounge overrides on 2026-08-24.
|
||
|
||
Each site's ingress serves `forgeway.services.falkvinge.net` for its own instance, so the
|
||
DNS answer alone decides which site you land on. Verified 2026-09-18 from real clients:
|
||
Lounge (OpenClaw VM, Lounge Kuma host) → `172.19.10.11`, `x-forgeway-site: lounge`;
|
||
Serenity (Niðavellir pod, both Serenity Pi-holes) → `172.24.10.11`,
|
||
`x-forgeway-site: serenity`. Never the reverse.
|
||
|
||
## Why one instance per cluster
|
||
|
||
A single central router would sit on one site. The cross-site link to forge is exactly
|
||
the kind of thing that goes down — and if the router lived on the far side of it, the
|
||
router *would be* the outage rather than the thing that reports it. So each cluster
|
||
runs its own, and each one answers for itself: when Serenity cannot reach forge,
|
||
Serenity's `local-optimistic` still serves from the cloud and its `local-strict` still
|
||
returns a clean 503, both without depending on anything at the Lounge.
|
||
|
||
It does **not** run on forge itself, for the same reason: a router that dies with its
|
||
upstream cannot report on it.
|
||
|
||
## Discovery — and what happens when silicon moves to Serenity
|
||
|
||
No instance is told which site forge is at. Each one holds an ordered candidate list
|
||
(`FORGEWAY_UPSTREAMS`) and finds forge at request time:
|
||
|
||
```
|
||
http://forge.dmz.falkvinge.net:11434 the stable name — DNS follows a host move
|
||
http://172.19.0.130:11434 forge at the Lounge (today)
|
||
http://172.24.0.130:11434 forge at Serenity (after the move)
|
||
```
|
||
|
||
Candidates are probed concurrently but awaited in priority order, so the happy path
|
||
costs one fast round trip and dead candidates never slow it down. The winner is sticky
|
||
and tried first next time.
|
||
|
||
**When silicon relocates to Serenity, nothing here needs rewriting.** Re-point
|
||
`forge.dmz.falkvinge.net` at its new address and both instances follow on their next
|
||
health refresh (seconds). If DNS is stale or broken, discovery still finds forge at the
|
||
Serenity literal already in the list. The only thing that changes materially is which
|
||
site pays the cross-site latency — and that flips automatically, because each instance
|
||
independently discovers the nearest working answer.
|
||
|
||
Two things do need a human when that happens:
|
||
|
||
1. **forge's ufw** — the allow rules are scoped to source ranges. Serenity-local
|
||
traffic will then arrive from `172.24.9.0/24` (already allowed); Lounge traffic
|
||
will arrive SNAT'd from the Serenity edge and will need its own rule.
|
||
2. **Kuma** — the monitors point at each local router by IP, so they keep working, but
|
||
the Lounge monitor will start reporting the slower cross-site path.
|
||
|
||
## Health and monitoring
|
||
|
||
`/healthz` performs a **real 1-token completion** against forge, never a ping. A ping
|
||
cannot tell the difference between a healthy forge and a forge that is up with Ollama
|
||
stopped — which was literally its state on the morning this was built.
|
||
|
||
| Result | Meaning |
|
||
|---|---|
|
||
| `200` | forge answered a real completion |
|
||
| `503` | forge did not |
|
||
| no response at all | the router itself is gone |
|
||
|
||
Verdicts are cached for a few seconds (`FORGEWAY_HEALTH_TTL`), so a burst of traffic
|
||
does not become a burst of probe completions against forge.
|
||
|
||
`/healthz/live` and `/healthz/ready` are **deliberately not gated on forge**. If
|
||
readiness were, Kubernetes would pull every pod out of the Service the moment forge
|
||
died — callers would get a connection failure instead of the 503 `local-strict`
|
||
promises, and Kuma could not distinguish "forge down" from "router gone".
|
||
|
||
Monitoring is **active HTTP probing, not push**: a push heartbeat cannot tell "router
|
||
dead" from "idle". One monitor per Kuma instance, each watching its own local router:
|
||
|
||
| Kuma | Monitor | Target |
|
||
|---|---|---|
|
||
| Lounge `192.168.81.98:3001` | #307 `forgeway — local LLM router (Lounge)` | `http://172.19.10.90/healthz` |
|
||
| Serenity `192.168.24.98:3001` | #57 `forgeway — local LLM router (Serenity)` | `http://172.24.10.21/healthz` |
|
||
|
||
Both: 60 s interval, 48 s timeout, 2 retries, accepted status `200-299`. The timeout is
|
||
wide because a **cold `gpt-oss:120b` load takes ~25–30 s** and is a legitimate success,
|
||
not a failure.
|
||
|
||
### The one operational conflict: monitoring vs. LoRA training
|
||
|
||
A 60 s probe interval keeps `gpt-oss:120b` (~67 GB) permanently resident on forge's
|
||
card 1. An ai-toolkit LoRA run needs ~88 GB on the same card, and the two do not fit —
|
||
`ollama stop gpt-oss:120b` is the documented pre-training step, but with monitoring
|
||
running, the next probe simply loads it again.
|
||
|
||
Before a training run, do one of:
|
||
|
||
- **pause the Kuma monitor** for that site (simplest), or
|
||
- set `FORGEWAY_HEALTH_COLD_LOAD=false` in the ConfigMap. The probe then checks
|
||
Ollama's `/api/ps` and reports DOWN instead of triggering a load. During training
|
||
that reading is *correct*: `local-strict` genuinely cannot serve, and
|
||
`local-optimistic` correctly falls through to the cloud.
|
||
|
||
## Configuration
|
||
|
||
Everything is environment-driven; one image serves every site. See
|
||
`apps/forgeway/manifests/configmap.yaml` in each cluster repo.
|
||
|
||
| Variable | Default | Notes |
|
||
|---|---|---|
|
||
| `FORGEWAY_NAME` | `forgeway` | the single place the service's name lives |
|
||
| `FORGEWAY_SITE` | `unknown` | `lounge` / `serenity` |
|
||
| `FORGEWAY_UPSTREAMS` | forge DNS name | comma-separated, priority order |
|
||
| `FORGEWAY_LOCAL_MODEL` | `gpt-oss:120b` | |
|
||
| `FORGEWAY_CLOUD_MODEL` | `gpt-5.4-mini` | optimistic only |
|
||
| `FORGEWAY_HEALTH_TTL` | `5` | cache for a healthy verdict |
|
||
| `FORGEWAY_HEALTH_FAIL_TTL` | `3` | cache for an unhealthy verdict |
|
||
| `FORGEWAY_HEALTH_TIMEOUT` | `45` | must exceed a ~25–30 s cold load |
|
||
| `FORGEWAY_DISCOVERY_TIMEOUT` | `4` | `6` at Serenity — the mesh hop |
|
||
| `FORGEWAY_HEALTH_COLD_LOAD` | `true` | see the training conflict above |
|
||
| `FORGEWAY_NUM_CTX` | `131072` | pinned on every forge-bound request and the probe; `0` disables |
|
||
| `FORGEWAY_REQUEST_TIMEOUT` | `600` | proxied completions |
|
||
| `OPENAI_API_KEY` | — | from the `forgeway-cloud` Secret, applied out of band |
|
||
|
||
The cloud credential is **never in git**. It is applied with
|
||
`scripts/apply_cloud_secret.py`, which reads the value from the process environment and
|
||
hands it to kubectl on stdin. The `secretKeyRef` is `optional: true`, so the Deployment
|
||
still runs without it — `local-strict` is unaffected and `local-optimistic` reports
|
||
that no fallback credential is configured.
|
||
|
||
## Deploying
|
||
|
||
```bash
|
||
./scripts/build-push.sh 0.1.1 # multi-arch; both clusters are arm64-majority
|
||
```
|
||
|
||
Then bump the image tag in each cluster repo and push — ArgoCD syncs:
|
||
|
||
- Lounge: `lounge-cluster-config/apps/forgeway/manifests/` + `base/apps/forgeway/`
|
||
- Serenity: `serenity-cluster-config/apps/forgeway/` + `base/apps/forgeway-app.yaml`
|
||
|
||
## Verifying
|
||
|
||
```bash
|
||
./scripts/local-smoke.sh # OpenAI dialect, against a real forge
|
||
./scripts/local-smoke-ollama.sh # Ollama dialect, incl. the exact client HA pins
|
||
./scripts/verify-cluster.sh ~/.kube/rocinante-admin.yaml lounge
|
||
./scripts/verify-cluster.sh ~/.kube/nidavellir.yaml serenity
|
||
```
|
||
|
||
`verify-cluster.sh` runs 23 checks from inside the cluster (13 OpenAI, 9 Ollama, 1 pin-vs-runner). It produces the
|
||
"forge unavailable" case with a throwaway Deployment pointed at a dead upstream —
|
||
**it never stops Ollama on forge.**
|