Dolon (Agent)andClaude Fable 5.1 1bec7c784c DNS + ingress: the confirmed names, split-horizon alias, providers on https
Rick confirmed forgeway.<site>.falkvinge.net and the local-first alias
forgeway.services.falkvinge.net. README documents the alias as the front
door, the per-site names as explicit handles, and exactly how the Serenity
override keeps the alias LOCAL (host-record + local= on both Serenity
Pi-holes; the local= line closes the dual-stack AAAA bypass).

OpenClaw providers now point at the https alias with a placeholder apiKey:
OpenClaw exempts plain-IP http base URLs from needing a key but not https
hostnames. scripts/technitium.py ships the Technitium API helper used for
the records (zones / records / add / resync / logtail).

REPORT.md addendum carries the verification table from real clients at both
sites and the findings made on the way.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-18 14:56:12 +00:00

forgeway

An OpenAI-compatible LLM router in front of forge's Ollama gpt-oss:120b. One instance runs per cluster, not one central service.

Names (confirmed by Rick, 2026-09-18). Explicit per-site names forgeway.lounge.falkvinge.net and forgeway.serenity.falkvinge.net (Chalet later: forgeway.chalet.falkvinge.net), plus the split-horizon local-first alias forgeway.services.falkvinge.net, which resolves to whichever instance is local to the querying client's site. .services. names are internal-only by convention: no entrance SAN, no external exposure, ever.

The two endpoints

Endpoint Behaviour when forge is available Behaviour when it is not
/local-optimistic/v1 serves from forge falls back to OpenAI gpt-5.4-mini
/local-strict/v1 serves from forge clean HTTP 503, no fallback, by design

Both expose /v1/chat/completions (including stream: true) and /v1/models.

Where they live:

Site Cluster Use this Raw LB IP (DNS-independent)
any — https://forgeway.services.falkvinge.net → the instance local to you —
Lounge Rocinante http://forgeway.lounge.falkvinge.net http://172.19.10.90
Serenity Niðavellir http://forgeway.serenity.falkvinge.net http://172.24.10.21
Chalet — when the cluster exists —

The alias is the front door. Use the per-site names only when you deliberately want a specific site's instance (for example, asking the Lounge one from Serenity). HTTPS with a valid certificate is available on the alias at both sites; the per-site names are http:// only — the internal ingress certificates carry *.services.falkvinge.net but not *.lounge. / *.serenity. (adding those SANs to the two internal ingress certificates is a one-line cert-manager change if ever wanted; the entrance bundle is not the place).

Calling them

local-strict — do not retry a 503 blindly. A 503 means the local model is unavailable right now: forge is off, Ollama is stopped, or the model is evicted for a training run. None of those clear in the next second, and retrying just adds load to something already struggling. Handle the error, or degrade to your own fallback.

# strict: one attempt, handle the 503 yourself
curl -sS -X POST https://forgeway.services.falkvinge.net/local-strict/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"gpt-oss:120b","messages":[{"role":"user","content":"hello"}]}' \
  -w '\n%{http_code}\n'
import httpx

# strict — no retries on 503; that status is an answer, not a hiccup
with httpx.Client(base_url="https://forgeway.services.falkvinge.net/local-strict/v1", timeout=600) as c:
    r = c.post("/chat/completions", json={
        "model": "gpt-oss:120b",
        "messages": [{"role": "user", "content": "hello"}],
    })
    if r.status_code == 503:
        ...          # local model is down: skip, queue, or use your own fallback
    else:
        r.raise_for_status()
        print(r.json()["choices"][0]["message"]["content"])

# optimistic — fallback is already handled inside the router.
# Do NOT add your own retry loop: a failure here means forge AND the cloud both failed.
with httpx.Client(base_url="https://forgeway.services.falkvinge.net/local-optimistic/v1", timeout=600) as c:
    r = c.post("/chat/completions", json={
        "model": "gpt-oss:120b",
        "messages": [{"role": "user", "content": "hello"}],
    })
    r.raise_for_status()
    print(r.headers["x-forgeway-backend"])   # "forge" or "cloud"

With the OpenAI SDK, point base_url at one of the two prefixes and use any non-empty api key (the router does not authenticate callers):

from openai import OpenAI
client = OpenAI(base_url="https://forgeway.services.falkvinge.net/local-optimistic/v1", api_key="unused")

Every response carries x-forgeway-backend: forge|cloud, x-forgeway-upstream, x-forgeway-site and x-forgeway-mode, so a caller can always tell what answered.

Model selection. The model field in a request is ignored — each endpoint routes to its own backend. Send anything; gpt-oss:120b is conventional. max_tokens and max_completion_tokens are interchangeable: the router translates between them, because Ollama only accepts the first and gpt-5.4-mini rejects it outright.

From OpenClaw agents

The same endpoints are registered as model providers (base URL = the .services. alias over HTTPS, so agents get the local instance too), so agents and scripts use one backend:

forgeway-strict/gpt-oss-120b        # forge or an error
forgeway-optimistic/gpt-oss-120b    # forge, else gpt-5.4-mini

DNS: how the alias finds the local instance

Every .services. name in the estate is a CNAME chain ending at a site's ingress-nginx, and the authoritative services.falkvinge.net zone lives on the Lounge Technitium. So at the Lounge nothing special is needed:

forgeway.services.falkvinge.net → forgeway.dmz.lounge.falkvinge.net → ingress.dmz.lounge.falkvinge.net → 172.19.10.11
forgeway.lounge.falkvinge.net   → forgeway.dmz.lounge.falkvinge.net → …                                 → 172.19.10.11
forgeway.serenity.falkvinge.net → forgeway.dmz.serenity.falkvinge.net → ingress.dmz.serenity.falkvinge.net → 172.24.10.11

Serenity's Pi-holes forward everything under falkvinge.net (except serenity.) to the Lounge — so without an override, a Serenity client would silently get the Lounge ingress and the remote instance, defeating the whole multi-site design. Both Serenity Pi-holes therefore pin the alias to the Serenity ingress:

Resolver Where the override lives
172.24.10.201 (k8s, two pods) serenity-cluster-config/apps/pihole/pihole.yaml → 03-serenity-forwards.conf
172.24.1.201 (LXC) Pi-hole v6 API: dns.hosts entry + misc.dnsmasq_lines

Both carry host-record=forgeway.services.falkvinge.net,172.24.10.11 and local=/forgeway.services.falkvinge.net/. The local= line matters: without it an AAAA query is still forwarded to the Lounge, and the day the Lounge ingress gains an IPv6 address every dual-stack Serenity client would prefer it and quietly reach the remote instance — the same dual-stack bypass that defeated the Lounge overrides on 2026-08-24.

Each site's ingress serves forgeway.services.falkvinge.net for its own instance, so the DNS answer alone decides which site you land on. Verified 2026-09-18 from real clients: Lounge (OpenClaw VM, Lounge Kuma host) → 172.19.10.11, x-forgeway-site: lounge; Serenity (Niðavellir pod, both Serenity Pi-holes) → 172.24.10.11, x-forgeway-site: serenity. Never the reverse.

Why one instance per cluster

A single central router would sit on one site. The cross-site link to forge is exactly the kind of thing that goes down — and if the router lived on the far side of it, the router would be the outage rather than the thing that reports it. So each cluster runs its own, and each one answers for itself: when Serenity cannot reach forge, Serenity's local-optimistic still serves from the cloud and its local-strict still returns a clean 503, both without depending on anything at the Lounge.

It does not run on forge itself, for the same reason: a router that dies with its upstream cannot report on it.

Discovery — and what happens when silicon moves to Serenity

No instance is told which site forge is at. Each one holds an ordered candidate list (FORGEWAY_UPSTREAMS) and finds forge at request time:

http://forge.dmz.falkvinge.net:11434   the stable name — DNS follows a host move
http://172.19.0.130:11434              forge at the Lounge (today)
http://172.24.0.130:11434              forge at Serenity (after the move)

Candidates are probed concurrently but awaited in priority order, so the happy path costs one fast round trip and dead candidates never slow it down. The winner is sticky and tried first next time.

When silicon relocates to Serenity, nothing here needs rewriting. Re-point forge.dmz.falkvinge.net at its new address and both instances follow on their next health refresh (seconds). If DNS is stale or broken, discovery still finds forge at the Serenity literal already in the list. The only thing that changes materially is which site pays the cross-site latency — and that flips automatically, because each instance independently discovers the nearest working answer.

Two things do need a human when that happens:

  1. forge's ufw — the allow rules are scoped to source ranges. Serenity-local traffic will then arrive from 172.24.9.0/24 (already allowed); Lounge traffic will arrive SNAT'd from the Serenity edge and will need its own rule.
  2. Kuma — the monitors point at each local router by IP, so they keep working, but the Lounge monitor will start reporting the slower cross-site path.

Health and monitoring

/healthz performs a real 1-token completion against forge, never a ping. A ping cannot tell the difference between a healthy forge and a forge that is up with Ollama stopped — which was literally its state on the morning this was built.

Result Meaning
200 forge answered a real completion
503 forge did not
no response at all the router itself is gone

Verdicts are cached for a few seconds (FORGEWAY_HEALTH_TTL), so a burst of traffic does not become a burst of probe completions against forge.

/healthz/live and /healthz/ready are deliberately not gated on forge. If readiness were, Kubernetes would pull every pod out of the Service the moment forge died — callers would get a connection failure instead of the 503 local-strict promises, and Kuma could not distinguish "forge down" from "router gone".

Monitoring is active HTTP probing, not push: a push heartbeat cannot tell "router dead" from "idle". One monitor per Kuma instance, each watching its own local router:

Kuma Monitor Target
Lounge 192.168.81.98:3001 #307 forgeway — local LLM router (Lounge) http://172.19.10.90/healthz
Serenity 192.168.24.98:3001 #57 forgeway — local LLM router (Serenity) http://172.24.10.21/healthz

Both: 60 s interval, 48 s timeout, 2 retries, accepted status 200-299. The timeout is wide because a cold gpt-oss:120b load takes ~25–30 s and is a legitimate success, not a failure.

The one operational conflict: monitoring vs. LoRA training

A 60 s probe interval keeps gpt-oss:120b (~67 GB) permanently resident on forge's card 1. An ai-toolkit LoRA run needs ~88 GB on the same card, and the two do not fit — ollama stop gpt-oss:120b is the documented pre-training step, but with monitoring running, the next probe simply loads it again.

Before a training run, do one of:

  • pause the Kuma monitor for that site (simplest), or
  • set FORGEWAY_HEALTH_COLD_LOAD=false in the ConfigMap. The probe then checks Ollama's /api/ps and reports DOWN instead of triggering a load. During training that reading is correct: local-strict genuinely cannot serve, and local-optimistic correctly falls through to the cloud.

Configuration

Everything is environment-driven; one image serves every site. See apps/forgeway/manifests/configmap.yaml in each cluster repo.

Variable Default Notes
FORGEWAY_NAME forgeway the single place the service's name lives
FORGEWAY_SITE unknown lounge / serenity
FORGEWAY_UPSTREAMS forge DNS name comma-separated, priority order
FORGEWAY_LOCAL_MODEL gpt-oss:120b
FORGEWAY_CLOUD_MODEL gpt-5.4-mini optimistic only
FORGEWAY_HEALTH_TTL 5 cache for a healthy verdict
FORGEWAY_HEALTH_FAIL_TTL 3 cache for an unhealthy verdict
FORGEWAY_HEALTH_TIMEOUT 45 must exceed a ~25–30 s cold load
FORGEWAY_DISCOVERY_TIMEOUT 4 6 at Serenity — the mesh hop
FORGEWAY_HEALTH_COLD_LOAD true see the training conflict above
FORGEWAY_REQUEST_TIMEOUT 600 proxied completions
OPENAI_API_KEY — from the forgeway-cloud Secret, applied out of band

The cloud credential is never in git. It is applied with scripts/apply_cloud_secret.py, which reads the value from the process environment and hands it to kubectl on stdin. The secretKeyRef is optional: true, so the Deployment still runs without it — local-strict is unaffected and local-optimistic reports that no fallback credential is configured.

Deploying

./scripts/build-push.sh 0.1.1               # multi-arch; both clusters are arm64-majority

Then bump the image tag in each cluster repo and push — ArgoCD syncs:

  • Lounge: lounge-cluster-config/apps/forgeway/manifests/ + base/apps/forgeway/
  • Serenity: serenity-cluster-config/apps/forgeway/ + base/apps/forgeway-app.yaml

Verifying

./scripts/local-smoke.sh                                       # against a real forge, from this VM
./scripts/verify-cluster.sh ~/.kube/rocinante-admin.yaml lounge
./scripts/verify-cluster.sh ~/.kube/nidavellir.yaml serenity

verify-cluster.sh runs 13 checks from inside the cluster. It produces the "forge unavailable" case with a throwaway Deployment pointed at a dead upstream — it never stops Ollama on forge.

S
Description
OpenAI-compatible LLM router in front of forge's Ollama gpt-oss:120b; one instance per cluster
Readme
306 KiB
Languages
Python 82.1%
Shell 17.5%
Dockerfile 0.4%