3 Commits
Author SHA1 Message Date
Dolon (Agent)andClaude Opus 5 f7cd9b6690 0.2.3: keep the model resident — pin keep_alive on the native dialect
A cold load of gpt-oss:120b is 21-66 s and lands entirely in front of the
caller, so every idle spell put that pause on Rick's next Home Assistant turn.
The 0.3-0.6 s figures from 2026-09-18 were warm-model numbers.

Measured on Ollama 0.34.0, and NOT symmetrical with the num_ctx pin:

  native /api/chat, keep_alive:-1     -> Forever
  native /api/chat, explicit finite   -> that duration, and it BEATS the
                                         server-side OLLAMA_KEEP_ALIVE default
  native /api/chat, absent            -> does NOT reset an existing value;
                                         the duration is fixed AT LOAD TIME
  /v1/chat/completions, keep_alive    -> IGNORED, like options.num_ctx
                                         (confirmed in openai/openai.go: the
                                         request struct has no such field)

So the router alone can never carry residency: an OpenAI-dialect caller cannot
set it, and whichever request happens to trigger the cold load fixes the
duration. forge therefore also gets OLLAMA_KEEP_ALIVE=-1 in
/etc/systemd/system/ollama.service.d/30-keepalive.conf. The two are
complementary rather than redundant — the server default covers dialects and
callers we never proxy, this pin defeats a native caller that sends a short
keep_alive of its own.

The health probe asserts residency rather than inheriting it: it runs before
every proxied request, so it is the most frequent caller forge has and the one
most likely to be what triggers a cold load.

/status gains keep_alive_pin, runner_resident and runner_expires_at, and the
locator logs a WARNING if forge's runner stops being permanently resident —
the same visibility pattern as num_ctx_pin vs runner_context.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 20:02:03 +00:00
Dolon (Agent)andClaude Fable 5.1 2d3c548542 0.2.2: pin num_ctx on every forge-bound request and on the health probe
Ollama keeps one runner config per model and reloads the 64 GB model on
every request whose num_ctx differs from the loaded runner's. The router
now sets options.num_ctx = FORGEWAY_NUM_CTX (default 131072) on the Ollama
dialect and carries the same value in the probe (native /api/chat; the
OpenAI /v1 endpoint ignores options, verified). /status exposes the pin and
the observed runner context and warns on mismatch; check 23 asserts it.

Verified by behaviour: an 8192 caller, a 4096 caller, probes and 40k/100k-
token retrieval prompts in one window -> 0 reloads, 0 timeouts, HA turns
0.3-0.6 s; before, 3 rounds of an 8192 caller = 7 loads and two probe
timeouts. 23/23 acceptance on both clusters.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-18 19:43:13 +00:00
Dolon (Agent)andClaude Opus 5 3155d4a230 forgeway 0.1.1 — per-cluster OpenAI-compatible LLM router
Two endpoints over forge's Ollama gpt-oss:120b:
  /local-optimistic/v1  forge, else cloud gpt-5.4-mini
  /local-strict/v1      forge, else a clean 503 the caller handles

Discovery, not configuration: each instance probes an ordered candidate list
(stable DNS name, then the Lounge and future Serenity literals) concurrently but
in priority order, so the happy path is one round trip and silicon's move to
Serenity needs no rewrite.

/healthz runs a real 1-token completion — forge can be up with Ollama stopped.
Liveness/readiness are deliberately NOT forge-gated: otherwise k8s would evict
the pods during an outage and callers would get a connection failure instead of
the promised 503, and Kuma could not tell "forge down" from "router gone".

max_tokens/max_completion_tokens are translated per backend: Ollama accepts only
the former, gpt-5.4-mini rejects it outright.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 14:30:14 +00:00