Files
Dolon (Agent)andClaude Opus 5 f7cd9b6690 0.2.3: keep the model resident — pin keep_alive on the native dialect
A cold load of gpt-oss:120b is 21-66 s and lands entirely in front of the
caller, so every idle spell put that pause on Rick's next Home Assistant turn.
The 0.3-0.6 s figures from 2026-09-18 were warm-model numbers.

Measured on Ollama 0.34.0, and NOT symmetrical with the num_ctx pin:

  native /api/chat, keep_alive:-1     -> Forever
  native /api/chat, explicit finite   -> that duration, and it BEATS the
                                         server-side OLLAMA_KEEP_ALIVE default
  native /api/chat, absent            -> does NOT reset an existing value;
                                         the duration is fixed AT LOAD TIME
  /v1/chat/completions, keep_alive    -> IGNORED, like options.num_ctx
                                         (confirmed in openai/openai.go: the
                                         request struct has no such field)

So the router alone can never carry residency: an OpenAI-dialect caller cannot
set it, and whichever request happens to trigger the cold load fixes the
duration. forge therefore also gets OLLAMA_KEEP_ALIVE=-1 in
/etc/systemd/system/ollama.service.d/30-keepalive.conf. The two are
complementary rather than redundant — the server default covers dialects and
callers we never proxy, this pin defeats a native caller that sends a short
keep_alive of its own.

The health probe asserts residency rather than inheriting it: it runs before
every proxied request, so it is the most frequent caller forge has and the one
most likely to be what triggers a cold load.

/status gains keep_alive_pin, runner_resident and runner_expires_at, and the
locator logs a WARNING if forge's runner stops being permanently resident —
the same visibility pattern as num_ctx_pin vs runner_context.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-18 20:02:03 +00:00

130 lines
6.6 KiB
Python

"""Runtime configuration for the router.
Everything is environment-driven so that one image serves every site. The only
value that encodes *identity* is ``FORGEWAY_NAME``; it is read here once and
referenced from everywhere else, because the final DNS name is still Rick's call.
"""
from __future__ import annotations
import os
from dataclasses import dataclass, field
def _env(key: str, default: str) -> str:
value = os.environ.get(key)
return default if value is None or value.strip() == "" else value.strip()
def _env_int(key: str, default: int) -> int:
try:
return int(_env(key, str(default)))
except ValueError:
return default
def _env_float(key: str, default: float) -> float:
try:
return float(_env(key, str(default)))
except ValueError:
return default
def _env_bool(key: str, default: bool) -> bool:
return _env(key, "true" if default else "false").lower() in {"1", "true", "yes", "on"}
def _env_list(key: str, default: str) -> list[str]:
raw = _env(key, default)
return [item.strip().rstrip("/") for item in raw.split(",") if item.strip()]
@dataclass(frozen=True)
class Config:
# --- identity -------------------------------------------------------
# The working name. The final DNS name is NOT decided yet, so this is the
# single place the service learns what it is called.
name: str = field(default_factory=lambda: _env("FORGEWAY_NAME", "forgeway"))
site: str = field(default_factory=lambda: _env("FORGEWAY_SITE", "unknown"))
# --- discovery ------------------------------------------------------
# Ordered candidate list. The router probes these to find forge; it never
# assumes which site forge currently lives at. Put the stable DNS name
# first so a host move is followed automatically by DNS alone.
upstreams: list[str] = field(
default_factory=lambda: _env_list(
"FORGEWAY_UPSTREAMS",
"http://forge.dmz.falkvinge.net:11434",
)
)
local_model: str = field(default_factory=lambda: _env("FORGEWAY_LOCAL_MODEL", "gpt-oss:120b"))
# Ollama keeps ONE runner config per model and fully reloads the model (64 GB,
# 15-30 s) whenever a request's num_ctx differs from the loaded runner's. So
# every forge-bound request from here carries the same num_ctx, overriding
# whatever the caller sent (HA defaults to 8192 on a new subentry). 131072 is
# gpt-oss:120b's trained window and what an unpinned request loads anyway, so
# it costs no extra VRAM; the retrieval workload needs ~100k tokens regardless.
# 0 disables the pin. NOTE: Ollama's OpenAI-compatible /v1 endpoint ignores
# options.num_ctx (verified 2026-09-18), so OpenAI-dialect requests get the
# model default; the pin must equal that default, and /status + a WARNING
# flag it if the runner's context is ever observed to differ.
num_ctx: int = field(default_factory=lambda: _env_int("FORGEWAY_NUM_CTX", 131072))
# Residency. A cold load of gpt-oss:120b costs 21-66 s, so an evicted model puts
# that whole pause in front of the next Home Assistant turn. -1 means "stay loaded
# forever"; 0 disables the pin and leaves the caller's value alone.
#
# MEASURED on Ollama 0.34.0, 2026-09-18. The semantics are NOT symmetrical with
# num_ctx, and the asymmetry is the part that surprises:
# * native /api/chat, keep_alive:-1 -> Forever
# * native /api/chat, explicit finite -> that duration, and it BEATS the server
# default — which is why we override it
# * native /api/chat, absent -> does NOT reset an existing value; the
# duration is fixed AT LOAD TIME
# * /v1/chat/completions, keep_alive -> IGNORED, exactly like options.num_ctx
# So this pin can only carry residency on the NATIVE dialect. OpenAI-dialect
# callers are covered instead by OLLAMA_KEEP_ALIVE=-1 on forge itself (drop-in
# /etc/systemd/system/ollama.service.d/30-keepalive.conf), which sets the load-time
# default for every caller and every path. The two are complementary: the server
# default covers dialects and callers we never see, this pin defeats a native
# caller that sends a short keep_alive of its own.
keep_alive: int = field(default_factory=lambda: _env_int("FORGEWAY_KEEP_ALIVE", -1))
# --- cloud fallback (local-optimistic only) -------------------------
cloud_base_url: str = field(
default_factory=lambda: _env("FORGEWAY_CLOUD_BASE_URL", "https://api.openai.com/v1").rstrip("/")
)
cloud_model: str = field(default_factory=lambda: _env("FORGEWAY_CLOUD_MODEL", "gpt-5.4-mini"))
# --- health ---------------------------------------------------------
# Health is gated on a REAL completion, never a ping: forge can be up with
# Ollama stopped, or Ollama up with the model evicted.
health_ttl: float = field(default_factory=lambda: _env_float("FORGEWAY_HEALTH_TTL", 5.0))
health_fail_ttl: float = field(default_factory=lambda: _env_float("FORGEWAY_HEALTH_FAIL_TTL", 3.0))
health_timeout: float = field(default_factory=lambda: _env_float("FORGEWAY_HEALTH_TIMEOUT", 45.0))
discovery_timeout: float = field(default_factory=lambda: _env_float("FORGEWAY_DISCOVERY_TIMEOUT", 4.0))
# When false, a probe that finds the model not resident reports DOWN instead
# of triggering a cold load. Used to keep monitoring from re-loading
# gpt-oss:120b during an ai-toolkit run (they collide on forge's card 1).
health_cold_load: bool = field(default_factory=lambda: _env_bool("FORGEWAY_HEALTH_COLD_LOAD", True))
# --- proxying -------------------------------------------------------
request_timeout: float = field(default_factory=lambda: _env_float("FORGEWAY_REQUEST_TIMEOUT", 600.0))
connect_timeout: float = field(default_factory=lambda: _env_float("FORGEWAY_CONNECT_TIMEOUT", 5.0))
# Reported by /{mode}/api/version when forge has never answered (clients may
# gate features on it); replaced by forge's real version once seen.
ollama_version_fallback: str = field(default_factory=lambda: _env("FORGEWAY_OLLAMA_VERSION_FALLBACK", "0.34.0"))
port: int = field(default_factory=lambda: _env_int("FORGEWAY_PORT", 8080))
log_level: str = field(default_factory=lambda: _env("FORGEWAY_LOG_LEVEL", "INFO").upper())
@property
def cloud_api_key(self) -> str:
"""Read lazily and never stored, logged, or echoed."""
return os.environ.get("OPENAI_API_KEY", "").strip()
@property
def cloud_enabled(self) -> bool:
return bool(self.cloud_api_key)
CONFIG = Config()