Four tasks in this file talk to the gateway, and each carries its own
tolerance for it not being up: 60 seconds for the keys, 50 for the closure
proof, 50 for the model warm-up. The first one to run is the keys, so a gateway
that takes longer than a minute fails there and the later budgets never apply.
54a6f3b533 already named this when it added the warm-up: "Five retries cover a
gateway that is not listening yet." It gave the last task 50 seconds and left
the first one at the 60 it already had.
One wait in front of all four, on the signal the failure actually reports.
CI shows ConnectionRefusedError from urllib after eleven attempts, which is the
TCP layer, so the probe opens a socket rather than asking an HTTP endpoint whose
path would be a guess. python3 is what the failing task itself runs in that
container, so nothing new has to exist there.
A healthcheck would have been the obvious shape and would have changed nothing:
sys-svc-compose/tasks/utils/post_deploy.yml waits on the database container's
health in compose and on swarm tasks being Running in swarm, and reads the
application container's health in neither.
This does not explain every job in that cluster. One role's rescue artifact
shows all three replicas reaching "Proxy initialized with Config" while
postgres resets and then terminates their connections (SqlState 57P01), which
is a different failure wearing the same stack trace. Where that is the cause,
this makes the deploy fail at a task that names the gateway instead of at the
one provisioning keys.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role declared sso.enabled false behind three suppressions, so the checks were switched off rather than satisfied. LiteLLM authenticates its own admin UI instead of sitting behind an oauth2 proxy, so the declaration is flavor oidc and the client id, the three Keycloak endpoints, the proxy base URL and the admin id all render together under one flag: a half-set group would let the gateway fall back to its own form with nothing complaining. The variants pin both states so the matrix runs with and without SSO.
The proof has to measure the departure, not the arrival: a gateway ignoring the client id answers /sso/key/generate with its own username and password form and never redirects, so the spec asserts the round trip to Keycloak and back. The guest spec became SSO-aware, since its canonical-domain and password-field assertions only hold where no SSO is configured.
Two more gateway defects: the key warm-up ran even where no model was served, falling back to openrouter/auto so the gateway then answered every prompt with 400, and it is now gated on a served-model expression mirroring the model selection itself. A lint requires every declared svc-ai backend to have a model_list branch, because a backend without a route is an alias nobody serves. probe_auth.py proves revocation rather than creation: it mints a throwaway key, revokes it, and fails unless a real consumer key survives.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A backend pulls its preload model at deploy time but only loads it into
memory on the first inference. That cold load outlives a reverse proxy's
read timeout, so the first consumer request comes back as a 504 while every
later one succeeds.
Send one max_tokens=1 completion from inside the litellm container right
after the per-consumer keys exist, where the container id, master key and
port are already resolved. Five retries cover a gateway that is not
listening yet.
This pairs with the ollama keep_alive setting: the warm-up pays the load
once, keep_alive stops it decaying after five idle minutes.
Note this hardens the deploy: a gateway that cannot serve its configured
model now fails the role instead of surfacing later as a consumer error.
That makes the openrouter/auto fallback with its empty API key loud rather
than silent.
The script ships with a mirrored unit test rather than a nocheck waiver; it
patches urlopen and asserts the URL, method, authorization header, model,
max_tokens and that the timeout exceeds a proxy read timeout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merged branch moved every role-local script out of files/ into a folder
named for its language (files/python/, files/php/, files/shell/), but the
moves lost their staging and never landed. HEAD still carried the old paths
while the worktree had the new ones.
Move the 24 remaining scripts and update the six places that resolve them:
the mcp-adapter instance/isolation/probe tasks, the litellm key task, the
adapter README and the endpoint-path lint. Every changed line in those
files is a path.
Also drop roles/sys-svc-compose/files/swarm/stack_ready.sh, a stale
duplicate of the copy already tracked under files/shell/swarm/.
The added comments are the branch's own `nocheck: mirrored-unit-test`
directives, which state why a script whose measurement only exists inside
its container cannot have a mirrored unit test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An inventory re-initialization mints a new virtual key while the proxy
database keeps the previous one under the same alias. Aliases are unique,
so creating the new key was refused and the constructor aborted after ten
identical retries. The alias is now released before the key is created,
and the release tolerates finding nothing.
The script moves out of the task body into files/ensure_keys.py, with the
port passed as an environment variable instead of templated into the
source, so it is plain Python and covered by unit tests.
Consumers of the model plane speak one OpenAI-compatible endpoint and
carry one virtual key; which backend answers is the gateway's business.
svc-ai-litellm renders its model_list from the round's actual topology,
so an alias only exists when its backend role is in group_names, and it
emits an explicit empty list when no backend is present at all because
LiteLLM rejects a bare model_list: key.
Keys reach the gateway through the per-consumer credential rather than
being minted there: 01_keys.yml probes /key/info and only POSTs
/key/generate on 404, so reruns stay idempotent. The consumer list is
built in 00_core.yml, not inside the included file, because 01_keys.yml
delegates to the hosting node and delegate_to is templated before when.
The gateway declares ollama and lmstudio as consumer flags so the
network renderer attaches it to its own backends; without them it cannot
reach them.
svc-ai-lmstudio runs the headless llmster daemon behind expose only, so
it is reachable from the gateway but never bound on the host, matching
svc-ai-ollama.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5