Job 104155149593 failed its own CLI test with
[FAIL] svc-ai-ollama is not deployed yet the gateway publishes
['qwen2.5:0.5b']
In that round ollama is deployed and the variant enables it, so the gateway was
right and the probe was wrong. templates/test.env.j2 renders through
lookup('template') in run_one.yml, which evaluates in the current play host's
scope, while config.yaml.j2 renders through _render_one_template.yml with
delegate_to STACK_HOST and run_once. On a worker the backend flags read false
and the probe compared its own view against a gateway built from another one.
The three flag-derived values are now frozen next to the config render, with
the same delegate_to plus run_once. Which host run_once elects does not matter;
what matters is that it is the same host for both, and that follows from the
same construct in the same play.
CLI_LOCAL_CID stays per host. 9fb8f63286 introduced it precisely because the
container lives on whichever node the service scheduled it to, and the test
script task carries no run_once, so it runs once per play host against its own
resolved node. Freezing the render itself would have pinned one node's
container id into every host's env and broken the CLI test for every role that
execs into its own container, not just this one.
Two facts this rests on, both measured rather than assumed: a run_once set_fact
does reach the other hosts, as web-app-gitea's mcp provisioning shows by
asserting on gitea_mcp_token from a task that carries no run_once; and
03_destructor.yml includes test-e2e-cli as a stage of the same play the deploy
ran in, so the fact is still there when the template renders.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
At -vvv Ansible prints each module argument and result, so a password in
a command line, a request body, a header or a fact lands in the deploy
log. Nothing enforced masking outside the MCP task files, and no check
looked at parents at all.
A new lint, test_credential_tasks_are_masked, finds tasks whose printed
parts (module arguments, environment, loop) reference an identifier ending
in password, secret, token, api_key or private_key, or
secrets.credentials. A task passes when it or an enclosing block sets
no_log, when every file that pulls it in does so through include_tasks
with apply.no_log or import_tasks with no_log (transitively), when the
secret only reaches a parameter the module masks itself, or when the
module masks its values. no_log on an include_tasks task itself does not
count, because it does not reach the included tasks. The rule can be
suppressed per task with `# nocheck: credential-masking`, now listed in
the suppression catalog, and the lint tests its parent and suppression
logic on synthetic cases.
It found 39 unmasked tasks in 19 roles, all real: join tokens, client
secrets, API tokens in Cloudflare headers, database passwords in
command lines, the ChatGPT bot login body and more. Each now sets
no_log from MASK_CREDENTIALS_IN_LOGS. The sys-dns-hetzner-rdns alias
hetzner_no_log, which only restated that switch, is gone.
Not covered yet: template and copy tasks whose rendered file carries a
secret print it as a diff under --diff.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four tasks in this file talk to the gateway, and each carries its own
tolerance for it not being up: 60 seconds for the keys, 50 for the closure
proof, 50 for the model warm-up. The first one to run is the keys, so a gateway
that takes longer than a minute fails there and the later budgets never apply.
54a6f3b533 already named this when it added the warm-up: "Five retries cover a
gateway that is not listening yet." It gave the last task 50 seconds and left
the first one at the 60 it already had.
One wait in front of all four, on the signal the failure actually reports.
CI shows ConnectionRefusedError from urllib after eleven attempts, which is the
TCP layer, so the probe opens a socket rather than asking an HTTP endpoint whose
path would be a guess. python3 is what the failing task itself runs in that
container, so nothing new has to exist there.
A healthcheck would have been the obvious shape and would have changed nothing:
sys-svc-compose/tasks/utils/post_deploy.yml waits on the database container's
health in compose and on swarm tasks being Running in swarm, and reads the
application container's health in neither.
This does not explain every job in that cluster. One role's rescue artifact
shows all three replicas reaching "Proxy initialized with Config" while
postgres resets and then terminates their connections (SqlState 57P01), which
is a different failure wearing the same stack trace. Where that is the cause,
this makes the deploy fail at a task that names the gateway instead of at the
one provisioning keys.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role declared sso.enabled false behind three suppressions, so the checks were switched off rather than satisfied. LiteLLM authenticates its own admin UI instead of sitting behind an oauth2 proxy, so the declaration is flavor oidc and the client id, the three Keycloak endpoints, the proxy base URL and the admin id all render together under one flag: a half-set group would let the gateway fall back to its own form with nothing complaining. The variants pin both states so the matrix runs with and without SSO.
The proof has to measure the departure, not the arrival: a gateway ignoring the client id answers /sso/key/generate with its own username and password form and never redirects, so the spec asserts the round trip to Keycloak and back. The guest spec became SSO-aware, since its canonical-domain and password-field assertions only hold where no SSO is configured.
Two more gateway defects: the key warm-up ran even where no model was served, falling back to openrouter/auto so the gateway then answered every prompt with 400, and it is now gated on a served-model expression mirroring the model selection itself. A lint requires every declared svc-ai backend to have a model_list branch, because a backend without a route is an alias nobody serves. probe_auth.py proves revocation rather than creation: it mints a throwaway key, revokes it, and fails unless a real consumer key survives.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A backend pulls its preload model at deploy time but only loads it into
memory on the first inference. That cold load outlives a reverse proxy's
read timeout, so the first consumer request comes back as a 504 while every
later one succeeds.
Send one max_tokens=1 completion from inside the litellm container right
after the per-consumer keys exist, where the container id, master key and
port are already resolved. Five retries cover a gateway that is not
listening yet.
This pairs with the ollama keep_alive setting: the warm-up pays the load
once, keep_alive stops it decaying after five idle minutes.
Note this hardens the deploy: a gateway that cannot serve its configured
model now fails the role instead of surfacing later as a consumer error.
That makes the openrouter/auto fallback with its empty API key loud rather
than silent.
The script ships with a mirrored unit test rather than a nocheck waiver; it
patches urlopen and asserts the URL, method, authorization header, model,
max_tokens and that the timeout exceeds a proxy read timeout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merged branch moved every role-local script out of files/ into a folder
named for its language (files/python/, files/php/, files/shell/), but the
moves lost their staging and never landed. HEAD still carried the old paths
while the worktree had the new ones.
Move the 24 remaining scripts and update the six places that resolve them:
the mcp-adapter instance/isolation/probe tasks, the litellm key task, the
adapter README and the endpoint-path lint. Every changed line in those
files is a path.
Also drop roles/sys-svc-compose/files/swarm/stack_ready.sh, a stale
duplicate of the copy already tracked under files/shell/swarm/.
The added comments are the branch's own `nocheck: mirrored-unit-test`
directives, which state why a script whose measurement only exists inside
its container cannot have a mirrored unit test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A role that enables its native AI surface can reach the gateway with an empty
credential, or point its base URL at something that resolves outside the
cluster, and still deploy green. Both are silent: the surface simply answers
from somewhere else, or not at all. This adds the shared contract that makes
either state fail the deploy, so a consuming role inherits one definition of
"inside the deployment" instead of restating it.
The key assertion refuses an empty credentials.litellm_api_key while
services.litellm is enabled. Without it the gateway leg is unauthenticated and
nothing in the run says so.
The URL assertion is four independent checks, each able to fail on its own:
the scheme is plain http, the hostname carries no dot, the hostname is not the
consuming role's own domain, and the port is the gateway's internal port rather
than its published one. The undotted check is the load-bearing one. The
container dns-search suffix completes a dotted host into the public FQDN, so
`litellm.infinito.example` leaves through the ingress while looking local in
the config; only a bare service name stays on the container network. The
own-domain check catches a role that points its AI surface back at itself, and
the internal-port check catches a config that would work only via the host
bind.
An earlier revision of this file compared two values that both came from the
same lookup and therefore could never fail. Every check here is falsifiable
against the value the role actually deploys.
group_vars carries the SPOT the contract measures against.
LITELLM_OPENAI_BASE_LOCAL_URL appends the OpenAI-compatible /v1 suffix once,
rather than in each role. LITELLM_CHAT_MODEL derives the model from whichever
backend is deployed -- an ollama preload model with embedding models rejected,
else lmstudio, else openrouter -- resolved through `lookup('deployment').running`
so it reflects what is running rather than what the inventory could contain.
Contract and vars land before the roles that include them; the reverse order
would commit roles referencing a file that does not exist yet.
Verified: the contract's assertions ran green at runtime for matrix,
mattermost, moodle and zammad. Zammad's is the strongest evidence, since
`Setting.set('ai_provider_config')` triggers upstream
`AI::Provider::CustomOpenAI.ping!` and therefore a live gateway round trip.
`make quality` green on test-lint, test-unit, test-integration, test-external.
Not covered: nothing yet forces a role that declares services.litellm to
include this contract, so a future consumer can still skip it silently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An inventory re-initialization mints a new virtual key while the proxy
database keeps the previous one under the same alias. Aliases are unique,
so creating the new key was refused and the constructor aborted after ten
identical retries. The alias is now released before the key is created,
and the release tolerates finding nothing.
The script moves out of the task body into files/ensure_keys.py, with the
port passed as an environment variable instead of templated into the
source, so it is plain Python and covered by unit tests.
Consumers of the model plane speak one OpenAI-compatible endpoint and
carry one virtual key; which backend answers is the gateway's business.
svc-ai-litellm renders its model_list from the round's actual topology,
so an alias only exists when its backend role is in group_names, and it
emits an explicit empty list when no backend is present at all because
LiteLLM rejects a bare model_list: key.
Keys reach the gateway through the per-consumer credential rather than
being minted there: 01_keys.yml probes /key/info and only POSTs
/key/generate on 404, so reruns stay idempotent. The consumer list is
built in 00_core.yml, not inside the included file, because 01_keys.yml
delegates to the hosting node and delegate_to is templated before when.
The gateway declares ollama and lmstudio as consumer flags so the
network renderer attaches it to its own backends; without them it cannot
reach them.
svc-ai-lmstudio runs the headless llmster daemon behind expose only, so
it is reachable from the gateway but never bound on the host, matching
svc-ai-ollama.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5