The ranking the last commits built toward is now enforced: a semver tag,
else a digest, and nothing else without a written reason. Two states fail.
A pin that is neither names something the registry can move underneath us,
and nothing here can tell whether it still points where it did when it was
written. A digest on an upstream that does publish semver takes the
fallback for no gain and hides the entry from the update job.
Thirteen entries stood in the way when the check was first run. Four were
resolved by the channel grammar, which made the Jitsi components orderable
without touching their pins. Three needed nothing once a moving tag could
declare itself with 'docker-version'. Two are fixed here and four are
declared.
LiteLLM leaves 'main-v1.77.3.dynamic_rates'. That tag exists exactly once
among 2609, so it is a dead end no updater can follow, and it misreports
itself: the image inside answers 1.77.2. The pin moves to
'main-v1.77.7-stable', five patches along the same minor and onto a channel
with 134 builds. Started against this repository's config shape, 1.77.7
registers both model entry forms and accepts drop_params and the
os.environ master key.
Pixelfed leaves 'latest' for '20260917'. The upstream stamps its dates as
one eight-digit number, which is depth 1 and orderable with no tooling
change at all, and the two tags resolve to the same digest, so nothing
about the deployed image moves.
The four declarations carry the count that justifies them rather than a
claim: LM Studio publishes two tags and neither is a version, the MIG
registry carries one, and the two Nextcloud Talk images publish 174 and 104
AIO build stamps between them.
Both branches of the gate were negative-controlled. The digest branch
skips in this repository because no digest pin exists at all, so it was
exercised by introducing one and watching it fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
API is a plain mapping, so an inventory that configures three providers replaces the whole declaration from group_vars/all/18_api.yml instead of merging into it. inventories/development/default.yml declares openai, anthropic and openrouter, so github, google and jira did not exist in CI at all and lookup('api', 'github.client_id') raised.
A raising lookup inside a value the applications renderer is templating does not fail the render: _templar_render_best_effort returns the string untouched and, for a value carrying a non-env lookup(, refuses the fallback as well. The raw template text reached plugin_value.enabled | bool, which coerced it to False and emitted the deprecation warning that fails the deploy gate. Run 34893363518, job 104153194109 lost web-app-nextcloud to exactly that, for three integrations at once. The configvalue entries would have stored template text as the client id.
utils/api.py merges the scope's override over the declaration, so an override narrows values and never the provider set, and holds the enable rule: every credential a provider declares must carry a value. The api lookup reads through it; the new api_enabled lookup gates on it and returns a real boolean instead of a string a filter has to coerce.
The same class sat in group_vars/all/16_ai.yml and roles/svc-ai-litellm/meta/secrets.yml as bare API.<provider>.<key> dereferences; both now go through the lookups.
Two lints keep it shut. test_addon_enabled_renders_to_bool renders the whole applications map against a scope with API blanked, the most hostile override an inventory may legally write, and requires every addon gate to come out a boolean. test_api_accessed_via_lookup forbids a bare API dereference in Jinja under roles/, group_vars/ and inventories/.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Job 104155149593 failed its own CLI test with
[FAIL] svc-ai-ollama is not deployed yet the gateway publishes
['qwen2.5:0.5b']
In that round ollama is deployed and the variant enables it, so the gateway was
right and the probe was wrong. templates/test.env.j2 renders through
lookup('template') in run_one.yml, which evaluates in the current play host's
scope, while config.yaml.j2 renders through _render_one_template.yml with
delegate_to STACK_HOST and run_once. On a worker the backend flags read false
and the probe compared its own view against a gateway built from another one.
The three flag-derived values are now frozen next to the config render, with
the same delegate_to plus run_once. Which host run_once elects does not matter;
what matters is that it is the same host for both, and that follows from the
same construct in the same play.
CLI_LOCAL_CID stays per host. 9fb8f63286 introduced it precisely because the
container lives on whichever node the service scheduled it to, and the test
script task carries no run_once, so it runs once per play host against its own
resolved node. Freezing the render itself would have pinned one node's
container id into every host's env and broken the CLI test for every role that
execs into its own container, not just this one.
Two facts this rests on, both measured rather than assumed: a run_once set_fact
does reach the other hosts, as web-app-gitea's mcp provisioning shows by
asserting on gitea_mcp_token from a task that carries no run_once; and
03_destructor.yml includes test-e2e-cli as a stage of the same play the deploy
ran in, so the fact is still there when the template renders.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Job 104155149606 died at the warm request with
litellm.APIConnectionError: OllamaException - Cannot connect to host
ollama:11434 [Connect call failed ('192.168.244.10', 11434)]
192.168.244.10 is not ollama. It is the manager node on the lab bridge: the
name never resolved through the overlay, because the litellm task was attached
to ingress, litellm and postgres and to nothing else. The lmstudio round is the
control, where the same replica does carry the lmstudio network.
The attachment comes from _is_consumer, which reads svc-ai-litellm's own
services.<backend>.enabled and .shared. Variant 1 pins both false. The config
template meanwhile asked `'svc-ai-ollama' in group_names`, which answers a
different question: is ollama deployed anywhere in the round. It is, six edges
away, because litellm pulls postgres, postgres pulls tor, tor pulls the mirror,
the mirror pulls prometheus and prometheus declares flowise, which declares
ollama. So the gateway advertised a route it had no network for.
The template and LITELLM_CHAT_MODEL_SERVED now read the same two flags the
network attachment reads, through one constant each. Publication and routing
cannot diverge any more, and in a variant that declines a deployed backend the
warm task no longer runs at all.
LITELLM_CHAT_MODEL and LITELLM_BACKEND_ROLES are deliberately left alone.
5324eadd8b made them inventory-scoped because web-app-mattermost's assert, the
one task in its file carrying neither delegate_to nor run_once, re-templates in
each worker's own scope; web-app-discourse's addon tasks carry no delegate_to
at all. A host-scoped model name fails there again. Both templates here render
delegate_to STACK_HOST plus run_once, where group_names is authoritative.
meta/services.yml keeps its `in group_names` flags, so the five separate
substring detectors that decide whether a flag counts as dynamic keep seeing
svc-ai-litellm.
Measured over every (deployed, pinned) combination against the real
expressions and the real template: the model consumers are told to ask for is
published whenever the warm task runs, and nothing is published without its
flag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gateway could reach exactly one remote provider, through a bare
OPENROUTER_API_KEY group_var with an empty-string default and an `{% if %}`
around a single hardcoded model entry. OpenAI and Anthropic had no path at all.
All three are now declared in roles/svc-ai-litellm/meta/secrets.yml, which
mapping.py already calls the application config schema, so a key lands at
applications.svc-ai-litellm.secrets.credentials.<provider>_api_key like every
other credential. Each carries the `type` and `regex` its value must satisfy and
defaults to the central API.<provider>.api_key block, which keeps one
operator-facing place to fill a provider in while the role owns the shape.
type and regex had to become real to be worth declaring. 192 `validation:`
blocks sit in the role schemas today and not one line of Python enforces them,
so a `regex` would have joined a pile of decoration. validate_supplied_value
now runs on every path a value arrives from outside — the --set override, a
value already in the inventory, and the value standing next to a `default:`.
A generated value stays exempt because the algorithm that produced it is named
by the same schema and generate_value takes no schema argument. min_length is
deliberately left unenforced: the generators ignore it, so holding them to it
would fail where nothing is wrong.
The pattern is written for the paste, not for the vendor. OpenAI moved issuance
from sk- to sk-proj- without notice and Anthropic issues sk-ant-api03-; a schema
pinning a prefix would reject a valid key and abort inventory creation. `\S{20,}`
accepts every form those three issue today and one nobody has announced, and
rejects a trailing newline, an embedded space, a truncated copy and a
<placeholder>. A key pasted into the wrong provider's slot is caught at deploy
instead, where the route it publishes answers nothing.
The gateway iterates one LITELLM_REMOTE_BACKENDS list instead of three `{% if %}`
blocks; config.yaml.j2 and env.j2 read the same list and selectattr('key')
decides. LITELLM_CHAT_MODEL falls back to the first configured remote alias
rather than to a hardcoded openrouter/auto.
For CI the keys travel as INFINITO_API_{OPENAI,ANTHROPIC,OPENROUTER}_KEY through
default.env, the passthrough allowlist, the infinito service in compose.yml, the
workflow env block and the development inventory. The compose leg is not
optional: the env lint rejected the change without it, because a key the
inventory reads but compose does not forward resolves to empty inside the
container with no error. The workflow reads `secrets.X || vars.X` so a variable
works as asked while a secret wins, which matters because a variable is not
masked and the development inventory sets MASK_CREDENTIALS_IN_LOGS: false.
An unset key is not an error anywhere: the provider's routes stay unpublished
and the CLI test asserts their absence rather than skipping.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gateway publishing a model proves nothing about a backend answering it: LM
Studio shipped a published lmstudio/default route for as long as it had no
model to load. This CLI test closes that gap by asking the running gateway what
it serves and comparing it against what the deployment enabled.
templates/test.env.j2 is both the input and the entry point — discover_cli_roles
finds a role by that file and filters on group_names, so the test runs only
where the gateway is deployed. meta/tests.yml names the container service key so
run_one.yml resolves the node holding it in swarm, and allows 600s for a cold
model load.
probe.py runs inside the gateway container, where the API answers on loopback
and the backends resolve by name, and uses the standard library only because
the image ships the litellm app rather than a test toolchain. It asserts that
every declared model is served, that the LM-Studio-only aliases are present
exactly when that backend is deployed, that no Ollama-backed model is published
without Ollama, that LITELLM_CHAT_MODEL is routable, that an empty deployment
publishes nothing, and that a real completion comes back non-empty.
The fourth assertion is the one worth having: LITELLM_CHAT_MODEL is built from
lookup('deployment').inventory while the config is built from this host's
group_names. Two sources for one fact, and this is where they are made to
agree.
The alias list is rendered whether or not LM Studio is deployed. Gating it
would empty the set in exactly the case the negative direction needs, turning
"a route must not stand without its backend" into a vacuous check.
The verdict logic sits in a pure evaluate() so it is exercised without mocks:
four deployment shapes, each with its positive and negative cases, including a
both-backends case that asserts a shared alias proves nothing about LM Studio
and only the exclusive one counts.
files/test/.nocheck is required rather than convenient. role-files-layout sorts
files/ by language while test-e2e-cli stages files/test/ under that fixed path;
the other 29 CLI test directories carry the same marker.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Variant 0 pinned both local backends on, so LITELLM_CHAT_MODEL always resolved
to an Ollama model and every completion went there. The LM Studio route was
published and never travelled. A third variant turns Ollama off and LM Studio
on, which is the only shape that exercises it, and variant 0 pins LM Studio
literally off so a WordPress round stops co-deploying a backend it never
prompts: 27 apps become 26.
bond drops to 0.5 and that is what makes the variant legal. test_order's
_direct_deps filters services on bond >= 1.0 before building a closure, so at
0.5 LM Studio leaves that computation and the new variant no longer violates
"variant 0 is the maximal deploy footprint". The deploy planner does not filter
on bond — resolve_service_dependency_roles_from_config reads enabled and shared
only — so the variant still deploys it. test_bond_baseline takes its bond < 1
branch, which asks for any variant pinning the flag true rather than variant 0
specifically, and variant 2 supplies it. No suppression and no guard change.
bond < 1 states that the provider is a separate-host partner rather than a
same-host service. Reaching the gateway through LMSTUDIO_BASE_LOCAL_URL, a
container-network hostname, is consistent with that.
Measured against the planner: variant 0 resolves 21 apps without LM Studio,
variant 1 stays at 1, variant 2 resolves 8 apps with LM Studio and without
Ollama, and a WordPress round 0 drops from 27 to 26.
The variant is reachable only when svc-ai-litellm is swept as a primary row and
pinned to it: legacy_resolver clamps a non-primary role to the round index, and
a two-round WordPress job never reaches index 2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
svc-ai-lmstudio had no model-load step at all: tasks/00_core.yml set the
once-flag and the container network, nothing else. The gateway published
lmstudio/default anyway, so the route existed and nothing could answer it.
wp-compose-tor.log:48994 shows the route written into the config, :51770 shows
the warm step reaching qwen2.5:0.5b on Ollama instead, and the green ai-engine
spec ran against that. LM Studio was deployed, published and never prompted.
The pre-pull mirrors the one that works. tasks/01_manager_ops.yml resolves the
hosting node and container id, loops the declared models through
`lms get --yes --gguf`, overlaps them with async/poll and reaps before
playwright, exactly like svc-ai-ollama. That `lms` exists is measured, not
assumed: /app/daemon-run.sh:67 in the image starts ./.bundle/lms server start,
and `lms get --help` documents -y for scripting and --gguf for the format.
Synonymity needs both namespaces in one place. Ollama knows qwen2.5:0.5b, LM
Studio knows qwen2.5-0.5b-instruct, so AI_LOCAL_MODELS carries both per entry:
`alias` is what the gateway publishes and what Ollama pulls, `name` is what
`lms get` resolves. Both roles now reference that one list, and config.yaml.j2
publishes each backend under the alias, routing Ollama to ollama/<alias> and LM
Studio to openai/<name>. The lmstudio loop skips an alias Ollama already
publishes, so two backends never produce a duplicate model_name and Ollama
keeps precedence. lmstudio/default is gone; the model name is now the same
whichever backend answers it.
Ollama's preload_models changes shape from a list of strings to the {alias,
name} entries. Every consumer moves with it: the pull command, the template
loop, and both expressions in 16_ai.yml. An inventory that overrides
`applications.svc-ai-ollama.services.ollama.preload_models` with the old string
list will break at map(attribute='alias'); this is a deliberate shape change,
in the spirit of 7b15c61dfc.
The CI model list is not a property of an installation, so AI_LOCAL_MODELS_CI
lives in inventories/development/default.yml, whose own header scopes that file
to the cross-cutting environment knobs. test_variable_definitions only knew
group_vars, so it reported the constant as undefined; the shipped inventories
are a definition source too, because the deploy CLI bakes them into host_vars,
and the guard's own docstring already claimed "defined somewhere in the
repository".
A latent crash came out of this: `| list | first` raises on an empty sequence,
so a backend deployed without preload models blew up LITELLM_CHAT_MODEL. Ollama
with only embedding models could already hit it; the existing test dodged it by
rendering only _SERVED. `| default('')` now yields the empty string that
_SERVED already reports as false.
Tests: config.yaml.j2 is rendered rather than read, over ollama-only,
lmstudio-only, both and neither, asserting the published model_name set, that
both backends name the same alias, that a shared alias is published once and
routed to Ollama, and that the render is valid YAML. test_litellm_chat_model
gains the lmstudio branch, the no-preload case and the same-model assertion.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
At -vvv Ansible prints each module argument and result, so a password in
a command line, a request body, a header or a fact lands in the deploy
log. Nothing enforced masking outside the MCP task files, and no check
looked at parents at all.
A new lint, test_credential_tasks_are_masked, finds tasks whose printed
parts (module arguments, environment, loop) reference an identifier ending
in password, secret, token, api_key or private_key, or
secrets.credentials. A task passes when it or an enclosing block sets
no_log, when every file that pulls it in does so through include_tasks
with apply.no_log or import_tasks with no_log (transitively), when the
secret only reaches a parameter the module masks itself, or when the
module masks its values. no_log on an include_tasks task itself does not
count, because it does not reach the included tasks. The rule can be
suppressed per task with `# nocheck: credential-masking`, now listed in
the suppression catalog, and the lint tests its parent and suppression
logic on synthetic cases.
It found 39 unmasked tasks in 19 roles, all real: join tokens, client
secrets, API tokens in Cloudflare headers, database passwords in
command lines, the ChatGPT bot login body and more. Each now sets
no_log from MASK_CREDENTIALS_IN_LOGS. The sys-dns-hetzner-rdns alias
hetzner_no_log, which only restated that switch, is gone.
Not covered yet: template and copy tasks whose rendered file carries a
secret print it as a diff under --diff.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34421944328 web-app-litellm#1 (README host-mode replay) answered
every health probe on llm.ai.infinito.test with 502: litellm
crash-looped on "P1000: Authentication failed against database server
at postgres" while postgres logged 'Role "litellm" does not exist'. The
README inventory has no svc-db-postgres group, so sys-svc-rdbms skipped
the central provisioning, yet svc-ai-litellm pinned postgres.shared to
true and kept consuming a shared database nobody had created its role in.
postgres.shared now reads "{{ 'svc-db-postgres' in group_names }}":
an inventory with the central group keeps the shared database, the
README inventory gets a dedicated one.
That dedicated instance, rendered by sys-svc-rdbms postgres.yml.j2,
joins only the compose default network, and the networks renderer
dropped default from every svc-db-*/svc-ai-* role. A local variant-1
deploy measured litellm-database on litellm_default alone and litellm on
the shared litellm network alone. _suppress_default now keeps default
when the role's database is local. The variants pin shared to true (0)
and false (1) so the matrix exercises both paths, and the generated
READMEs show the postgres relation as 0..1.
The three attachment helpers move from render.py to attachments.py,
which render.py re-exports, and test_render.py splits along the same
seams into five files plus _render_helpers.py with every test id kept.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The marker names the reason a role has no upstream homepage or demo video, and
it sat in meta/main.yml because a comment-only info.yml parses to an empty
document, which the schema lint rejected outright. So the exemption lived one
file away from the two fields it explains, and a reader looking for the video
found neither the field nor the reason it was missing.
The schema lint now accepts an empty document exactly when it carries the
marker, and that is what allows the move. The media lint reads the marker from
info.yml rather than main.yml, and its failure message names that same file, so
the fix it suggests lands where the absent field would have gone.
A hundred and two roles follow. Every occurrence was a standalone comment on
the second line of main.yml, directly under the document start, so the move is
a line delete and an insert at the same offset. Seventy-three roles already had
an info.yml and gain the line; twenty-nine get one whose whole content is the
document start and the marker. All hundred and two reasons carry over verbatim,
including the thirty-five bespoke ones.
Nothing downstream sees a change. Of the four readers of meta/info.yml, three
coerce a null parse through `or {}` behind an isinstance guard, and
utils/cache/applications.py skips a falsy parse before assignment, so
applications.<role>.info stays absent for the twenty-nine exactly as it was
while the file did not exist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
13f5ca6d95 put upstream_timeout above every consumer budget on the reasoning that the gateway should only fire once each caller had already given up. That enumerated the consumers and missed a proxy inside one of them: Zammad ships its own nginx with a fixed proxy_read_timeout of 300 in location /, and the role mounts no override. At 420 that hop always won, so a wedged backend would reach the caller as an opaque HTML 504 rather than a gateway error carrying a message.
240 sits below the tightest hop in the path. The front proxy allows 900s and the only tighter timeouts in the repository are the 3s and 5s of the Prometheus healthz routes, which are not on the AI path. Measured on qwen2.5:0.5b, the model the matrix serves: a generation bounded at max_output_tokens 512 finishes in under five seconds, so no legitimate answer under the configured cap can approach the new ceiling.
This does NOT fix Zammad#2 of run 33923190220, and an earlier draft of this message claimed it would. The rescue artifacts say otherwise: ollama logged no generation at all during the three attempts at 09:00, 09:06 and 09:11, litellm had been up since 08:28:46 with zero restarts, and each attempt hung for exactly 300 seconds before rails raised AI::Provider::ResponseError from custom_open_ai.rb:40. The gateway never saw the request, so no gateway timeout can act on it. The same path had worked minutes earlier, warm-up 2.69s and the deploy's provider ping 10.84s from inside the rails container. Rails and litellm share the litellm overlay but sit on different swarm nodes, so that failure belongs to cross-node overlay connectivity, not to this setting.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gateway registered ollama and lmstudio with an rpm cap and nothing
else, so a single request could generate without an output limit and hold
a worker for as long as the backend kept producing tokens. Requests per
minute bound how often a caller may ask, not how much one answer costs.
max_output_tokens caps the answer itself. upstream_timeout is the backstop
for a wedged backend, not a latency budget, so it sits above every
consumer's own budget: the longest wait on the AI path is 360s
(discourse-ai.spec.js), with 300s and 240s behind it, and TIMEOUT_FACTOR
defaults to 1. At 420s the gateway can only fire once every caller has
already given up, which keeps a slow generation from being reported as a
gateway error instead of the slow answer it is.
openrouter/auto stays unbounded here. It never carried an rpm cap either,
and its limits are the provider's rather than ours, so giving it one is a
separate decision from closing the local-backend hole.
The gateway publishes only models it has a backend for and renders model_list: [] when neither ollama nor lmstudio is deployed and no OpenRouter key is set. LITELLM_CHAT_MODEL fell back to openrouter/auto without checking that key, so consumers were configured for a model the gateway does not carry and every prompt failed with a 400.
Observed in the Mattermost#1 CI job: litellm.log carries 12 occurrences of "Invalid model name passed in model=openrouter/auto" next to 400 Bad Request, and tests/test-litellm-agent.js failed on every attempt while the rest of the suite passed.
Make the fallback conditional on the key so the name is empty when nothing is served, promote LITELLM_CHAT_MODEL_SERVED from the role vars to group_vars where consumers can see it (it was the third copy of this logic and the only correct one), and switch the 14 consumers from "litellm is deployed" to "litellm serves a model". The Playwright service flag reads the same lookup, so the spec skips together with the Ansible gate. The bool filter carries weight: the folded scalar yields the string False, which Jinja treats as truthy.
A new unit test renders both expressions against Jinja and pins the invariant that a model name exists exactly when a model is served, including the case where ollama preloads embedding models only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The Production block never set one. inventory provision writes only
ansible_connection for a local host, so DOMAIN_PRIMARY stayed at the
group_vars default of infinito.localhost and the deployment served a zone its
own resolver knows nothing about. CoreDNS templates answers for
INFINITO_DOMAIN_RE and nothing else, so every lookup of the deployment's own
name came back NXDOMAIN.
Mastodon is where that surfaced. tootctl accounts create validates the address
through EmailMxValidator, which resolves MX and then, for the domain itself,
A and AAAA. All six queries returned NXDOMAIN in run 33574056480, the
validator recorded email :unreachable, and the admin account was never created.
Only the guide rows were affected: the matrix rows write
DOMAIN_PRIMARY: "{{ lookup('env', 'INFINITO_DOMAIN') }}" into their host_vars
and resolve fine.
The template now carries DOMAIN=<your-domain> in both the host and the
container form and threads it into --vars, with the container form passing it
through the docker run boundary as well. The two guide runners substitute it
the way they already substitute the SSH key, so the replay deploys under
INFINITO_DOMAIN like every other CI deploy. compose_deploy.sh moves its env
load ahead of the substitutions to have the value; host_deploy.sh had no env
load at all.
A production reader is asked for a domain now instead of silently inheriting
infinito.localhost, which no real deployment wants either. The shipped default
keeps its value: sys-svc-nss documents that *.localhost resolves through the
myhostname NSS module without any zone, and host-side tasks lean on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four tasks in this file talk to the gateway, and each carries its own
tolerance for it not being up: 60 seconds for the keys, 50 for the closure
proof, 50 for the model warm-up. The first one to run is the keys, so a gateway
that takes longer than a minute fails there and the later budgets never apply.
54a6f3b533 already named this when it added the warm-up: "Five retries cover a
gateway that is not listening yet." It gave the last task 50 seconds and left
the first one at the 60 it already had.
One wait in front of all four, on the signal the failure actually reports.
CI shows ConnectionRefusedError from urllib after eleven attempts, which is the
TCP layer, so the probe opens a socket rather than asking an HTTP endpoint whose
path would be a guess. python3 is what the failing task itself runs in that
container, so nothing new has to exist there.
A healthcheck would have been the obvious shape and would have changed nothing:
sys-svc-compose/tasks/utils/post_deploy.yml waits on the database container's
health in compose and on swarm tasks being Running in swarm, and reads the
application container's health in neither.
This does not explain every job in that cluster. One role's rescue artifact
shows all three replicas reaching "Proxy initialized with Config" while
postgres resets and then terminates their connections (SqlState 57P01), which
is a different failure wearing the same stack trace. Where that is the cause,
this makes the deploy fail at a task that names the gateway instead of at the
one provisioning keys.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role declared sso.enabled false behind three suppressions, so the checks were switched off rather than satisfied. LiteLLM authenticates its own admin UI instead of sitting behind an oauth2 proxy, so the declaration is flavor oidc and the client id, the three Keycloak endpoints, the proxy base URL and the admin id all render together under one flag: a half-set group would let the gateway fall back to its own form with nothing complaining. The variants pin both states so the matrix runs with and without SSO.
The proof has to measure the departure, not the arrival: a gateway ignoring the client id answers /sso/key/generate with its own username and password form and never redirects, so the spec asserts the round trip to Keycloak and back. The guest spec became SSO-aware, since its canonical-domain and password-field assertions only hold where no SSO is configured.
Two more gateway defects: the key warm-up ran even where no model was served, falling back to openrouter/auto so the gateway then answered every prompt with 400, and it is now gated on a served-model expression mirroring the model selection itself. A lint requires every declared svc-ai backend to have a model_list branch, because a backend without a route is an alias nobody serves. probe_auth.py proves revocation rather than creation: it mints a throwaway key, revokes it, and fails unless a real consumer key survives.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DATABASE_URL was assembled by hand from five separate database lookups and
string concatenation. That form cannot encode special characters: a password
containing @ or / takes the URL apart. Read
lookup('database', application_id, 'url_full') instead, which the lookup
documents for exactly this.
LITELLM_CONFIG_PATH_DOCKER hardcoded /etc/litellm/config.yaml although
meta/volumes.yml already declares the litellm_config volume. Read its target
so the declaration and its use cannot drift.
Both lookups are established: url_full is documented in database.py and
lookup('volume', ...) is used by 51 roles. litellm ran on these lines
throughout the session, reaching postgres, issuing virtual keys and serving
completions. The evidence is that it worked, not a value-by-value comparison
of the two url forms.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The tor merge brought svc-net-tor in on 192.168.74.0/24 and left
web-app-socialhome on 192.168.75.0/24, both of which the AI services already
claimed. Move litellm to 192.168.80.0/24 and lmstudio to 192.168.108.0/24;
each is unique across the repository and nothing hardcodes the old values.
A different layer from 0f6728a4c: that one moved Tor's virtual address range
out of the docker pool in 10/8, this one resolves concrete docker networks in
192.168/16.
Both services ran on these subnets throughout the session; litellm answered
consumers on 192.168.80.6.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
acc9e25c0f moved the volume lookups to context-wired signatures, taking
application_id from the task context instead of an argument:
lookup('container_volumes', application_id, service_name) -> (service_name)
lookup('compose_volumes', application_id) -> ()
Six templates were left on the old form: the three AI services and hermes,
homeassistant and openclaw. One of them also picks up the matching
container_healthcheck lookup.
This is the same failure mode as the php paths: a refactor that leaves
callers behind, and the break only appears when the template renders, not in
the gate. A deploy in this session died exactly that way, on a template still
passing application_id to a lookup that no longer takes terms.
No file in the repository carries the old signature after this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eleven roles still declared their credentials in meta/schema.yml while the
rest of the repository had moved to meta/secrets.yml. The name is not
cosmetic: the inventory resolves credentials at that path, as
d86913225b and c541e7c22f record, so a role left on the old name drops out
of that resolution.
All eleven are pure renames with no content change.
Also correct the four remaining places that still named the old path in
prose: the credentials reset CLI help, two test docstrings and the espocrm
README. None of them resolved a file, but they misdescribe where credentials
live, which is exactly what sent an earlier reader to the wrong location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A backend pulls its preload model at deploy time but only loads it into
memory on the first inference. That cold load outlives a reverse proxy's
read timeout, so the first consumer request comes back as a 504 while every
later one succeeds.
Send one max_tokens=1 completion from inside the litellm container right
after the per-consumer keys exist, where the container id, master key and
port are already resolved. Five retries cover a gateway that is not
listening yet.
This pairs with the ollama keep_alive setting: the warm-up pays the load
once, keep_alive stops it decaying after five idle minutes.
Note this hardens the deploy: a gateway that cannot serve its configured
model now fails the role instead of surfacing later as a consumer error.
That makes the openrouter/auto fallback with its empty API key loud rather
than silent.
The script ships with a mirrored unit test rather than a nocheck waiver; it
patches urlopen and asserts the URL, method, authorization header, model,
max_tokens and that the timeout exceeds a proxy read timeout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merged branch moved every role-local script out of files/ into a folder
named for its language (files/python/, files/php/, files/shell/), but the
moves lost their staging and never landed. HEAD still carried the old paths
while the worktree had the new ones.
Move the 24 remaining scripts and update the six places that resolve them:
the mcp-adapter instance/isolation/probe tasks, the litellm key task, the
adapter README and the endpoint-path lint. Every changed line in those
files is a path.
Also drop roles/sys-svc-compose/files/swarm/stack_ready.sh, a stale
duplicate of the copy already tracked under files/shell/swarm/.
The added comments are the branch's own `nocheck: mirrored-unit-test`
directives, which state why a script whose measurement only exists inside
its container cannot have a mirrored unit test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The requirement corrects two rows against what the pinned releases actually
expose: Moodle 4.5 configures aiprovider_openai per action rather than through a
single base URL, and n8n 1.95.3 reads its endpoint from a stored credential,
after N8N_INSTANCE_AI_MODEL_URL turned out not to exist in that release. espocrm
and peertube move into the exclusion table with measured evidence rather than
vendor claims, and the variant-coverage section records which round exercises the
gateway.
Criteria 2, 5, 8 and 9 are ticked. Criteria 3, 4, 6 and 7 stay open on purpose:
the wiring is complete across all nine confirmed roles, but only zammad's spec
has been observed answering a prompt through the gateway, and the last deploy
ended before Playwright ran. Criterion 1 is still open because homeassistant,
xwiki and baserow have not been probed.
The 027 generator gains litellm as a framework key and litellm and n8n on the
web-app axis, so every consumer's gateway edge appears in the matrix. Its
regenerated output is committed alongside it; the generator writes the file in
place rather than to stdout, and a fresh run reproduces the committed matrix
byte-for-byte. Touching the generator also brought its comment history into the
lint's scope, so about thirty section banners and narration lines go with it.
The espocrm README carries the exclusion evidence -- at tag 10.0.4 only Crm ships
under application/Espo/Modules, the config metadata declares no AI key, and the
release ZIP leaves custom/Espo/Modules empty -- plus four steps for enabling the
commercial Intelligence extension once licensed. A version bump does not help:
upstream documents that extension as requiring 10.0.3 or greater, so newer cores
host it rather than absorb it. The provider field names are only discoverable by
unpacking the purchased artifact, and the note says so rather than guessing.
The root README and the litellm README are generated; both now show the gateway
among the dependencies of its eight consumers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A role that enables its native AI surface can reach the gateway with an empty
credential, or point its base URL at something that resolves outside the
cluster, and still deploy green. Both are silent: the surface simply answers
from somewhere else, or not at all. This adds the shared contract that makes
either state fail the deploy, so a consuming role inherits one definition of
"inside the deployment" instead of restating it.
The key assertion refuses an empty credentials.litellm_api_key while
services.litellm is enabled. Without it the gateway leg is unauthenticated and
nothing in the run says so.
The URL assertion is four independent checks, each able to fail on its own:
the scheme is plain http, the hostname carries no dot, the hostname is not the
consuming role's own domain, and the port is the gateway's internal port rather
than its published one. The undotted check is the load-bearing one. The
container dns-search suffix completes a dotted host into the public FQDN, so
`litellm.infinito.example` leaves through the ingress while looking local in
the config; only a bare service name stays on the container network. The
own-domain check catches a role that points its AI surface back at itself, and
the internal-port check catches a config that would work only via the host
bind.
An earlier revision of this file compared two values that both came from the
same lookup and therefore could never fail. Every check here is falsifiable
against the value the role actually deploys.
group_vars carries the SPOT the contract measures against.
LITELLM_OPENAI_BASE_LOCAL_URL appends the OpenAI-compatible /v1 suffix once,
rather than in each role. LITELLM_CHAT_MODEL derives the model from whichever
backend is deployed -- an ollama preload model with embedding models rejected,
else lmstudio, else openrouter -- resolved through `lookup('deployment').running`
so it reflects what is running rather than what the inventory could contain.
Contract and vars land before the roles that include them; the reverse order
would commit roles referencing a file that does not exist yet.
Verified: the contract's assertions ran green at runtime for matrix,
mattermost, moodle and zammad. Zammad's is the strongest evidence, since
`Setting.set('ai_provider_config')` triggers upstream
`AI::Provider::CustomOpenAI.ping!` and therefore a live gateway round trip.
`make quality` green on test-lint, test-unit, test-integration, test-external.
Not covered: nothing yet forces a role that declares services.litellm to
include this contract, so a future consumer can still skip it silently.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An inventory re-initialization mints a new virtual key while the proxy
database keeps the previous one under the same alias. Aliases are unique,
so creating the new key was refused and the constructor aborted after ten
identical retries. The alias is now released before the key is created,
and the release tolerates finding nothing.
The script moves out of the task body into files/ensure_keys.py, with the
port passed as an environment variable instead of templated into the
source, so it is plain Python and covered by unit tests.
Open WebUI talked to Ollama directly. It now speaks the OpenAI-compatible surface of svc-ai-litellm with a per-consumer virtual key, so a second backend or an external provider becomes a gateway config change instead of a role change. A lint keeps future roles off the model servers.
The gateway advertised a hardcoded model list (ollama/llama3, ollama/nomic-embed-text) while the matrix variant preloads smollm2:135m, so it offered only models the backend did not have and hid the one it did. The list is derived from the ollama preload SPOT now.
The chat spec followed the rename: it gates on litellm instead of the removed ollama service flag, and picks the first non-embedding model from /api/models rather than pinning a variant-specific name, so it exercises the chain rather than a fixture.
Routing openwebui through the gateway pulls both model backends into the closure of every role that consumes it, which pushed nextcloud variant 2 over the memory budget. The budget stays at 32/64 GB; instead the matrix variants of svc-ai-ollama and svc-ai-lmstudio now declare the size their test model needs (1g/2g) while the base keeps the 6g/8g that llama3 and mistral require. The budget guard only saw half of that: it replaced the evaluated role with its variant but counted every dependency from meta/services.yml, a combination no matrix round deploys. It now evaluates every variant-carrying role at the same round index, falling back to variant 0, the way the deploy planner does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The eight new roles shipped with the README template unfilled: every one
of them described itself as "[Name](https://example.com/) is an
application" with a single "Describe a capability" feature. They were
implemented, deployed and tested; their documentation was placeholder
text.
Each role now carries a Description, Overview and Features written
against its own services, tasks and templates, with TODO.md as the
binding list of what is not wired. That caught a false claim: OpenClaw
advertised routing through the shared gateway, but it reads model
providers from openclaw.json rather than the environment, so the gateway
only shapes its deploy topology. The claim is gone.
The overview table derives its Name column from each README H1, and all
eight had kept the humanised fallback. Worst of these, both LiteLLM roles
rendered as "Litellm" on adjacent rows, so a reference in the Integrated
with column could not say which one it meant; they are now the Gateway
and the Admin UI. The rest take their official spelling.
The galaxy descriptions all repeated the product name the Name column
already carries, one named a role id in prose and one used a hyphen as a
dash. svc-virt-kata also claimed to install Kata, which it does not: it
installs gVisor and registers an already-present Kata shim.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Consumers of the model plane speak one OpenAI-compatible endpoint and
carry one virtual key; which backend answers is the gateway's business.
svc-ai-litellm renders its model_list from the round's actual topology,
so an alias only exists when its backend role is in group_names, and it
emits an explicit empty list when no backend is present at all because
LiteLLM rejects a bare model_list: key.
Keys reach the gateway through the per-consumer credential rather than
being minted there: 01_keys.yml probes /key/info and only POSTs
/key/generate on 404, so reruns stay idempotent. The consumer list is
built in 00_core.yml, not inside the included file, because 01_keys.yml
delegates to the hosting node and delegate_to is templated before when.
The gateway declares ollama and lmstudio as consumer flags so the
network renderer attaches it to its own backends; without them it cannot
reach them.
svc-ai-lmstudio runs the headless llmster daemon behind expose only, so
it is reachable from the gateway but never bound on the host, matching
svc-ai-ollama.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
