#832 probes /healthz/ready on every consumer and proxies it to the app root,
where Bridgy Fed, Hermes and the XMPP web listener answer 404 by design, so
their probe_success stayed 0. A consumer can now name the path in
services.prometheus.ready_path; the accepted status codes stay unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Fourteen roles repeated the same two lines to say they use the gateway: a
`shared` flag gated on group_names, and an `enabled` flag conjoining that gate
with LITELLM_CHAT_MODEL_SERVED. Both are now single-sourced.
`shared` becomes literal true. The flag means the provider is a shared instance
rather than one deployed per consumer, and a central gateway is always that;
the dynamic form only said "shared if deployed", which `enabled` already
decides. The 50 shared pins under `litellm:` leave meta/variants.yml with it.
The tree had already noticed: 20 of those 50 lines carried
`# nocheck: variants-static-override`, two of them with copy-pasted text about
container_backup. This narrows e5de4e085a for one service key rather than
reversing it; every other dynamic flag keeps the group_names form.
`enabled` becomes `{{ LITELLM_USABLE }}`, defined in group_vars/all/16_ai.yml.
That shorthand is only possible because is_explicit_truth learns about it.
The resolver reads a dependency out of the flag TEXT — "in group_names" in
value — so a constant hides the role id and the provider silently leaves the
static closure of every role referencing it. Measured: the old expression
resolves True, `{{ LITELLM_USABLE }}` resolves False. GROUP_GATED_CONSTANTS
names the constants whose own definition carries the gate, and the docstring
states the obligation: a SPOT of this kind that is not listed there drops its
provider without a word.
The wider fix, accepting any Jinja as dynamic, was measured and rejected: 35
other flags carry a Jinja value without the marker (MCP sidecars, javascript
and SSO gates, recaptcha), and they would all enter closures they are not in
today.
A literal shared flag is no longer dynamic, so test_dynamic_flags wants the
`# nocheck: dynamic-flag` its own message documents for legitimate exceptions;
where a marker already existed the rule list is extended rather than a second
comment added.
Behaviour is unchanged where it counts. The closure keeps litellm with the same
app counts (wordpress 26, openwebui 23, hermes 23, mediawiki 22, flowise 23),
unrelated Jinja still resolves False, and the truth table of the new enabled
form matches the old one in all four served/deployed combinations. Three roles
resolve without litellm, and `git show HEAD:` proves that predates this change:
zammad pins enabled false in variant 0 at line 39, nextcloud at line 97.
Six unit tests cover the resolver, including one asserting that every
registered constant is actually defined in group_vars, so an entry cannot
become a dead promise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
confluence 10.2.16 to 10.2.18, erpnext v16.34.1 to v16.34.2, espocrm 10.0.7 to 10.0.8, hermes v2026.7.20 to v2026.9.7, mattermost 11.10.1 to 11.11.0, nextcloud context_agent 2.7.0 to 2.8.0, seaweedfs 4.45 to 4.46, coturn 4.17.2 to 4.18.0. The automated updater opens one PR per role and had not reached these eight; playwright and elasticsearch were already current in the tree.
audit.md follows rather than leads: the MCP report carries a version column and test_mcp_audit_report compares the committed file against a fresh generation, so hermes and mattermost move there too. Regenerated with audit.gen.py as that test instructs. Note that make docs does not run this generator, so the report only tracks reality when somebody calls it by hand.
Verified by the docker-version check itself: eleven Outdated warnings before, zero after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gateway publishes only models it has a backend for and renders model_list: [] when neither ollama nor lmstudio is deployed and no OpenRouter key is set. LITELLM_CHAT_MODEL fell back to openrouter/auto without checking that key, so consumers were configured for a model the gateway does not carry and every prompt failed with a 400.
Observed in the Mattermost#1 CI job: litellm.log carries 12 occurrences of "Invalid model name passed in model=openrouter/auto" next to 400 Bad Request, and tests/test-litellm-agent.js failed on every attempt while the rest of the suite passed.
Make the fallback conditional on the key so the name is empty when nothing is served, promote LITELLM_CHAT_MODEL_SERVED from the role vars to group_vars where consumers can see it (it was the third copy of this logic and the only correct one), and switch the 14 consumers from "litellm is deployed" to "litellm serves a model". The Playwright service flag reads the same lookup, so the spec skips together with the Ansible gate. The bool filter carries weight: the folded scalar yields the string False, which Jinja treats as truthy.
A new unit test renders both expressions against Jinja and pins the invariant that a model name exists exactly when a model is served, including the case where ollama preloads embedding models only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two leftovers of my own. The sweep that removed the reverse client entries
appended a newline where it cut a block, which at end of file leaves a blank
line after the last key in four services.yml. And the two cache-read nocheck
comments in the bond editor were written past the line limit, so the
formatter breaks their read_text calls across three lines and carries the
comment to the closing paren.
The wrap does not cost the suppression: test_no_uncached_file_reads matches
the statement, not the line the directive sits on, and passes on this form.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four MCP clients each carried a services entry for homeassistant, and
openwebui and flowise one for baserow, with matching run_after lines. Those
predate the admission refactor: since then the co-deploy edge lives on the
provider, where mcp-consumer-services-entry enforces it, and homeassistant
and baserow already carry an entry for every client they admit. The client
copies stated the same relation a second time.
The asymmetry gives them away. Open WebUI consumes twenty MCP servers and
declared three of them; the other seventeen work because the bridge is built
from MCP_DISCOVERED_SERVERS, never from the client's own services map.
Drop the run_after lines in the same move: a run_after target must be backed
by a services entry, so one cannot go without the other. Hermes and OpenClaw
lose the whole key, homeassistant being its only entry, because a bare
run_after parses as null rather than an empty list.
Keep qdrant on flowise, which renders QDRANT_URL and QDRANT_API_KEY into the
container and is a real dependency, and keep prometheus everywhere as the
ordinary metrics edge.
Open WebUI stops dragging both providers into every round it joins, which
brings moodle variant 2 from 64.75 GB back to 62.75 GB, under the budget the
preceding commit pushed it over.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
83c1c729de gave every provider a group_names-gated services entry per MCP
client, including the clients it refuses with mcp_consumer: false. The gate
is what makes the inventory resolver plan provider and client into one round,
so 19 of the repo's 82 co-deploy pairs started a client for a pairing the
provider had already declined. flowise is a live MCP client of qdrant,
homeassistant and prometheus; the other 17 providers were bringing up its
node stack for nothing.
The entries now read enabled: false, shared: false, mcp_consumer: false, and
the variants that pinned them to true are gone. Setting those variants to
false instead is not an option: test_variants_no_static_override forbids any
variant override of a literal base value in either direction.
bond leaves with them, and a new lint holds that line: collect_role_resources
skips a literally disabled entry before it reads any resource key, so a bond
there describes a cost nobody pays. Thirteen entries unrelated to MCP carried
one and are cleared too.
test_mcp_consumer_services_entry demanded the group_names shape for every
client entry, refused ones included, and now demands the opposite there.
dynamic-flag keeps its rule and takes its own documented per-key exemption.
make docs redraws twenty graphs: the edges go, the nodes stay as
'openwebui ❌' and 'flowise ❌', so the refusal is visible rather than read as
an integration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
allowed_consumers was twenty hand-kept lists of which fourteen were identical, so every new client meant an edit in every provider. A client now declares services.<own-entity>.mcp_consumer: true once and a provider refuses one with mcp_consumer: false, which is also what tells a deliberate exclusion apart from a forgotten entry.
The group_names-gated consumer entries give the inventory resolver the co-deploy edge meta/mcp.yml never had: one of sixty-nine matrix rounds carried an MCP client before, twenty-three role/round pairs do now. bond 0.5 keeps the clients out of round 0 so the host memory budget still holds.
BREAKING CHANGE: meta/mcp.yml no longer accepts allowed_consumers; declare services.<entity>.mcp_consumer on the client instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four roles gained a canonical domain without a tor service block. Declare it
the way 84b22b37a8 did for n8n: bond 0.5, enabled on 'svc-net-tor' in
group_names, shared, with the variant pins alongside.
The nocheck directives carry the two exemptions: the service has no persona
surface for the playwright flag, and the sso-variant-tor combination requires
the onion in every variant, so the false polarity has no variant left to
live in.
hermes also moves its local http port from 8089 to 8092, because 8089
belongs to web-app-socialhome. That is a separate concern riding along in the
same file; splitting it would need interactive staging.
This is merge work rather than authored here: it follows the established
pattern and the port collision is real, but none of it has been deployed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The marker claimed concurrent replicas serialise into a lock error, a symptom
that never appeared in the logs, and then that NFS carries POSIX advisory
locks unreliably, which the captured mount options contradict: the export is
mounted with local_lock=none over NFSv4.2, so locks do reach the server.
The reason that holds is the one the application states itself. It disables
WAL on a network filesystem because the shared-memory index WAL coordinates
writers through is per host and not coherent between nodes, and the board is a
single SQLite file every replica shares.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
config.yaml reaches the container as a docker config, which swarm mounts
read-only. The image runs its schema migration on every boot, and because the
rendered file carries no _config_version it counts as legacy, so the migrator
backs the file up and writes it back, failing with EACCES every start. The
image offers HERMES_SKIP_CONFIG_MIGRATION for exactly this case, and the
config is rendered fresh on every deploy, so there is no persisted legacy
shape left to migrate.
Drops the volume ownership block that a wrong reading of the same failure
introduced. The rescue artifact of that run shows the directory was already
owned by the runtime user, and the log shows that user creating directories in
it, so the block only widened the mode and fixed nothing.
Keeps the single replica but states the reason the evidence supports: SQLite
on an NFS-backed shared volume, where advisory locks do not carry reliably.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The agent runs as uid 10000 and keeps its kanban state in embedded
SQLite. Without an nfs block the export directory is created root:root,
no_root_squash helps only root, and uid 10000 gets EACCES, which SQLite
reports as "attempt to write a readonly database". Concurrent writers
would report "database is locked" instead, which is how the two apart.
Compose has no NFS bind, which is why only swarm failed.
An earlier series gave five other non-root roles the same block; this
role was the one left out.
replicas: 1 stays as hardening, not as the fix: once the writes land,
three replicas contend for one SQLite writer lock.
The agent drives MCP clients that reach providers with the credentials
the deployment issued, so an unauthenticated surface hands that reach to
anyone. sso was declared off behind a suppression; it is now group-gated
like every other role, admitting the administrator and mcp groups.
The API path and the health endpoint stay open, because clients
authenticate there with their own token and a gate would break both. A
Playwright spec drives the redirect through Keycloak and skips itself
when the service is off.
The block sat under services but never behaved like a service. A service has
exactly one providing role, declared with provides, so the registry can map the
key to that role and consumers bind their flag to its presence. No role declares
provides: mcp. There are seventeen providers and the pairing is done by
allowed_consumers inside the block itself.
Every rule that enforces the service model therefore needed an exemption for it:
a shared-provider suppression in fourteen roles, a playwright-service-flag
suppression, and two freshly added exemptions in the dynamic-flag and
variants-static-override checks. Four independent rules, four escape hatches,
one cause. They are all gone now, together with a bond key that only the
resource budgeter reads and only for entries under services, and a wordpress
addon bridge that pointed at a service block that no longer exists.
The block moves to meta/mcp.yml. The applications loader maps meta/<name>.yml to
applications.<app>.<name> generically, so the move needs no wiring; the file
takes the same shape as meta/rbac.yml. Every config path follows, the variant
overrides are lifted out of their services map, and the vocabulary module leaves
the services package because it no longer describes one. roles_with_service
gains an explicit topic parameter rather than an implicit fallback, so nothing
silently resolves against the wrong container.
Files that are otherwise unrelated to this move are included where the committed
tree would otherwise keep pointing at a path that no longer resolves: thirteen
vars files, eleven playwright env templates and one task file.
Render every discovered server into the agent's own config, filtered to
the auth schemes a client can actually present, so no entry carries a
credential the server would reject.
Render hermes' config after the stack exists and before the handlers
flush; the other order fired compose up while the config was still
missing.
Own the config file as the container user. Rendered as root it stays
unreadable to the agent, which then falls back to its built-in defaults
silently and reports only that the server is unknown, pointing the reader
at the wrong layer.
Assert what these clients expose: neither ships an admin surface listing
its configured servers, so the specs hold the API to refusing an
unauthenticated caller, which is what keeps the per-server bearers out of
reach.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The sso and logout rules are real opt-out mechanisms, named as such in
the failure messages of test_sso and test_logout_dashboard. Replacing
them with dynamic-flag silenced a different rule and broke both lints,
plus test_env_services_match as a consequence.
All three rules apply to these entries: the role opts out of SSO because
an oauth2 proxy in front of a bearer-token API breaks OpenAI-compatible
clients, the static false is deliberate rather than topology-dependent,
and no Playwright spec gates on either service, so rendering an env flag
for them would be noise. web-app-homeassistant already carries the same
three-token form for the same situation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The first agent workload of requirement 032. Its compose service pins
runtime: to the fact svc-virt-kata publishes, so model-generated code
runs behind a kernel or syscall boundary rather than plain runc. The pin
goes through compose_only because docker stack deploy discards the key;
swarm relies on the node-default runtime and a placement constraint
instead.
Hermes is not a dashboard. The container starts its gateway and serves an
OpenAI-compatible API on 8642, which shapes three decisions: the env
block configures API_SERVER_*, inference points OLLAMA_BASE_URL at the
LiteLLM gateway (Hermes' Ollama client speaks the same wire format), and
the vhost declares status_codes [200, 404] because the API server maps
only /health and /v1/*, so the bare root answers 404 by design and the
webserver health unit would otherwise fail the host.
SSO is off deliberately: an oauth2 proxy in front of a bearer-token API
breaks OpenAI-compatible clients. Authentication is API_SERVER_KEY, and
the guest spec asserts that /v1/models without it is rejected while the
administrator spec asserts the key unlocks the listing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
