The gateway could reach exactly one remote provider, through a bare
OPENROUTER_API_KEY group_var with an empty-string default and an `{% if %}`
around a single hardcoded model entry. OpenAI and Anthropic had no path at all.
All three are now declared in roles/svc-ai-litellm/meta/secrets.yml, which
mapping.py already calls the application config schema, so a key lands at
applications.svc-ai-litellm.secrets.credentials.<provider>_api_key like every
other credential. Each carries the `type` and `regex` its value must satisfy and
defaults to the central API.<provider>.api_key block, which keeps one
operator-facing place to fill a provider in while the role owns the shape.
type and regex had to become real to be worth declaring. 192 `validation:`
blocks sit in the role schemas today and not one line of Python enforces them,
so a `regex` would have joined a pile of decoration. validate_supplied_value
now runs on every path a value arrives from outside — the --set override, a
value already in the inventory, and the value standing next to a `default:`.
A generated value stays exempt because the algorithm that produced it is named
by the same schema and generate_value takes no schema argument. min_length is
deliberately left unenforced: the generators ignore it, so holding them to it
would fail where nothing is wrong.
The pattern is written for the paste, not for the vendor. OpenAI moved issuance
from sk- to sk-proj- without notice and Anthropic issues sk-ant-api03-; a schema
pinning a prefix would reject a valid key and abort inventory creation. `\S{20,}`
accepts every form those three issue today and one nobody has announced, and
rejects a trailing newline, an embedded space, a truncated copy and a
<placeholder>. A key pasted into the wrong provider's slot is caught at deploy
instead, where the route it publishes answers nothing.
The gateway iterates one LITELLM_REMOTE_BACKENDS list instead of three `{% if %}`
blocks; config.yaml.j2 and env.j2 read the same list and selectattr('key')
decides. LITELLM_CHAT_MODEL falls back to the first configured remote alias
rather than to a hardcoded openrouter/auto.
For CI the keys travel as INFINITO_API_{OPENAI,ANTHROPIC,OPENROUTER}_KEY through
default.env, the passthrough allowlist, the infinito service in compose.yml, the
workflow env block and the development inventory. The compose leg is not
optional: the env lint rejected the change without it, because a key the
inventory reads but compose does not forward resolves to empty inside the
container with no error. The workflow reads `secrets.X || vars.X` so a variable
works as asked while a secret wins, which matters because a variable is not
masked and the development inventory sets MASK_CREDENTIALS_IN_LOGS: false.
An unset key is not an error anywhere: the provider's routes stay unpublished
and the CLI test asserts their absence rather than skipping.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role merges the gateway service and bot into the stored Agents config and
then cycles the plugin so it adopts them, but agents/reload.yml and
agents/enable.yml both run mmctl in one container (run_once, a single cid).
mattermost-team-edition has no cluster bus, so no other replica hears of it.
Job 102705652088 of run 34421944328 (swarm, three replicas) shows exactly
that in the service log:
02:43:51-52 "Plugin activated" mattermost-ai on .1, .2 and .3
02:44:45 Point the Agents bot at the platform gateway (the merge)
02:44:48 Deactivate the Agents plugin so it drops its cached config
02:44:51 "Plugin activated" mattermost-ai on .1 only
Every post the plugin handled in the test's direct-message channel then
landed on .2 or .3, which still ran the config from 02:43 ("Notification
modified by plugin." at 03:10:30 and 03:25:32 on .2, at 03:35:42, 03:50:42
and 03:52:55 on .3), and the bot never answered:
Timeout 900000ms exceeded while waiting on the predicate ... Received: 0
After the reload the role now forces a rolling update of the service, so
every replica restarts and activates the plugin against the merged config,
and wait_converged.yml blocks until the update has completed and every task
is running again, before the stored config is re-read and asserted. Compose
runs one container, which the reload already covers, so both steps are swarm
only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
svc-ai-litellm has two variants: 0 with both backends on, 1 with both off. The planner assigns a dependency round_index when it has that many variants and falls back to 0 otherwise (inventory/planner.py:111), so round 1 is the one round in which the gateway deliberately serves nothing - a topology that is green in its own job.
Commit e0048f157b gated 14 consumers on LITELLM_CHAT_MODEL_SERVED so a consumer switches itself off when no model is served. Exactly two of them pinned litellm on in round 1, and a variant's literal beats the base Jinja gate, so mattermost rendered LITELLM_SERVICE_ENABLED=true against an empty model_list. The Agents plugin then failed client-side with 'bifrost error: model is required' 47ms after the prompt, never reaching the gateway.
Moving both pins to variant 2 lands them where the fallback supplies svc-ai-litellm=v0 with svc-ai-ollama=v0. Measured with the planner: round 2 pulls both. Both polarities stay pinned, so the variant-coverage lint still holds. The contract assert now also refuses an empty model, which the stored-vs-intended comparison read as converged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The contract check ran as an Ansible task in the middle of every provider deploy, and the two client checks ran the same way. A test that is part of the deploy cannot be run without deploying, cannot be re-run on its own, and reports its verdict as a deploy failure. Twenty-seven task files lose 491 lines to this.
It is now what it always was. Each provider ships files/test/test.sh, templates/test.env.j2 and meta/tests.yml, and test-e2e-cli stages one shared harness beside it: run.sh drives contract.sh in a throwaway container on the provider's own network, then isolation.sh for adapter-fronted providers. The read probe each provider offers moved from the calling task into meta/mcp.yml next to the allowlist it must satisfy, and the schema now rejects a read probe naming a tool outside that allowlist.
The bearer the test presents is the one a client resolves through mcp.credential, via the new mcp_client_bearer lookup over resolve_credential, which moved to utils.roles.applications.mcp so it is the same function mcp_servers renders every client connection with. The probe presented whatever the calling role passed, which for the fifteen adapter-fronted providers is a different secret than any client holds.
Sidecar isolation is read from the engine rather than measured inside the container, so it no longer needs a delegation to the node holding the task: attachment is compared by network name, which drops the gwbridge special case, and a swarm worker says it cannot measure instead of skipping silently.
Retrying is bounded by wall clock against MCP_DEADLINE_SECONDS, which scales through the same lookup('timeout') as the harness budget it has to stay under. A flag that is neither true nor false aborts rather than reading as off.
mcp-provider-probe becomes mcp-provider-cli-test: an enabled provider must ship the test rather than include the task. mcp-probe-bearer becomes mcp-adapter-bearer, tying ADAPTER_BEARER to the declared credential key now that no probe include carries it. A new integration test renders the shared env template for every provider through Ansible's own templar, so a later change to it breaks visibly instead of poisoning twenty deploys quietly.
Hermes and OpenClaw keep their client-side proof as CLI tests of their own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The four SQL bodies behind the Agents integration were Jinja templates read
through lookup('template'), which meant the raw query: parameter and a
"# nocheck: database-query-raw" on all five call sites in
tasks/utils/ai_gateway.yml.
They now use the mechanism the database_query module already offers: static
.sql files under files/sql/, reached by query_file:, with the values supplied
as named_args and %(name)s placeholders. All five suppressions are gone.
Same move as 4e9d4b53db made for odoo, for the reason that commit gives: an
interpolation straight into SQL carries an injection surface that a bound
parameter does not. Postgres dollar-quoting was doing that job here; the
module's own escaping does it now, doubling single quotes per the _ENGINES
postgres entry.
The | to_json filters did not disappear, they moved to the call site.
MATTERMOST_LITELLM_SERVICE and MATTERMOST_LITELLM_BOT are dicts, and the
module raises TypeError on those, so pre-serialising is what it asks for.
set_ai/ became update/ai/ to match the select/ and update/ verbs the role's
existing files/sql already uses.
The api key inside MATTERMOST_LITELLM_SERVICE is now a named_args value, which
the module adds to no_log_values; as part of the query string it was not.
Verified statically only: the database-query-raw lint passes, the task file
still parses to the same 11 tasks in the same order, every placeholder has a
matching named_args entry, and no reference to the old paths survives. Nothing
ran against a database, and the adversarial review of this change had not
finished when it was committed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
set-ai-agent.sql.j2 and set-ai-service.sql.j2 shared the delimiter-bounded
prefix `set-ai`, which the repository lint treats as a folder in disguise.
The same rule was already applied in 8e8dbbd08c.
Move both into templates/sql/set_ai/ and drop the prefix, updating the two
lookups in ai_gateway.yml.
The violation was introduced by 43f6ff62e, which added the second file: the
gate ran before that commit series but not between its commits.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Agents 2.5.1 migrates config->'bots' into its own agents_useragents table on
startup and logs "Migrated legacy config bots to self-service agents table".
The wiring still treated the legacy location as the source of truth, which
broke in three ways.
set-ai-service.sql.j2 re-inserted the bot into config->'bots' unconditionally,
so both stores held a bot named `agent`. EnsureBots aborts on that with
`duplicate bot name: agent`, leaves its in-memory bot list empty, and
handleMessages then returns without logging: no reply, no gateway request,
no error. Only write the legacy entry while no live agent row owns the name.
set-ai-agent.sql.j2 retargets the migrated row, because once
legacy_config_bots_migrated is set, writing config->'bots' is a permanent
no-op and the role could never change its model or service again.
select/ai/service.sql.j2 asserts against agents_useragents and additionally
reports stale when a legacy entry collides with a live agent row; the old
check reported converged over a plugin whose bot list had failed to load.
ai_gateway.yml fixes two orderings: the retarget now runs before the plugin
cycle, because the plugin reads its agents into memory at startup, and the
cycle is unconditional, because a converged-looking database was preventing
the very reload that repairs a drifted plugin.
Verified live: legacy bots 0, no duplicate bot name, the bot posting real
replies, and test-litellm-agent.js passing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_prefix_groups_belong_in_subfolders flagged three groups added by this change
set: a shared delimiter-bounded prefix is a folder in disguise, so the group moves
into a folder named after the prefix and the prefix leaves the file names.
sql/select/ai-{bot-user,service}.sql.j2 -> sql/select/ai/{bot-user,service}.sql.j2
files/provision_{ai,mcp}.py -> files/provision/{ai,mcp}.py
files/ai_engine_{configure,endpoint}.php -> files/ai_engine/{configure,endpoint}.php
All six are recorded as renames, so history follows them. Every reference moves
with them: four templated query paths in mattermost's ai_gateway.yml, the two
lookup('file', ...) calls in n8n, the two in wordpress, and SCRIPT_PATH plus the
module docstring in the n8n unit test. lookup('file', ...) resolves relative to
the role's files/ directory, so the new argument is provision/mcp.py rather than
a repo-relative path.
Worth recording why this survived every gate: make quality was green on each run
this session, including immediately before the commits that introduced these
files. The lint enumerates through git ls-files, and the files were untracked, so
it could not see them. Committing them is what made them visible. A green gate
taken before new files are tracked says nothing about the gate after.
Verified: repository lints go from 88 passed with 1 failure to 89 passed, the
unit suite passes at 4448, and a repo-wide grep for every old path returns
nothing.
Not verified: no deploy has run. The lookup('file', ...) arguments and the
templated SQL paths resolve at play time, so a wrong path would surface as a task
failure rather than a test failure.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Deletions only, no line added. The rationale each of these carried is already in
the commit message of the change that introduced it, which is where it belongs.
Three of the twenty Exception blocks this change set accumulated had already been
committed when the audit ran, so they were invisible to it: an audit that
enumerates git status stops seeing a file the moment it is committed. Two further
details are worth recording because they are what let these survive.
The purge pattern (^|\{)#\s*Exception: does not match an indented comment, so a
block nested inside YAML never matched while the report claimed none remained.
Anchoring it as ^\s*(#|//|--|\{#)\s*Exception: raised the count from 12 to 17.
One of the three was written as a paragraph inside a Python docstring rather than
as a comment. Docstrings are an allowed kind, so no comment grep matches them and
the repository's own comment lint passes over them; using the allowed kind to
carry the banned label is the same rule-gaming in a syntax the checks do not
cover.
Verified the deletion removed nothing else: the diff contains zero insertions,
every touched YAML file still parses, and the unit suite and comment lint are
green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous contract could not fail. Its guard and its closing assertion both
keyed on apiURL -- the one field the merge itself writes -- so once any run had
set the URL, every later run skipped the merge on a false_condition and still
reported all assertions passed. The live row shows what that hides: an apiKey of
`sk-` followed by 48 literal `a` characters, which the gateway answers 401 to,
and defaultModel `llama3:latest` while the round's gateway serves only
`smollm2:135m`. The merge task is `skipping` in every deploy log on this host.
The read now compares the whole desired service and bot with jsonb equality and
answers `converged` or `stale`, so a drifted key or model is drift the guard can
see. jsonb equality is key-order independent, which keeps the check from
depending on how Jinja happened to serialise the dict. Both SQL templates render
from the same two role vars, so read and write cannot disagree about what
correct means. The `jsonb_typeof(...) = 'array'` guards stay: before the first
write those keys exist as JSON null, coalesce does not fire on JSON null, and
jsonb_array_elements then raises "cannot extract elements from a scalar".
A row edit cannot create the bot account, which is why the Playwright spec 404s
on user `agent`. In mattermost-plugin-agents v2.5.1 the config lives in an
in-memory atomic pointer; bots.EnsureBots() is the only path that calls
pluginAPI.Bot.Create, and it runs from OnActivate and from the config update
listener. Writing straight into agents_confighistory reaches neither. The role's
only reload lever was `notify: container recreate`, which never fired -- the
container is unchanged with RestartCount 0 while the config row was written
inside it. The plugin is now cycled explicitly through mmctl, and the enable step
moves into tasks/utils/agents/enable.yml so the MCP path and the AI path share
one definition.
The closing gate is deliberately not another read of what we wrote: it polls the
bots-to-users join until the agent account exists. That table is written by the
plugin and never by this role, so it cannot pass by echoing the deploy's own
work. The reload fires on stale config OR a missing bot, which covers the state
this stack is actually in, where the config matched but no account existed.
Verified: the stale row, the absent bot account and the always-skipping merge are
all measured on the live stack; the EnsureBots path was read from upstream v2.5.1
source rather than inferred; every changed task file parses.
Not verified: none of this has executed yet. Specifically open are whether
`MATTERMOST_LITELLM_SERVICE | to_json` renders nested Jinja inside a dict var,
whether the converged/stale SQL is valid against live Postgres, and whether a
disable/enable cycle triggers EnsureBots on this version.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The plugin caches mcp.enablePluginServer in process and refreshes it only
on activation or on a plugin cluster event, which Team Edition drops
silently. Toggling the plugin over mmctl reached one container, so the
other replicas kept serving 404 on the MCP route behind the round-robin
VIP. Patching the config row now notifies container recreate, which the
repository already defines for in-database state a process must reload.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every MCP role keeps its task tree under tasks/utils/ without numeric
prefixes, so a caller can reach any of them through include_role
tasks_from. utils/roles/mapping.py holds that path as the single source.
Ansible resolves a relative include against the including file's own
directory when that file was reached by path, but against tasks/ when it
was reached through tasks_from. The same file therefore loaded from its
own role and failed from a caller using tasks_from. Every nested include
now names {{ role_path }}/tasks/utils/..., which holds under both bases,
and a lint walks each tasks_from entrypoint transitively to keep it so.
The exec-gating lint learns the same form, otherwise it loses the include
chain and reports inherited IS_STACK_HOST gates as missing.
The expression resolving a provider's stored token appeared eighteen
times. plugins/lookup/mcp_credential.py replaces it, and mcp_probe_owner
disappears from eight roles: it was only ever a step towards the bearer
and no probe read it.
The Hermes and OpenClaw probes stop aborting the play while it is still
running. Roles deploy in dependency order, so a client probing during its
own role can run before a provider exists at all, and failing there
prevents that provider from ever deploying. The verdict moves to the
reconciliation pass, gated on MCP_RECONCILE_STRICT, which the reconciler
sets and group_vars declares so the readers need no default.
The sys-svc-mcp-reconcile role is added here; wiring it into a stage is a
separate change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
