The gateway could reach exactly one remote provider, through a bare
OPENROUTER_API_KEY group_var with an empty-string default and an `{% if %}`
around a single hardcoded model entry. OpenAI and Anthropic had no path at all.
All three are now declared in roles/svc-ai-litellm/meta/secrets.yml, which
mapping.py already calls the application config schema, so a key lands at
applications.svc-ai-litellm.secrets.credentials.<provider>_api_key like every
other credential. Each carries the `type` and `regex` its value must satisfy and
defaults to the central API.<provider>.api_key block, which keeps one
operator-facing place to fill a provider in while the role owns the shape.
type and regex had to become real to be worth declaring. 192 `validation:`
blocks sit in the role schemas today and not one line of Python enforces them,
so a `regex` would have joined a pile of decoration. validate_supplied_value
now runs on every path a value arrives from outside — the --set override, a
value already in the inventory, and the value standing next to a `default:`.
A generated value stays exempt because the algorithm that produced it is named
by the same schema and generate_value takes no schema argument. min_length is
deliberately left unenforced: the generators ignore it, so holding them to it
would fail where nothing is wrong.
The pattern is written for the paste, not for the vendor. OpenAI moved issuance
from sk- to sk-proj- without notice and Anthropic issues sk-ant-api03-; a schema
pinning a prefix would reject a valid key and abort inventory creation. `\S{20,}`
accepts every form those three issue today and one nobody has announced, and
rejects a trailing newline, an embedded space, a truncated copy and a
<placeholder>. A key pasted into the wrong provider's slot is caught at deploy
instead, where the route it publishes answers nothing.
The gateway iterates one LITELLM_REMOTE_BACKENDS list instead of three `{% if %}`
blocks; config.yaml.j2 and env.j2 read the same list and selectattr('key')
decides. LITELLM_CHAT_MODEL falls back to the first configured remote alias
rather than to a hardcoded openrouter/auto.
For CI the keys travel as INFINITO_API_{OPENAI,ANTHROPIC,OPENROUTER}_KEY through
default.env, the passthrough allowlist, the infinito service in compose.yml, the
workflow env block and the development inventory. The compose leg is not
optional: the env lint rejected the change without it, because a key the
inventory reads but compose does not forward resolves to empty inside the
container with no error. The workflow reads `secrets.X || vars.X` so a variable
works as asked while a secret wins, which matters because a variable is not
masked and the development inventory sets MASK_CREDENTIALS_IN_LOGS: false.
An unset key is not an error anywhere: the provider's routes stay unpublished
and the CLI test asserts their absence rather than skipping.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role merges the gateway service and bot into the stored Agents config and
then cycles the plugin so it adopts them, but agents/reload.yml and
agents/enable.yml both run mmctl in one container (run_once, a single cid).
mattermost-team-edition has no cluster bus, so no other replica hears of it.
Job 102705652088 of run 34421944328 (swarm, three replicas) shows exactly
that in the service log:
02:43:51-52 "Plugin activated" mattermost-ai on .1, .2 and .3
02:44:45 Point the Agents bot at the platform gateway (the merge)
02:44:48 Deactivate the Agents plugin so it drops its cached config
02:44:51 "Plugin activated" mattermost-ai on .1 only
Every post the plugin handled in the test's direct-message channel then
landed on .2 or .3, which still ran the config from 02:43 ("Notification
modified by plugin." at 03:10:30 and 03:25:32 on .2, at 03:35:42, 03:50:42
and 03:52:55 on .3), and the bot never answered:
Timeout 900000ms exceeded while waiting on the predicate ... Received: 0
After the reload the role now forces a rolling update of the service, so
every replica restarts and activates the plugin against the merged config,
and wait_converged.yml blocks until the update has completed and every task
is running again, before the stored config is re-read and asserted. Compose
runs one container, which the reload already covers, so both steps are swarm
only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
svc-ai-litellm has two variants: 0 with both backends on, 1 with both off. The planner assigns a dependency round_index when it has that many variants and falls back to 0 otherwise (inventory/planner.py:111), so round 1 is the one round in which the gateway deliberately serves nothing - a topology that is green in its own job.
Commit e0048f157b gated 14 consumers on LITELLM_CHAT_MODEL_SERVED so a consumer switches itself off when no model is served. Exactly two of them pinned litellm on in round 1, and a variant's literal beats the base Jinja gate, so mattermost rendered LITELLM_SERVICE_ENABLED=true against an empty model_list. The Agents plugin then failed client-side with 'bifrost error: model is required' 47ms after the prompt, never reaching the gateway.
Moving both pins to variant 2 lands them where the fallback supplies svc-ai-litellm=v0 with svc-ai-ollama=v0. Measured with the planner: round 2 pulls both. Both polarities stay pinned, so the variant-coverage lint still holds. The contract assert now also refuses an empty model, which the stored-vs-intended comparison read as converged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The four SQL bodies behind the Agents integration were Jinja templates read
through lookup('template'), which meant the raw query: parameter and a
"# nocheck: database-query-raw" on all five call sites in
tasks/utils/ai_gateway.yml.
They now use the mechanism the database_query module already offers: static
.sql files under files/sql/, reached by query_file:, with the values supplied
as named_args and %(name)s placeholders. All five suppressions are gone.
Same move as 4e9d4b53db made for odoo, for the reason that commit gives: an
interpolation straight into SQL carries an injection surface that a bound
parameter does not. Postgres dollar-quoting was doing that job here; the
module's own escaping does it now, doubling single quotes per the _ENGINES
postgres entry.
The | to_json filters did not disappear, they moved to the call site.
MATTERMOST_LITELLM_SERVICE and MATTERMOST_LITELLM_BOT are dicts, and the
module raises TypeError on those, so pre-serialising is what it asks for.
set_ai/ became update/ai/ to match the select/ and update/ verbs the role's
existing files/sql already uses.
The api key inside MATTERMOST_LITELLM_SERVICE is now a named_args value, which
the module adds to no_log_values; as part of the query string it was not.
Verified statically only: the database-query-raw lint passes, the task file
still parses to the same 11 tasks in the same order, every placeholder has a
matching named_args entry, and no reference to the old paths survives. Nothing
ran against a database, and the adversarial review of this change had not
finished when it was committed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
set-ai-agent.sql.j2 and set-ai-service.sql.j2 shared the delimiter-bounded
prefix `set-ai`, which the repository lint treats as a folder in disguise.
The same rule was already applied in 8e8dbbd08c.
Move both into templates/sql/set_ai/ and drop the prefix, updating the two
lookups in ai_gateway.yml.
The violation was introduced by 43f6ff62e, which added the second file: the
gate ran before that commit series but not between its commits.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Agents 2.5.1 migrates config->'bots' into its own agents_useragents table on
startup and logs "Migrated legacy config bots to self-service agents table".
The wiring still treated the legacy location as the source of truth, which
broke in three ways.
set-ai-service.sql.j2 re-inserted the bot into config->'bots' unconditionally,
so both stores held a bot named `agent`. EnsureBots aborts on that with
`duplicate bot name: agent`, leaves its in-memory bot list empty, and
handleMessages then returns without logging: no reply, no gateway request,
no error. Only write the legacy entry while no live agent row owns the name.
set-ai-agent.sql.j2 retargets the migrated row, because once
legacy_config_bots_migrated is set, writing config->'bots' is a permanent
no-op and the role could never change its model or service again.
select/ai/service.sql.j2 asserts against agents_useragents and additionally
reports stale when a legacy entry collides with a live agent row; the old
check reported converged over a plugin whose bot list had failed to load.
ai_gateway.yml fixes two orderings: the retarget now runs before the plugin
cycle, because the plugin reads its agents into memory at startup, and the
cycle is unconditional, because a converged-looking database was preventing
the very reload that repairs a drifted plugin.
Verified live: legacy bots 0, no duplicate bot name, the bot posting real
replies, and test-litellm-agent.js passing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_prefix_groups_belong_in_subfolders flagged three groups added by this change
set: a shared delimiter-bounded prefix is a folder in disguise, so the group moves
into a folder named after the prefix and the prefix leaves the file names.
sql/select/ai-{bot-user,service}.sql.j2 -> sql/select/ai/{bot-user,service}.sql.j2
files/provision_{ai,mcp}.py -> files/provision/{ai,mcp}.py
files/ai_engine_{configure,endpoint}.php -> files/ai_engine/{configure,endpoint}.php
All six are recorded as renames, so history follows them. Every reference moves
with them: four templated query paths in mattermost's ai_gateway.yml, the two
lookup('file', ...) calls in n8n, the two in wordpress, and SCRIPT_PATH plus the
module docstring in the n8n unit test. lookup('file', ...) resolves relative to
the role's files/ directory, so the new argument is provision/mcp.py rather than
a repo-relative path.
Worth recording why this survived every gate: make quality was green on each run
this session, including immediately before the commits that introduced these
files. The lint enumerates through git ls-files, and the files were untracked, so
it could not see them. Committing them is what made them visible. A green gate
taken before new files are tracked says nothing about the gate after.
Verified: repository lints go from 88 passed with 1 failure to 89 passed, the
unit suite passes at 4448, and a repo-wide grep for every old path returns
nothing.
Not verified: no deploy has run. The lookup('file', ...) arguments and the
templated SQL paths resolve at play time, so a wrong path would surface as a task
failure rather than a test failure.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The previous contract could not fail. Its guard and its closing assertion both
keyed on apiURL -- the one field the merge itself writes -- so once any run had
set the URL, every later run skipped the merge on a false_condition and still
reported all assertions passed. The live row shows what that hides: an apiKey of
`sk-` followed by 48 literal `a` characters, which the gateway answers 401 to,
and defaultModel `llama3:latest` while the round's gateway serves only
`smollm2:135m`. The merge task is `skipping` in every deploy log on this host.
The read now compares the whole desired service and bot with jsonb equality and
answers `converged` or `stale`, so a drifted key or model is drift the guard can
see. jsonb equality is key-order independent, which keeps the check from
depending on how Jinja happened to serialise the dict. Both SQL templates render
from the same two role vars, so read and write cannot disagree about what
correct means. The `jsonb_typeof(...) = 'array'` guards stay: before the first
write those keys exist as JSON null, coalesce does not fire on JSON null, and
jsonb_array_elements then raises "cannot extract elements from a scalar".
A row edit cannot create the bot account, which is why the Playwright spec 404s
on user `agent`. In mattermost-plugin-agents v2.5.1 the config lives in an
in-memory atomic pointer; bots.EnsureBots() is the only path that calls
pluginAPI.Bot.Create, and it runs from OnActivate and from the config update
listener. Writing straight into agents_confighistory reaches neither. The role's
only reload lever was `notify: container recreate`, which never fired -- the
container is unchanged with RestartCount 0 while the config row was written
inside it. The plugin is now cycled explicitly through mmctl, and the enable step
moves into tasks/utils/agents/enable.yml so the MCP path and the AI path share
one definition.
The closing gate is deliberately not another read of what we wrote: it polls the
bots-to-users join until the agent account exists. That table is written by the
plugin and never by this role, so it cannot pass by echoing the deploy's own
work. The reload fires on stale config OR a missing bot, which covers the state
this stack is actually in, where the config matched but no account existed.
Verified: the stale row, the absent bot account and the always-skipping merge are
all measured on the live stack; the EnsureBots path was read from upstream v2.5.1
source rather than inferred; every changed task file parses.
Not verified: none of this has executed yet. Specifically open are whether
`MATTERMOST_LITELLM_SERVICE | to_json` renders nested Jinja inside a dict var,
whether the converged/stale SQL is valid against live Postgres, and whether a
disable/enable cycle triggers EnsureBots on this version.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5