The swarm wait added for the search precheck reads the newest task of
the init service through job_state.sh, which takes the first line of
`container service ps`. That holds for a one-shot job with one task,
the contract the akaunting setup and keycloak import jobs meet by
creating their job with `container service create`.
Zammad's init is a stack service, so it took the topology default of
compose_replicas: one task per host in the web-app-zammad group. Only
the primary placement keeps the CI swarm at one host; an inventory
that puts several hosts into the group would start one zammad-init per
host, each migrating the same database, and the wait would pass on the
first task to complete. init now pins replicas: 1, like the scheduler.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`docker stack deploy` ignores `depends_on`, so in swarm `init` started
even when the `precheck` service exited 1. The stack then looked exactly
as it did before #982: `init` hung, and `stack_ready.sh` reported "not
converged" until all 600 retries of the convergence handler ran out.
The single failed `precheck` task never turned fatal, because
`has_fatal_task` needs three errors.
`init` now runs `search_allocation_precheck.sh` in front of
`zammad-init` as well. In compose the `precheck` service still stops
`compose up` before `init` starts. In swarm, `init` keeps
`restart: on-failure`, so a refusing Elasticsearch makes its tasks fail
again and again. After three errors past `swarm_stack_fatal_grace`
(180 s), `has_fatal_task` marks the service stuck and the deploy stops
with the precheck output in the task report.
Reported by CodeRabbit on #982.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On a data disk with less free space than the high disk watermark asks
for, Elasticsearch creates indices but assigns none of their primary
shards. `zammad-init` then reports every search index update as failed
and never exits, so `compose up` waits on
`service_completed_successfully` until its 1800 s task timeout and the
deploy fails without naming the cause (#850). Elasticsearch's default
high watermark is an absolute 150 GB headroom on a large disk.
A new one-shot `precheck` service runs `search_allocation_precheck.sh`
with `restart: "no"`, and `init` depends on it with
`service_completed_successfully`:
- it waits a bounded time for `_cluster/health`;
- it creates a one-shard, zero-replica probe index with
`wait_for_active_shards=1`;
- once Elasticsearch allocates it, it deletes the probe and exits 0;
- otherwise it prints the creation answer and the answer of
`_cluster/allocation/explain` and exits 1.
The check sits in its own service because `init` keeps
`restart: on-failure`: wrapped into `init`, the failing check was
restarted seven times within eight minutes on a local deploy while
`compose up` kept waiting. Inside the stack, a `restart: "no"`
dependency that exits 1 made `compose up` fail after one second with
`service "pre" didn't complete successfully: exit 1`.
Every curl call carries --connect-timeout and --max-time, so a hanging
request cannot outlast the bounded wait. The image's entrypoint passes
any non-`zammad-*` command through `exec "$@"` (bin/docker-entrypoint of
7.1.2), and the image ships `curl`. The watermark values themselves come
with feature/store-i18n (b82591ba9d); this change covers inventories
without a variant cap.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The bundled Elasticsearch ran with its default watermarks. On a large
disk the high watermark's max_headroom demands 150 GB free, so a 1.8 TB
development host with about 56 to 112 GB free assigned no shard,
zammad-init never finished and the compose-up handler timed out after
1800 seconds (#850).
meta/services.yml declares the watermarks on the search service under
search.watermark: 85 / 90 / 95 % of the disk with max_headroom caps of
200 / 150 / 100 GB, which are Elasticsearch's own defaults, so a
production inventory keeps today's behaviour. Every entry in
meta/variants.yml lowers only the caps to 10 / 5 / 2 GB. Variants reach
development and CI inventories, and a production inventory only when
inventory provision is called with --app-variants;
utils/cache/applications.py
deep-merges them onto the base config, so low, high and flood_stage stay
on the base values. compose.yml.j2 passes all six settings to the
container through ZAMMAD_SEARCH_WATERMARK.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Docker updater watches image plus version, the repository updater
repository plus ref. A dozen pins fell outside both - an app_version, a
release, a tag in a registry neither speaks - so nothing ever told anyone
they had gone stale.
Such a pin now names its upstream next to itself in an `update:` block:
git_tags, registry_tags, npm, http_regex or a role-local script, with
`match` selecting a tag flavour and `strip` removing a prefix the pin does
not carry. utils/update/source.py resolves them beside the two existing
updaters, `python -m cli.contributing.update.source` bumps them, the
update-version-sources job opens one PR per role, and
tests/external/update/source warns while a bump is pending.
Two lints hold the rule: every semver pin is watched by one of the three
updaters or opted out with a reason, and a semver only ever sits behind
`version`, `<what>_version` or `ref` - which renames moodle's release to
app_version and simpleicons' local_version to local_tag, the latter being
the tag of a locally built image rather than an upstream version. A
declaration naming a key the entity does not carry, or a type the resolver
does not know, fails the lint instead of resolving nothing in silence.
Picked from feature/store-i18n (94d607fd0a) onto current main. Two things
differ from the original: svc-ai-lmstudio no longer carries an app_version
pin, so its declaration is dropped, and web-app-stalwart's webui_version,
added since, declares the git tags of the webui repository.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Fourteen roles repeated the same two lines to say they use the gateway: a
`shared` flag gated on group_names, and an `enabled` flag conjoining that gate
with LITELLM_CHAT_MODEL_SERVED. Both are now single-sourced.
`shared` becomes literal true. The flag means the provider is a shared instance
rather than one deployed per consumer, and a central gateway is always that;
the dynamic form only said "shared if deployed", which `enabled` already
decides. The 50 shared pins under `litellm:` leave meta/variants.yml with it.
The tree had already noticed: 20 of those 50 lines carried
`# nocheck: variants-static-override`, two of them with copy-pasted text about
container_backup. This narrows e5de4e085a for one service key rather than
reversing it; every other dynamic flag keeps the group_names form.
`enabled` becomes `{{ LITELLM_USABLE }}`, defined in group_vars/all/16_ai.yml.
That shorthand is only possible because is_explicit_truth learns about it.
The resolver reads a dependency out of the flag TEXT — "in group_names" in
value — so a constant hides the role id and the provider silently leaves the
static closure of every role referencing it. Measured: the old expression
resolves True, `{{ LITELLM_USABLE }}` resolves False. GROUP_GATED_CONSTANTS
names the constants whose own definition carries the gate, and the docstring
states the obligation: a SPOT of this kind that is not listed there drops its
provider without a word.
The wider fix, accepting any Jinja as dynamic, was measured and rejected: 35
other flags carry a Jinja value without the marker (MCP sidecars, javascript
and SSO gates, recaptcha), and they would all enter closures they are not in
today.
A literal shared flag is no longer dynamic, so test_dynamic_flags wants the
`# nocheck: dynamic-flag` its own message documents for legitimate exceptions;
where a marker already existed the rule list is extended rather than a second
comment added.
Behaviour is unchanged where it counts. The closure keeps litellm with the same
app counts (wordpress 26, openwebui 23, hermes 23, mediawiki 22, flowise 23),
unrelated Jinja still resolves False, and the truth table of the new enabled
form matches the old one in all four served/deployed combinations. Three roles
resolve without litellm, and `git show HEAD:` proves that predates this change:
zammad pins enabled false in variant 0 at line 39, nextcloud at line 97.
Six unit tests cover the resolver, including one asserting that every
registered constant is actually defined in group_vars, so an entry cannot
become a dead promise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The root-README overview renders a Video link per role and a reader following it expects a demo, but 26 of the 62 entries addressed a channel, a playlist or a product page: youtube.com/@zammadhq, /channel/UC..., framatube.org/c/mobilizon/videos, pgadmin.org/videos. None of those can be embedded and none shows the product without a second click.
Each replacement was resolved through YouTube's oEmbed endpoint, which returns the real title and channel, so a candidate could be checked rather than assumed. That caught a supposed LibreOffice intro published by an unrelated channel, a Discourse overview belonging to a hosting vendor, and a Zoom video answering 403 with embedding disabled. Fourteen of the new entries come from the vendor itself and two are PeerTube.
The reachability half needed the same endpoint for a different reason: YouTube answers a removed video with HTTP 200 and a Video-unavailable page, so a plain fetch proves only that YouTube is up, while oEmbed answers 400. Measured both ways before relying on it.
The check is external because reachability needs live HTTP, and like its siblings it reports an unanswered probe as unverified rather than failing, so a firewalled runner cannot redden a good URL. Only the shape is asserted, because that is a repository fact.
README.md follows rather than leads: overview.py renders the Video column from this field.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gateway publishes only models it has a backend for and renders model_list: [] when neither ollama nor lmstudio is deployed and no OpenRouter key is set. LITELLM_CHAT_MODEL fell back to openrouter/auto without checking that key, so consumers were configured for a model the gateway does not carry and every prompt failed with a 400.
Observed in the Mattermost#1 CI job: litellm.log carries 12 occurrences of "Invalid model name passed in model=openrouter/auto" next to 400 Bad Request, and tests/test-litellm-agent.js failed on every attempt while the rest of the suite passed.
Make the fallback conditional on the key so the name is empty when nothing is served, promote LITELLM_CHAT_MODEL_SERVED from the role vars to group_vars where consumers can see it (it was the third copy of this logic and the only correct one), and switch the 14 consumers from "litellm is deployed" to "litellm serves a model". The Playwright service flag reads the same lookup, so the spec skips together with the Ansible gate. The bool filter carries weight: the folded scalar yields the string False, which Jinja treats as truthy.
A new unit test renders both expressions against Jinja and pins the invariant that a model name exists exactly when a model is served, including the case where ollama preloads embedding models only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Line-length reflow in three test modules and a stray blank line in zammad variants. In router.yml the formatter also moved the nsswitch lineinfile regex from single to double quotes with the backslashes doubled to match; single-quoted YAML already passes a backslash through literally, so the string reaching lineinfile is byte-identical and the rule still changes only NOTFOUND.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
supported_version and minimum_version sat in 24 roles and were read by no template, no task and no lookup. Their only consumer was the audit report version column, which turned false the moment an automated image bump ran: qdrant reported a verified version nobody had verified, two releases long. Both leave MCP_KEYS, so a role that declares them again fails the schema lint instead of carrying a field with no effect.
Requirement 025 mandated minimum_version as a MUST and 035 carried it in an example; both are corrected here, otherwise the repository would contradict its own code.
The gate then surfaced five dead docs.mattermost.com links. Mattermost moved to /administration-guide/ and /deployment-guide/, and every replacement was probed for 200 rather than guessed. The MCP page is gone with no successor, so mattermost now points at mcpserver/README.md in the Agents repository, the source rather than a rendered mirror.
Both concerns share three files - mattermost's meta/mcp.yml, requirement 025, and the generated audit.md, which derives from a source of each - so splitting them would commit an artifact that does not match its sources.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An MCP client spec signs in through the client's own OIDC button, which
redirects to Keycloak. Every provider put its MCP pairing in the lean variant,
and that variant pinned sso off - so the round that exists to prove the pairing
was the one round without an identity provider. Run 33574056480 shows what that
costs: openwebui's MCP specs reached _shared.js:131 and died on "openwebui OIDC
sign-in button must be visible on the auth page".
The rule now demands two things of the variant that carries the consumers: the
provider pins sso on, and each admitted consumer, in the variant that round
actually pulls in, does not pin it off. The second half matters because a round
deploys every dependency at its own index and falls back to variant 0 only when
the dependency has fewer, so the client shipping alongside is that one and not
the client's variant 0. A role that declares no sso, or pins it to a literal
false because upstream speaks no OIDC, is exempt from a flag it cannot have.
Twenty roles moved to satisfy it. Ten already had a second sso: false pin and
only needed the flip. Six could host the pairing in variant 0, which already
has sso on, and dropped back to two variants. Six kept a third lean variant to
carry the false polarity variants-coverage wants. moodle and mattermost took
swaps instead: moodle's variant 1 is the mandated LDAP-only branch, and
mattermost was already at the three-variant cap with sso on in its variant 2.
nextcloud also turns context_agent on in that round, the ExApp that serves the
endpoint - with it off the round proved nothing it was built to prove.
The rule moves to tests/lint/ansible/services/ because it reads meta YAML and
executes nothing, which is where integration.md says such a check belongs, and
its suppression key joins the catalog it was never listed in.
nextcloud_exapp_reregister becomes nextcloud_exapp_renew: mcp-secret-masking
matches `register:` unanchored, so the substring in the old name read as a task
that prints a secret. register.yml already masks the two tasks that carry one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mcp-consumer-variant guard demanded three things of a provider's MCP
round: every admitted consumer enabled, each of them pinned `services: null`
so it dragged no dependencies along, and no other service left enabled. Two
of those were the rule legislating a round's contents rather than checking
that the pairing gets proven at all.
Keep the first and drop the rest. What else the round carries is the role's
own call, so the tor/litellm exemption, the datastore exemption and the
`allowed` set they fed have no reader left and go with them.
Strip the 63 `services: null` pins the rule forced across 20 variants, all
written bare as `services:`. The topic-emptying capability behind them
stays: apply_topic still normalises None to {} for the inventory payload,
and the unit tests still cover it. Only the uses and the mandate are gone.
That leaves overridden_providers scanning a repository with nothing left to
find, so its three assertions passed over an empty dict and would have
stayed green however the function broke. Rebuild them on a synthetic role
tree, which also covers the case the repository scan never held: a variant
naming a plain flag is not an override.
Removing the pins changes what deploys. Measured across all 20 variants: 18
identical, prometheus#1 grows 23 to 31 roles, moodle#2 grows 31 to 32 and
crosses the memory budget until the following commit shrinks it back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A round deploys a provider once, so the role that pulls it in is the only place that can say how lean it has to be. A variant entry may now carry any application config topic under a service it depends on: a mapping deep-merges, so naming one key changes that key alone, and null empties the topic. Empty was the case that needed a marker of its own, because merging an empty mapping is a no-op and would leave exactly the services the round was trying to shed.
The topic vocabulary and the merge semantics live in utils.roles.applications.topics, which is also where resources.py now gets them rather than spelling the same null-or-merge out a second time. Collecting who demands what stays in the inventory tree, refuses two roles demanding different values for one provider in the same round, and skips a role's own entity entry so the existing services.<own>.users declarations are not read as overrides.
The CI set-cover learns the same fact. Its greedy cover retires a role's variant-0 row when an earlier row of another role embeds it, on the stated premise that embedding pulls the provider's variant 0. A variant that dictates that provider's config breaks the premise from the other side: whatever ran, it was not the provider as the provider declares it. Such a pair now covers nothing, so the retired row keeps its job, its artefact and its verdict.
Measured on the current tree: 20 override pairs, 41 covered rows before and after, so nothing is revoked today. The rule bites the moment an override lands in a row that covers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The contract check ran as an Ansible task in the middle of every provider deploy, and the two client checks ran the same way. A test that is part of the deploy cannot be run without deploying, cannot be re-run on its own, and reports its verdict as a deploy failure. Twenty-seven task files lose 491 lines to this.
It is now what it always was. Each provider ships files/test/test.sh, templates/test.env.j2 and meta/tests.yml, and test-e2e-cli stages one shared harness beside it: run.sh drives contract.sh in a throwaway container on the provider's own network, then isolation.sh for adapter-fronted providers. The read probe each provider offers moved from the calling task into meta/mcp.yml next to the allowlist it must satisfy, and the schema now rejects a read probe naming a tool outside that allowlist.
The bearer the test presents is the one a client resolves through mcp.credential, via the new mcp_client_bearer lookup over resolve_credential, which moved to utils.roles.applications.mcp so it is the same function mcp_servers renders every client connection with. The probe presented whatever the calling role passed, which for the fifteen adapter-fronted providers is a different secret than any client holds.
Sidecar isolation is read from the engine rather than measured inside the container, so it no longer needs a delegation to the node holding the task: attachment is compared by network name, which drops the gwbridge special case, and a swarm worker says it cannot measure instead of skipping silently.
Retrying is bounded by wall clock against MCP_DEADLINE_SECONDS, which scales through the same lookup('timeout') as the harness budget it has to stay under. A flag that is neither true nor false aborts rather than reading as off.
mcp-provider-probe becomes mcp-provider-cli-test: an enabled provider must ship the test rather than include the task. mcp-probe-bearer becomes mcp-adapter-bearer, tying ADAPTER_BEARER to the declared credential key now that no probe include carries it. A new integration test renders the shared env template for every provider through Ansible's own templar, so a later change to it breaks visibly instead of poisoning twenty deploys quietly.
Hermes and OpenClaw keep their client-side proof as CLI tests of their own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Thirteen of sixteen MCP sidecar entries carried no enabled at all, so
collect_role_resources fell back to _looks_like_container and billed every
one of them as running. Three roles already gated theirs on mcp.enabled;
the rest now do the same, gitea twice because it is the only role with a
vendor container behind the adapter.
The fifteen adapter roles then read services.<key>.enabled instead of
mcp.enabled, so the compose block and the task gates follow the services
entry rather than running beside it on a second path. The six plugin and
native roles keep reading the topic: they ship no sidecar entry to point at.
The three-level resolution is not new. KEYCLOAK_LOGOUT_PANEL_ENABLED has
the same shape and depth and gates a task the same way, and the same value
reaches JAVASCRIPT_SERVICE_ENABLED, where service-gating.js throws on
anything but "true" or "false" — an unresolved chain would have been
killing keycloak's Playwright run since 575a5a87ac.
This does not change what the resource model counts: _is_enabled reads the
unrendered Jinja and still treats it as on. What it buys is one switch
instead of two, and a declaration that matches it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
83c1c729de gave every provider a group_names-gated services entry per MCP
client, including the clients it refuses with mcp_consumer: false. The gate
is what makes the inventory resolver plan provider and client into one round,
so 19 of the repo's 82 co-deploy pairs started a client for a pairing the
provider had already declined. flowise is a live MCP client of qdrant,
homeassistant and prometheus; the other 17 providers were bringing up its
node stack for nothing.
The entries now read enabled: false, shared: false, mcp_consumer: false, and
the variants that pinned them to true are gone. Setting those variants to
false instead is not an option: test_variants_no_static_override forbids any
variant override of a literal base value in either direction.
bond leaves with them, and a new lint holds that line: collect_role_resources
skips a literally disabled entry before it reads any resource key, so a bond
there describes a cost nobody pays. Thirteen entries unrelated to MCP carried
one and are cleared too.
test_mcp_consumer_services_entry demanded the group_names shape for every
client entry, refused ones included, and now demands the opposite there.
dynamic-flag keeps its rule and takes its own documented per-key exemption.
make docs redraws twenty graphs: the edges go, the nodes stay as
'openwebui ❌' and 'flowise ❌', so the refusal is visible rather than read as
an integration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
enabled and shared were hand-maintained in 24 roles and wrong in both
directions. Both now resolve from the mcp_consumer edges already declared
in meta/services.yml: a server takes the consumers it admits, a client
takes the servers that admit it, and the pair renders as "'peer' in
group_names or ...", or False when no peer is admitted at all. setdefault
keeps meta/mcp.yml and meta/variants.yml as the override channel, so
pinning either key still wins.
The resolved block replaces the mapping instead of being written into it.
_build_role_base_config parks the YAML cache's own object under mcp, so an
in-place setdefault injected the derived keys into every later raw read and
made test_mcp_schema pass alone but fail in a directory run.
The derivation keys on peer presence, never on a peer's own value: a
re-entrant read under _RENDER_GUARD returns the unrendered dict.
Four lints now ask the resolver instead of reading meta/mcp.yml, which they
could not do while the key was hand-written. That is what exposed the eleven
missing hostile-path specs the previous commit closed.
Carries one change that belongs to the WordPress probe but shares a file
with this one: the tool allowlist moves to the hyphenated names the adapter
actually serves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
allowed_consumers was twenty hand-kept lists of which fourteen were identical, so every new client meant an edit in every provider. A client now declares services.<own-entity>.mcp_consumer: true once and a provider refuses one with mcp_consumer: false, which is also what tells a deliberate exclusion apart from a forgotten entry.
The group_names-gated consumer entries give the inventory resolver the co-deploy edge meta/mcp.yml never had: one of sixty-nine matrix rounds carried an MCP client before, twenty-three role/round pairs do now. bond 0.5 keeps the clients out of round 0 so the host memory budget still holds.
BREAKING CHANGE: meta/mcp.yml no longer accepts allowed_consumers; declare services.<entity>.mcp_consumer on the client instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/volumes.yml separates type: secret from type: config, and the
distinction is what a reader relies on: a config volume is the one you
may show someone or mount read-only for every process. Four roles
rendered a live client secret or an account password into one anyway.
The mode makes it concrete rather than theoretical. 9e7f28db20 widened
these to 0444 for a real reason - swarm config objects mount root:root
and 0440 crashloops a non-root app user - so every process in the
container can read what the type says is safe to show.
The vocabulary comes from how this repository actually writes secrets,
not from a word list. All four are suppressed with the upstream reason
they cannot be split: Jenkins CasC reads one file, Zammad's wizard takes
no reference. They are recorded rather than fixed, and a fifth role
cannot join them silently.
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Zammad 6.5.0 has no AI surface to configure: neither `ai_provider` nor
`ai_provider_config` exists in that line, so the role could not point Smart
Assist anywhere. 7.1.2 is the newest version whose contract is still that
setting pair; from 7.2 the provider moves into an `AI::ProviderConnection`
record and the shape changes, so the bump lands here rather than on latest.
Elasticsearch is unaffected: 7.1.2 accepts >= 7.8, < 10 and the role's 8.13.4
pin stays.
The provider is written by piping files/scripts/apply/ai_provider.rb into
`rails r -` over stdin, with url, model and token passed as environment
variables instead of interpolated into Ruby source, so no quoting path can
mangle a virtual key. Order inside the script is load-bearing and documented
there: `ai_provider` is the boolean "a provider is configured" and its
validator reads `ai_provider_config`, so the config hash has to land first or
the flag is refused. `UserInfo.current_user_id` is set because `Setting.set`
needs an acting user.
This is the strongest evidence in the requirement, because it is not a
read-back of something the deploy just wrote: `Setting.set("ai_provider_config")`
runs `AI::Provider::CustomOpenAI.ping!` upstream, so the write itself is a live
round trip. An unreachable gateway or a rejected key fails the task.
Two consequences of the bump ride along, because the version does not work
without them.
`/api/v1/getting_started` answers 403 from Zammad 7.0 on, so the Playwright
helper could no longer lift a csrf token there; it uses `/api/v1/signshow`,
the unauthenticated seed. Its old fallback read `csrf_token` from a JSON body
that endpoint never returns, so the failure surfaced as a confusing throw
rather than the 403 it was.
env.j2 drops the hand-built DATABASE_URL in favour of POSTGRESQL_OPTIONS
`?pool=`. Zammad composes its DSN from the POSTGRESQL_* parts itself, so the
explicit URL both duplicated and competed with them while leaving the
connection pool at its default.
The spec decodes ZAMMAD_LITELLM_BASE_URL and ZAMMAD_LITELLM_MODEL through
decodeDotenvQuotedValue. Without it the values arrive still wrapped in the
literal quotes dotenv_quote adds and the strict comparison can never match in
compose mode -- measured as expected `"\"http://litellm:4000/v1\""` against
received `http://litellm:4000/v1`.
Verified: the spec passes against the live stack (1 passed, via
scripts/tests/e2e/rerun-spec.sh --grep=Assist, which restages the spec from the
repo), and both the contract and the Smart Assist write ran ok in a variant-2
deploy. `make quality` green on test-lint, test-unit, test-integration,
test-external.
Not covered: the no-gateway path (ZAMMAD_LITELLM_ENABLED false) has not run
since the bump, and Zammad features this deploy does not exercise -- email
channel, LDAP sync, ticket workflows -- are unproven on 7.1.2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mount pipeline was split across two layers: compose_volumes and
container_volumes lived as filter modules that ansible's filter loader
rejected on every start ("has no attribute 'FilterModule'", two warnings
per run), while the same-named lookups wrapped them thinly. The filter
bodies now live inside their lookups (the tor_socks precedent), the
filter files are gone, and the render tests moved next to the lookup
tests.
As lookups they see the templating context, so the application_id term
every call site passed back to its own role is gone too: both lookups
read application_id from context and fail loudly when it is missing.
107 templates and nfs_prep.yml drop the term; the calls collapse to
lookup('compose_volumes') and lookup('container_volumes', service_name).
The container_volumes call is now canonical - four leading spaces,
service_name in the service position, | indent(4) - enforced by a new
lint (test_container_volumes_call_shape). Normalizing the 85 literal
call sites surfaced that erpnext keyed its meta/volumes.yml mounts by
container name instead of compose service key, the only role to do so;
its meta now uses the service keys. wordpress's column-0 call, which
rendered its first line at top level, is indented like every other.
Template logic that only restated meta/volumes.yml is deleted: mailu fed
compose_volumes an extra_volumes dict whose eight names were looked up
from the very meta file compose_volumes reads itself; zammad's init and
railsserver extra mounts were the volumes.yml entries minus the two
services, which the meta now lists (the when-gated OIDC binds included).
Conditionality that meta cannot express stays in code, but in role-local
lookup plugins instead of template loops: matrix_bridge_mounts renders
the per-bridge registration mounts, nextcloud_extra_volumes the
feature-gated whiteboard/recording volumes (type: volume entries are
declared unconditionally by compose_volumes, so gating must happen
before it).
Parity is proven, not assumed: every mechanically rewritten template is
byte-identical to HEAD after the transformation regexes (95/95), and the
semantic roles render identically through the real functions against
HEAD's meta - zammad across six sso/tls polarities, all seven erpnext
services, mailu's top level. make test-lint passes complete.
Also: suitecrm's compose template pins container_extra_hosts with
host_alias (the SOCKS proxy URL host.docker.internal never resolved in
compose+tor), and the five disabled mautrix bridge addons use the
empty-default config lookup so templating the applications dict stops
emitting 125 errors per batch in every CI job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
unsafe-inline re-admits the inline execution an XSS needs and unsafe-eval re-admits string-to-code, so a CSP relaxation must be verified against the running app rather than copied from another role. The new lint requires a nocheck marker plus a Reason on the flag's own line, so a marker cannot drift to the neighbouring directive.
The 197 pre-existing grants across 70 roles are checked out with a TODO: they were never verified and still have to be.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zammad: the probe sat on the 'zammad' entity while the compose template applied it to the railsserver service by literal name. Both entities carry name: zammad-railsserver, so they are one container - but 'zammad' declares the app/proxy port 8080 and railsserver declares 3000, which is what Rails binds. The probe therefore addressed a port nothing listens on inside that container. The declaration moves to railsserver and the template looks up service_name, so entry and consumer can no longer drift.
The flavor becomes connect: a bare socket open instead of an HTTP request over /dev/tcp, with a 15m start_period instead of the 30s the tcp flavor inherits. The role's own readiness task budgets five minutes for the same boot, so the old probe declared the container dead while the code beside it was still waiting.
seaweedfs: flavor curl without a path probes the root, which the S3 gateway does not answer. tasks/00_core.yml has queried http://127.0.0.1:8333/status with until rc == 0 since it was written, and api_port equals ports.internal.http, so the probe now asks what the role already knows to be the ready signal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A container without a healthcheck is accepted the moment its process starts. The restore drill asks the host's health service for its verdict, so a service that came back empty passes as long as it is running - which is how an empty volume survives a drill unnoticed.
Twenty-one services get a probe and thirteen get a marker naming why none is possible: a one-shot client that exits, a build image with no service port, a media bridge on UDP. The flavor comes from each image's own layer history, read from the registry through the mirror, so no image had to be pulled to find out whether it ships curl, wget or a shell.
Two entries carry more weight than the rest. The central Postgres, which every replay of the recover chain lands in, had no probe at all; it gets the pg_isready line that sys-svc-rdbms already runs for the per-app engine. And the matrix runner starts systemd within a second while its inner daemon is still pulling every image of the deployment, so it counted as healthy long before anything served - it now waits for Synapse to answer.
No probe replaces one: nineteen of the twenty-one images ship no HEALTHCHECK, and the two that do already carried a declaration before this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
72fd9c870 required the onion in every variant but exempted variant 1, on the
argument that the minimal round is the one place the stack runs without it. That
argument does not hold: the onion is part of the deployment under test, not a
feature bolted onto the full shape. A round that skips it is not a smaller
deployment, it is a different one.
So the exemption is gone and 95 variant-1 entries pin the onion on. Two
exclusions remain, both read from data rather than a comment: a role with no tor
block has no onion to pin, and a role whose meta/services.yml pins tor.enabled to
a literal false has already declared it has none.
That put the rule in conflict with test_coverage, which requires both polarities
of every dynamic flag: with tor on everywhere, the false side has no variant left
to live in. 94 roles take the exemption that contract already provides for
exactly this - "flags whose group-derived value is a hard invariant that variants
must never override", the same one 4e6ef2542 used for the seaweedfs frontend.
web-app-nextcloud needs none: its variant 8 pairs the clearnet-only PeerTube and
keeps the false polarity.
Measured, because the declaration is not what CI runs. assign_tor gives the onion
to every second capable entry with the run number as the parity switch, so more
capable variants do not mean proportionally more onion jobs: across five roles
and 21 entries the count goes 8 -> 10 on an even run and 9 on an odd one. The
clearnet path also survives - checkmk reports v1=false on run 0 and v1=true on
run 1, so both states are still covered across two consecutive runs. What the
94 markers give up is a declarative guarantee in variants.yml, not the coverage
itself.
The real cost is weight, not job count. web-app-nextcloud variant 1 goes from
3.09GB reservation / 7.27GB limit to 18.26GB / 43.01GB once svc-net-tor and its
closure join, so the cheap round is no longer cheap. Every variant stays under
the 32/64GB budget, and the same effect applies to the other 94 unmeasured.
Verified: tests/lint 336 passed / 307 subtests, tests/integration/roles/meta/
variants plus the role-meta lints 39 passed. The rule was checked in both
directions - it reported the variant-1 offenders before the flips and none after.
Not measured: what the heavier rounds do to wall-clock, against swarm jobs capped
at 355 minutes and one that has hit that ceiling before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
7a70d27b7 made a variant that switches SSO on also switch the onion on, because
that pairing is where the platform breaks in ways clearnet never shows. The same
argument reaches further than SSO: a Secure cookie is dropped, an https identity
provider refuses to embed, and crypto.subtle is undefined for any feature on a
non-secure origin, not only for a login. A variant that never runs the onion is
not testing the deployment users get.
So the rule now covers every variant of a role that declares a tor block.
Variant 1 stays exempt - it is the minimal, partner-free round every role
declares and the one place the stack is exercised without the onion at all - and
a role whose meta/services.yml pins tor.enabled to a literal false is skipped
outright, because that declaration already says the role has no onion and needs
no comment repeating it.
That found 137 variants. 101 were variant 1. Of the rest, 28 are the LDAP-only
auth-isolation rounds, which gain real coverage from running LDAP over the onion
and are flipped on here. Four cannot: svc-net-tor is the provider itself, and
nextcloud's variant 8 pairs the clearnet-only PeerTube. Both carry a marker
naming the reason; opentalk and coturn need none, since their services.yml pins
the literal false the rule now reads.
Flipping them exposed a contract that contradicted itself. test_auth_isolation
and test_non_baseline_explicit_disables demanded `shared: false` beside
`enabled: false`, while variants-static-override flagged that same line as a
static override - which is why all 133 of them carried a nocheck marker. The
demand only makes sense where `shared` is dynamic: a literal base value cannot
leak through deep-merge, so restating it protects nothing. Both contracts now
ask for it only in that case, and the 133 lines are gone.
The result is a stricter rule with fewer lines: 189 insertions against 193
deletions.
Verified: tests/lint 336 passed / 307 subtests, tests/integration/roles 107
passed / 610 subtests. The rule is checked in both directions - it reported 137
offenders before the resolution and none after.
Not measured: 28 additional onion rounds cost matrix wall-clock, against swarm
jobs already capped at 355 minutes and one that has hit that ceiling before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every variants.yml header states the rule the bond feeds: variant 0 carries each
tightly-coupled (bond>=1, same-host) service, while loosely-coupled (bond<1,
separate-host) partners are excluded from the closure and distributed across
later variants to keep every host under budget. tor was declared bond: 1 in all
101 consumers, so it had to sit in every variant-0 closure.
It does not belong there. The onion is a separate-host provider reached over the
node's hidden service, the same shape as the bridged partners that already carry
0.5, and f1dff50fb established both the reasoning and the precedent when it
dropped xwiki and bigbluebutton from 1 to 0.5: bridged partners with their own
role and their own spec, whose bond declaration was the outlier rather than
their placement.
This also unblocks the variant-level tor contract. Requiring a non-baseline
variant to pin the onion on while tor still counted as tightly coupled would
have pulled variant 0's closure requirements along with it.
Verified: tests/lint 336 passed / 307 subtests and tests/integration/roles 107
passed / 610 subtests, including test_resource_budget over every variant's
deduplicated footprint and the bond-aware test_order closure.
Not measured, and the honest cost: f1dff50fb's own caveat applies unchanged
here. Whether redistributing a service out of the baseline closure buys back
wall-clock is not established; role-runtimes has the per-role seconds and no run
has been made with this shape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A volume that reaches several services repeated its container path once per
mount. Written out each time it has no single point of truth: a rename that
misses one line still parses, still deploys, and mounts one service somewhere
nobody intended. Nothing fails until that service looks for content which is
not there.
YAML already solves this, and 62722a6ab used it here first -- web-app-openproject
anchors await_migrations_target and aliases it for the second service. Nineteen
roles now follow, and tests/lint/repository/spot/test_volumes_target_single_spot.py
fails on any target spelled out twice. An alias carries no path, so the rule
cannot see it, which is the point.
The conversion was mechanical and is provable rather than reviewed: every file
was rewritten and then reloaded, and any file whose yaml.safe_load result
differed from the original was discarded instead of kept. Nineteen written,
none discarded. Anchors and aliases are a serialisation form; the loader
resolves them before any consumer sees a document.
validate_volumes_meta over all 81 meta/volumes.yml reports no violation.
make test-lint: 328 passed.
web-app-prometheus carries two unrelated deletions: comments that predate this
change and fell under the comment lint once the file was touched. They explained
a 0444 mode and a compose-only bind; both facts are visible in the entries
themselves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The adapter was the only image in the tree pinned by a hand-maintained digest,
and the pin was never recomputed by anything. It drifted twice: run 31374959721
lost seven of eight jobs to mirror/docker.io/python@sha256:6771159c... not
found, and the renewal to the newest upstream digest earlier today would have
reproduced it, because builds resolve FROM against the GHCR mirror while the
mirror is copied by tag and lags upstream by design.
The base is now FROM <image>:<version>, taken from the sidecar service in
meta/services.yml - the only file iter_role_images scans, so the tag the build
asks for is by construction the tag the mirror was told to copy. The version
moves from the moving 3.13-slim to 3.13.15-slim, which upstream stops rewriting
once published, and which the existing version drift job renews like every other
image.
With the build reading services.yml, the adapter block's digest, image and
version had no consumer left. They are removed together with the schema check
for the digest form, the warn-only drift test, and the manifest_digest helper
that existed only for it. What remains in the block describes the adapter rather
than the image it is built from.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reverts the twelve pins to 9662417a, the digest the GHCR mirror currently
carries for python:3.13-slim. The previous commit renewed them to the newest
Docker Hub digest, which is the wrong registry: builds resolve FROM against the
mirror whenever a mirror prefix is configured, and the mirror is copied by tag,
so it holds whatever the tag pointed at when the sync last ran. A pin ahead of
the mirror resolves to nothing there while a local build keeps working - run
31374959721 lost seven of its eight jobs to exactly that, as
mirror/docker.io/python@sha256:6771159c...: not found.
The external check measured Docker Hub and would therefore have kept advising
the change that breaks the build. It now resolves the source the build uses via
mirror_image, asks whether the pin exists there at all, and only falls back to
upstream when no mirror is configured.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three fields pinned artefacts and nothing recomputed any of them.
The adapter base digest was stale for all twelve provider roles: the tag
python:3.13-slim moved to ffb752e1 while every role still pinned 9662417a.
The GHCR mirror copies by tag, so the pinned manifest is one it was never
asked to copy - a mirrored build resolves FROM to nothing while a direct
build keeps working.
adapter.specification_sha256 matched no revision of the file it pins in ten
of twelve roles; three of the old values were byte-identical to the same
role's tools.schema_sha256, so they had been copied rather than computed.
The field is enforced nowhere at runtime, which is exactly why it needs a
check.
Add a hard lint for both offline pins, computing the tool digest with the
adapter's own policy.schema_digest so it cannot drift from the code that
enforces it in production, and a warn-only external check that reports a
base digest the upstream tag no longer resolves to.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nine roles reached their MCP surface as the deployment administrator or with
an account password kept beside the account in credentials. A Moodle site
admin bypasses every capability check, a Nextcloud app password is not
scopable, a Gitea token reads every private repository, and a GitLab token on
root reads every private project. Each role now declares its own principal in
meta/users.yml and provisions it, so the exposed surface is what the service
role permits. jellyfin, zammad and snipe-it kept their account but took its
password from a credential named token; they read the registry too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/schema.yml held credential definitions and their runtime values, but the
name said 'schema' and read like structural metadata next to services.yml and
volumes.yml. Nothing in the file is a schema in the sense the other meta files
use the word, and a contributor had to open it to learn it holds secrets.
62 role files renamed. ROLE_FILE_META_SCHEMA becomes ROLE_FILE_META_SECRETS and
keeps its role as the SPOT the path is built from.
Two readers bypassed that constant and spelled the name themselves:
utils/cache/applications.py built meta_dir / 'schema.yml' twice, and
plugins/lookup/application_gid.py carries the filename in the marker list it
detects application roles by. Left behind, the rename would not have raised —
credentials would have resolved empty and the role would have stopped counting
as an application.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The digest shipped with the digest-pinning change was a valid but
superseded python:3.13-slim index that no tag points at any more. CI
pulls through a mirror that only copies manifests reachable from the
tags it syncs, so that digest was absent there and every adapter build
failed to resolve its base image.
The replacement is the digest the tag carries today, which resolves both
upstream and through the mirror. Removing the digest instead would trip
the schema lint that requires one.
The metadata named infinito/mcp-adapter:1.0.0 with an all-zero digest,
an image that does not exist, while the build actually used
python:3.13-slim. Declaration and artifact disagreed, so the declaration
proved nothing. Both now name the real base image and pin it by digest,
which a tag cannot guarantee across a rebuild.
The adapter base moves from the provider's service version to
mcp.adapter.digest, alongside the tool specification and its checksum.
Being an MCP client was enough to join a provider's overlay, so a single
admission anywhere reached every provider that opened one. Attachment now
requires both that the role speaks as a client and that the provider
lists it in mcp.allowed_consumers.
consumer.kind accepts a list so one overlay can carry both its
service-flag consumers and its MCP clients, which is what the vector
store needs.
A user password reaches its application through Ansible, a shell, a container
exec -e and finally the runtime's own environment reader. The generator seeded
it from the same alphabet as the become password, punctuation included, and
that does not survive the chain intact.
Three roles had already worked around it by declaring a separate alphanumeric
credential for an identity that their meta/users.yml already declares, and
saying so in the description. Removing those without fixing the generator would
have reintroduced exactly the bug they were built against, so the generator is
fixed first and all three now read the user's own password. At length 64 the
smaller alphabet costs nothing worth measuring, and the become password keeps
the old one because it travels a different path.
Zammad additionally used the bot's password for its wizard-bypass admin, one
secret for two users. Each has its own declaration and now its own value.
A test asserts the generated password is alphanumeric, so a future change to
the alphabet fails there rather than inside a Ruby ENV.fetch in a container.
Ten applications with no MCP server of their own get one without a line of code
inside them: qdrant, checkmk, fider, jellyfin, listmonk, pretix, prometheus,
snipe-it, zammad and libretranslate.
The sidecar is confined rather than trusted. It runs read-only, drops every
capability, forbids privilege escalation, publishes no port and joins only its
own provider's network, carrying neither that provider's alias nor the consumer
networks the provider itself joined. Its tool surface is a JSON contract, not
code: which tools exist, which upstream path each maps to and what shape the
arguments take. Adding a tool is a line of JSON, not a new image.
The image is built locally from the audited adapter source with the base image
as a build argument, so all ten share one reviewed build instead of ten upstream
dependencies. Each role provisions its own read-only identity in the language
its application speaks, and every adapter include is guarded by the role's MCP
flag, so switching it off leaves no container behind.
The block sat under services but never behaved like a service. A service has
exactly one providing role, declared with provides, so the registry can map the
key to that role and consumers bind their flag to its presence. No role declares
provides: mcp. There are seventeen providers and the pairing is done by
allowed_consumers inside the block itself.
Every rule that enforces the service model therefore needed an exemption for it:
a shared-provider suppression in fourteen roles, a playwright-service-flag
suppression, and two freshly added exemptions in the dynamic-flag and
variants-static-override checks. Four independent rules, four escape hatches,
one cause. They are all gone now, together with a bond key that only the
resource budgeter reads and only for entries under services, and a wordpress
addon bridge that pointed at a service block that no longer exists.
The block moves to meta/mcp.yml. The applications loader maps meta/<name>.yml to
applications.<app>.<name> generically, so the move needs no wiring; the file
takes the same shape as meta/rbac.yml. Every config path follows, the variant
overrides are lifted out of their services map, and the vocabulary module leaves
the services package because it no longer describes one. roles_with_service
gains an explicit topic parameter rather than an implicit fallback, so nothing
silently resolves against the wrong container.
Files that are otherwise unrelated to this move are included where the committed
tree would otherwise keep pointing at a path that no longer resolves: thirteen
vars files, eleven playwright env templates and one task file.
Three conflicts, none resolved by picking a side wholesale:
- Makefile: took the incoming ACT_INPUTS, which routes through SWARM_DISTROS
instead of inlining the same expression a second time.
- test-e2e-playwright/02_run_one.yml: took the incoming rc handling. An absent
rc now counts as failure; the previous default(0) let an aborted Playwright
run pass silently, contradicting the debug message right above it.
- web-app-peertube spec: combined both. Kept gotoOnion and resolveTimeout,
which this branch's lint enforces, and adopted the incoming retry-inside-poll
for the OIDC button plus its networkidle wait.
The declared values were guesses that ran up to 60x the size of the image they belong to: seaweedfs asked for 30GB against a 104MB image, nextcloud for 20GB against 353MB. Because min_storage feeds the storage guard, the runner packer and the sufficient-storage filter, those numbers made hosts look too small and pruned caches that deploys did not need to lose.
Every service that declares min_storage together with image and version now carries the measured compressed transfer size of that image plus 200MB of headroom. Sizes were read from the registries through utils.docker.registry.manifest_transfer_size, the same path the headroom test uses, so the declarations and the gate agree by construction. Values are written in MB throughout, matching the existing 500MB / 100MB / 50MB style.
119 declarations across 89 roles. Range: 206MB (memcached) to 3473MB (ollama); four exceed 1GB. The 84 min_storage entries that name no image are untouched, since there is nothing to measure them against.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
