#1180 gave web-app-matrix a canonical `jitsi` domain for its own Jitsi
and listed it under `services.jitsi.domains`. When Jitsi is shared or
disabled nothing serves that domain, and the blackbox healthz job of
web-app-prometheus probed every canonical domain, so the probe of
https://jitsi.matrix.<domain>/healthz/ready failed with "tls:
unrecognized name" and `prometheus scrape: every consumer role reports
up=1 and passes its probe` went red (main run 40017166046: Matrix#2,
Nextcloud#12).
web_health_expectations already skipped such keys. The rule now lives
in utils/domains/unserved.py: a key claimed by a service entity that is
disabled, or enabled but shared, is not served by that app. The health
filter uses it unchanged, and prometheus.yml.j2 filters its targets
through the new `served_domains` filter, which also drops the
`<key>_onion` mirror of a skipped key.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
proxy_hide_header is replaced, not merged, by a location that sets its own,
so upstream_strips.conf.j2 only takes effect where a location includes it.
The health locations of web-app-prometheus (/healthz/ready, /_live_probe),
which every standard vhost includes, and the extra locations of
web-app-bluesky (/config, /ipcc) and web-app-opentalk (/v1/, /signaling,
/livekit) proxied without it, so X-Powered-By and the upstream's
Strict-Transport-Security leaked on those paths.
All seven locations now include the strips. A new lint checks every role
template without a server block, i.e. every fragment the front proxy
renders into a vhost, and fails on a proxying location that lacks them.
Templates with their own server block are app-internal nginx configs
behind such a location and stay out of scope.
Closes#1124
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The readiness module http_ready rendered insecure_skip_verify from
TLS_MODE and lookup('ca_injected', application_id) (#851). The lookup is
true only when Prometheus itself has a domain, TLS and the self-signed
mode, so in a self-signed deployment where Prometheus runs in another
mode the exporter had no CA file and skipped verification. All targets
share the module, so a target with a publicly trusted certificate was
probed without verification too, and anyone on the probe's path could
answer for a failed application.
http_ready now always verifies. A second module, http_ready_private_ca,
skips verification, and only the targets that need it select it: those
in the self-signed mode while the CA is not mounted into Prometheus,
through the __param_module label, which overrides the job's module for
that target. The default self-signed deployment mounts the CA and keeps
every target on the verifying module.
Verified on a local self-signed Tor round, where the CA is mounted:
the rendered blackbox.yml holds http_ready with insecure_skip_verify:
false and http_ready_private_ca with true, and no target of the scrape
configuration carries the module label. All targets of that round are
http onion names, so the https path itself ran only in the unit tests:
tests/unit/python/roles/web-app-prometheus pass, among them a publicly
trusted target that keeps the verifying module while the CA is missing,
a self-signed target that switches to the private-CA module, and the
verifying module rendering insecure_skip_verify: false without the CA.
Against the templates of main the three new cases fail.
Closes#851
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Host, X-Real-IP, X-Forwarded-For, X-Forwarded-Proto, X-Forwarded-Port and
X-Forwarded-Ssl were written out separately in the html, ws and upload
locations and in the Prometheus health probes, and the copies had drifted:
ws and upload never sent Port and Ssl, and the probes sent only Host, which
is how the Shopware readiness probe came to see plain http.
headers/forwarded.conf.j2 now holds the set and every one of those locations
includes it. The html location renders the same headers as before. ws and
upload additionally send Port and Ssl with the values the html location
already gives the same backend. The ready and live probes gain X-Real-IP and
X-Forwarded-For, and the live probe the three X-Forwarded headers as well.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The /healthz/ready location proxied to the app with only the Host header.
An app that matches its configured domain including the scheme then sees a
plain http request. Shopware has no sales channel domain for
http://shop.infinito.test and answers 400, so blackbox reported
probe_success 0 and the scrape spec failed in Shopware#0 (swarm) of main run
37903645290: status_code=400 on every probe once Shopware was up, and
shopware_web logged GET / -> 400 for host shop.infinito.test.
The location now sends X-Forwarded-Proto, X-Forwarded-Port and
X-Forwarded-Ssl with the same values the html location gives every page
request to the same backend.
Verified on a local Shopware stack: /healthz/ready answers 200 with the
headers and 400 again once they are removed from the live vhost.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
#832 probes /healthz/ready on every consumer and proxies it to the app root,
where Bridgy Fed, Hermes and the XMPP web listener answer 404 by design, so
their probe_success stayed 0. A consumer can now name the path in
services.prometheus.ready_path; the accepted status codes stay unchanged.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Since the targets take their scheme from the domain they probe, two
switches of the http_ready module no longer fit. Both still followed the
TLS state of the Prometheus vhost.
fail_if_not_ssl refused an http target, an onion vhost for example, as
soon as Prometheus itself was served over TLS. It is false now. The
module does not follow redirects, so the scheme of the target URL already
decides the transport.
insecure_skip_verify switched certificate checks off whenever Prometheus
itself had no TLS, also for an https target. Checks now stay on, except in
a self-signed deployment whose CA is not mounted into the Prometheus
containers, where the exporter has nothing to verify against.
The per-role test of the spec parsed every answer of the query API as
JSON. An answer that is not OK now counts as "no probe yet", so the poll
keeps waiting instead of ending on a parse error.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every blackbox probe of a Tor deployment answered 404, so AppDown fired for
all deployed apps although they were up. The job built its targets from
canonical_domains_map(DOMAIN_PRIMARY), which knows no onion domain. In a Tor
deployment the vhosts answer under <sub>.<onion> only, and the front proxy
served the clearnet names from its default server.
The targets now come from lookup('domains'), the map the vhosts are built
from, and each target takes its scheme from the domain it probes instead of
from the Prometheus vhost.
Which roles are probed follows lookup('roles_with_service', 'prometheus'),
the list the role's own spec expects metrics for. It leaves out the database
roles, which have a domain in the map and no vhost, and the roles that
declare scrape: false because their vhost carries no Prometheus block.
Two roles declared more than their vhost delivers. web-svc-mirror renders a
vhost of its own with no backend behind it, and on a live stack its probe
answered 404 under the onion name as well, so it declares scrape: false now.
web-app-peertube has a backend, but its vhost template never included the
Prometheus locations. It includes them now.
The spec could not see any of this in a compose deployment. dotenv_quote
wraps PROMETHEUS_TARGET_ROLES_JSON in quotes there, JSON.parse returned a
string, the role list came out empty and the per-role test was skipped. The
value is decoded now, and the test also requires probe_success == 1 for
every role in the list.
Closes#792
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Only two files were touched by both sides. env.js merged automatically: the
incoming side owns recordVideoOptions and this branch had cherry-picked that
same function from it, so the result carries one definition. README.md is the
generated roles index and both sides append rows; resolved as a union, taking
the incoming side for the three rows it edits (OnlyOffice, Open WebUI, OpenClaw)
and keeping ours only for OpenBao, then re-sorted.
The incoming side adds lint rules and resources that this role then violated.
Five fixes, each the remedy the failing rule names:
* services.openbao.ports.local.http 8081 -> 8093; svc-ai-litellm now holds 8081.
* meta/networks.yml subnet 192.168.221.0/24 -> 192.168.222.0/24; svc-ai-mcp-adapter
now holds 221, and the suggester only sub-allocates inside occupied /24s, so
the next free block was taken directly.
* services.email bond 1 -> 0; the entry is disabled and the resource model skips
it before reading the weight.
* README Quick Setup regenerated from the template it had drifted from.
* Credits link the implementer, matching how they are credited elsewhere.
* docs/requirements/035-mcp-proxy-expansion.md disposes of web-app-openbao as
privileged_control_plane, which is what a secrets manager is, and the list
count follows.
Verified against a worktree of #454 alone so the baseline is measured rather
than assumed: lint 501 passed with only the branch-name rule failing, which
fails identically on that baseline because the worktree branch is named
merge/*; unit 5665 passed with the same three host-environment failures the
baseline has; integration 324 passed, 0 failed.
The previous commit wrote the same proxy-aware logout expression into
four javascript.js.j2 templates. The rule it encodes - sign out of
oauth2-proxy first when the app is proxy-gated, else go straight to the
provider - belongs where proxy-gating is already decided.
utils/roles/applications/services/sso.py gains logout_url(url,
is_proxy_gated) next to get_sso_config, with OAUTH2_SIGN_OUT_PATH as the
path constant; the rd value is quoted exactly as Jinja's urlencode
filter does it. The sso lookup exposes it as the logout_url term: it
renders OIDC through the templar, takes is_proxy_gated from the same
resolution, and refuses a context without OIDC. Prometheus, Checkmk,
Baserow and Jitsi now render lookup('sso', application_id, 'logout_url').
Behaviour is unchanged: the resolver tests pin the rd encoding against
jinja2's do_urlencode, and the lookup tests cover a proxy-gated app, an
OIDC-native app, a disabled SSO and a missing OIDC.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 34421944328 Fediwall#1: Prometheus, Checkmk, Baserow and Jitsi inject
their own logout control, and it pointed straight at
OIDC.CLIENT.LOGOUT_URL. For an app behind oauth2-proxy that ends the
Keycloak session but leaves the proxy's own session cookie on the app,
so the app stays signed in.
When lookup('sso', application_id, 'is_proxy_gated') holds, the control
now goes through /oauth2/sign_out?rd=<url-encoded Keycloak logout>: the
proxy clears its cookie, then redirects to Keycloak. oauth2-proxy only
follows rd into its whitelist_domains, which carry the primary domain
and the node onion, so the redirect is allowed in clearnet and in tor
mode; the pihole spec already drives the same sign-out path. Apps
without the proxy keep the direct URL, and to_json quotes the value as a
JS string literal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The contract check ran as an Ansible task in the middle of every provider deploy, and the two client checks ran the same way. A test that is part of the deploy cannot be run without deploying, cannot be re-run on its own, and reports its verdict as a deploy failure. Twenty-seven task files lose 491 lines to this.
It is now what it always was. Each provider ships files/test/test.sh, templates/test.env.j2 and meta/tests.yml, and test-e2e-cli stages one shared harness beside it: run.sh drives contract.sh in a throwaway container on the provider's own network, then isolation.sh for adapter-fronted providers. The read probe each provider offers moved from the calling task into meta/mcp.yml next to the allowlist it must satisfy, and the schema now rejects a read probe naming a tool outside that allowlist.
The bearer the test presents is the one a client resolves through mcp.credential, via the new mcp_client_bearer lookup over resolve_credential, which moved to utils.roles.applications.mcp so it is the same function mcp_servers renders every client connection with. The probe presented whatever the calling role passed, which for the fifteen adapter-fronted providers is a different secret than any client holds.
Sidecar isolation is read from the engine rather than measured inside the container, so it no longer needs a delegation to the node holding the task: attachment is compared by network name, which drops the gwbridge special case, and a swarm worker says it cannot measure instead of skipping silently.
Retrying is bounded by wall clock against MCP_DEADLINE_SECONDS, which scales through the same lookup('timeout') as the harness budget it has to stay under. A flag that is neither true nor false aborts rather than reading as off.
mcp-provider-probe becomes mcp-provider-cli-test: an enabled provider must ship the test rather than include the task. mcp-probe-bearer becomes mcp-adapter-bearer, tying ADAPTER_BEARER to the declared credential key now that no probe include carries it. A new integration test renders the shared env template for every provider through Ansible's own templar, so a later change to it breaks visibly instead of poisoning twenty deploys quietly.
Hermes and OpenClaw keep their client-side proof as CLI tests of their own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nine roles enabled an MCP surface without a browser-side spec, so nothing
ever asked what the endpoint answers a caller with no credential at all.
Lift gitea's assertion into two shared registrars and register them per
role, and give each role the MCP_SERVICE_ENABLED flag they gate on.
mcpEndpointUrl now takes the base URL rather than reading APP_BASE_URL
unconditionally: jellyfin and zammad export their own base variable, and
declaring APP_BASE_URL for them would un-skip the shared guest persona,
which collapses itself on that variable being absent.
The env-gate scanner follows require("./x") into the shared harness so a
gate declared there counts as consumed, and strips block comments first
so service-gating.js's own doc examples do not credit EMAIL, SSO and OICD
to every role that imports it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Home Assistant pinned the MCP service account to its read-only group when it created the account and never again, so an account promoted to administrator by hand kept every mutating tool that scope denies while the deploy stayed green, because nothing looked at the existing case. The group is now asserted on every run and a refusal aborts instead of passing.
Nextcloud minted an app password under a fixed managed name and never revoked the previous one, so every run that found the stored password rejected added another valid token while the old ones kept full account access, accumulating across deploys. The mint is now preceded by a listing and the revocation of every token carrying that name. Both new tasks mask their output, because the listing names the account and its token ids even though the JSON strips the secrets.
Prometheus ran with no query timeout, so its two-minute default applied: a query the adapter abandoned after fifteen seconds kept the server busy for the rest of it. The flag is now derived from the adapter's own limit and doubled, which gives the server room past the adapter's deadline while keeping the two from drifting apart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six findings from a full audit of the roles this branch touched. None was
behind a failing job; all of them lie about what the code does.
Two defaults that can never fire, and would do harm if they did.
vars/main.yml gave services.tor.flavor a 'public' fallback, but
meta/services.yml:28 always defines the key - so the fallback is
unreachable, and it defeats the assert at tasks/00_core.yml:4-9: remove
the key and the role silently deploys public instead of failing loudly.
templates/dnsmasq-tor-onion.conf.j2 guarded twice on
`TOR_DNSMASQ_OWNS_LISTENER | default(true)`, while tasks/router.yml:35-37
sets that fact unconditionally two tasks earlier; had it ever gone
missing, `true` would have reshaped a dnsmasq this role does not own.
Two tasks with no readers. sys-front-inj-logout/tasks/01_deploy.yml
stat'ed logout.js to build INJ_LOGOUT_JS_VERSION; neither that fact nor
INJ_LOGOUT_JS_STAT is referenced anywhere under roles/ or group_vars/.
Cache busting already runs independently through lookup('local_mtime_qs')
in templates/head_sub.j2:1.
One README claim inverted. svc-net-tor's feature list named
TOR_EGRESS_ENABLED and called it off by default. That variable exists only
in docs/requirements/031-*.md - the implementation settled on
TOR_EGRESS_ROUTER_ENABLED, which group_vars/all/19_tor.yml:4 sets to true.
The README stated the opposite of its own default under a name nothing
reads.
Three files nothing can load, in web-app-prometheus. tasks/reset.yml is a
single comment and is included nowhere; templates/compose.override.yml is
one byte, and sys-svc-compose/tasks/04_files.yml:52-54 searches only
templates/compose.override.yml.j2 and files/compose.override.yml;
templates/style.css is a single comment, and sys-front-inj-css/tasks/main.yml:10
searches only templates/style.css.j2 and files/style.css.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The injected logout built `?redirect_uri=<web_protocol>://<DOMAIN_PRIMARY>`
and appended it to the end-session endpoint. Keycloak 26.7.1
(roles/web-app-keycloak/meta/services.yml:49-50) does not accept
redirect_uri there - it takes post_logout_redirect_uri plus id_token_hint
or client_id - so the parameter was dropped on arrival. Run 32284541284
job 96178507650 shows the proof: Keycloak answered with its logout-confirm
interstitial carrying client_id=account, i.e. it resolved no client from
the request at all.
The value was wrong as well as inert. DOMAIN_PRIMARY is claimed only by
roles/svc-prx-openresty, which declares no sso block, so it is never a
registered redirect URI - the parameter would have been rejected on
clearnet too the moment it became effective. Under tor it also glued the
app's own scheme to the portal's host.
Removing rather than correcting is what keeps this safe: every way this
could break requires a redirect target to be validated, and after the diff
there is no target. The emitted request is a strict prefix of today's.
The same three lines were duplicated verbatim in four web-app roles;
they collapse to the bare endpoint the shared role already passed. The
caller graph is closed - initLogoutPatch appears exactly three times
(definition, export, call site) - so the arity change leaves no orphan.
Fixes nothing that was failing; no CI job was red because of this.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mount pipeline was split across two layers: compose_volumes and
container_volumes lived as filter modules that ansible's filter loader
rejected on every start ("has no attribute 'FilterModule'", two warnings
per run), while the same-named lookups wrapped them thinly. The filter
bodies now live inside their lookups (the tor_socks precedent), the
filter files are gone, and the render tests moved next to the lookup
tests.
As lookups they see the templating context, so the application_id term
every call site passed back to its own role is gone too: both lookups
read application_id from context and fail loudly when it is missing.
107 templates and nfs_prep.yml drop the term; the calls collapse to
lookup('compose_volumes') and lookup('container_volumes', service_name).
The container_volumes call is now canonical - four leading spaces,
service_name in the service position, | indent(4) - enforced by a new
lint (test_container_volumes_call_shape). Normalizing the 85 literal
call sites surfaced that erpnext keyed its meta/volumes.yml mounts by
container name instead of compose service key, the only role to do so;
its meta now uses the service keys. wordpress's column-0 call, which
rendered its first line at top level, is indented like every other.
Template logic that only restated meta/volumes.yml is deleted: mailu fed
compose_volumes an extra_volumes dict whose eight names were looked up
from the very meta file compose_volumes reads itself; zammad's init and
railsserver extra mounts were the volumes.yml entries minus the two
services, which the meta now lists (the when-gated OIDC binds included).
Conditionality that meta cannot express stays in code, but in role-local
lookup plugins instead of template loops: matrix_bridge_mounts renders
the per-bridge registration mounts, nextcloud_extra_volumes the
feature-gated whiteboard/recording volumes (type: volume entries are
declared unconditionally by compose_volumes, so gating must happen
before it).
Parity is proven, not assumed: every mechanically rewritten template is
byte-identical to HEAD after the transformation regexes (95/95), and the
semantic roles render identically through the real functions against
HEAD's meta - zammad across six sso/tls polarities, all seven erpnext
services, mailu's top level. make test-lint passes complete.
Also: suitecrm's compose template pins container_extra_hosts with
host_alias (the SOCKS proxy URL host.docker.internal never resolved in
compose+tor), and the five disabled mautrix bridge addons use the
empty-default config lookup so templating the applications dict stops
emitting 125 errors per batch in every CI job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Compose interpolates a bare $ in an inline environment value, so a tool
whose input schema carries a "$schema" key reaches the adapter with that
key renamed to the empty string. The tools mapping then hashes differently
than meta/mcp.yml pinned, assert_no_drift refuses to serve, and the
container crash-loops with a symptom three steps from its cause.
Only web-app-jenkins carries a $ today; the other twelve render the
contract identically, so the same capture would have broken any of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A container without a healthcheck is accepted the moment its process starts. The restore drill asks the host's health service for its verdict, so a service that came back empty passes as long as it is running - which is how an empty volume survives a drill unnoticed.
Twenty-one services get a probe and thirteen get a marker naming why none is possible: a one-shot client that exits, a build image with no service port, a media bridge on UDP. The flavor comes from each image's own layer history, read from the registry through the mirror, so no image had to be pulled to find out whether it ships curl, wget or a shell.
Two entries carry more weight than the rest. The central Postgres, which every replay of the recover chain lands in, had no probe at all; it gets the pg_isready line that sys-svc-rdbms already runs for the per-app engine. And the matrix runner starts systemd within a second while its inner daemon is still pulling every image of the deployment, so it counted as healthy long before anything served - it now waits for Synapse to answer.
No probe replaces one: nineteen of the twenty-one images ship no HEALTHCHECK, and the two that do already carried a declaration before this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/volumes.yml says where a volume lands inside its container. Wherever a
second file spelled the same path out, the two drifted independently: the mount
moves, the consumer keeps pointing at the old place, and nothing fails until the
process looks for content that is not there. e031b7f61 fixed exactly this for
web-app-moodle; this generalises it and makes the rule hold.
The lookup gains what a caller needs. `targets` maps each mounting service to
its container path, and `target` is the one path an entry mounts at -- present
only when every mount agrees. An entry that lands at different paths per service
omits `target` rather than picking one, and the caller names the service:
"{{ lookup('volume', application_id, 'prometheus_config').target }}"
"{{ lookup('volume', 'web-app-prometheus', 'host_sys').targets['node-exporter'] }}"
66 literals across 48 role files now derive. Replacement only happened where the
path stood alone -- /srv/app inside /srv/app/data is a different path -- and
every rewritten file was re-parsed, Jinja and YAML alike. Three sites the tool
refused rather than guess were done by hand: zammad's path sat in a Jinja list
literal, seaweedfs' inside a quoted string feeding a filter, and openresty's
/var/run/docker.sock is not a mount target at all but the socket on the host,
which only happens to spell the same string. That one carries the single
nocheck.
tests/lint/repository/spot/test_volumes_path_single_spot.py keeps it that way.
roles/*/files/ is out of scope by construction: it is copied verbatim into an
image and executed there, so a Dockerfile or entrypoint has no Ansible to
evaluate a lookup with.
One defect this found, and it was in the tooling rather than the roles: deciding
ambiguity AFTER filtering short paths made web-app-prometheus' host_sys look
single-valued -- it mounts /sys for cadvisor and /host/sys for node-exporter,
and /sys is below the length filter. The template would have rendered an
undefined `.target`. Both the converter and the rule now decide ambiguity over
every target and filter afterwards, and that call reads targets['node-exporter'].
Verified by resolving rather than by reading: all 70 accesses in the diff were
run against the real registry and every one yields a non-empty path equal to the
literal it replaced. make test-lint: 328 passed.
Not verified: no Ansible render, so a call site whose surrounding context cannot
build the applications registry would still fail at deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
extra_hosts is the one compose key a service may not carry twice. The onion SSO
back-channel pin was emitted by a Jinja partial that six roles included, while
six other services wrote the key by hand; nothing stopped a service from
receiving both. YAML then either rejects the document or keeps one list and
drops the other, silently.
The partial becomes plugins/lookup/container_extra_hosts.py, and every pin now
goes through it. Callers add their own entries with extra_hosts=[...], the way
container_volumes already takes extra_volumes=, and the lookup merges them after
the automatic pin and deduplicates in first-seen order. With nothing to emit it
returns an empty string.
The swarm address reasoning from 7b1839681 is preserved and now written down in
the plugin rather than only in that commit: the pin must NOT be host-gateway in
swarm, because that alias resolves per node while openresty publishes :80 with
mode: host under node.role == manager. It carries the stack host's address,
which reaches every task because the compose file is rendered once with the
manager as inventory host. The mode is read as compose_mode_force or
DEPLOYMENT_MODE, not the bare cluster mode, so a role with an override does not
lose its pin and its overlay at once.
Two roles kept a pin that now collides with the automatic one, and both are
resolved the same way -- the hand-written entry stays for clearnet and yields
the .onion case to the lookup, which picks the address per mode. In
web-app-mailu the three conditional entries move into a role-local
mailu_extra_hosts lookup exposed as MAILU_EXTRA_HOSTS, because an eight-line
conditional expression in the template argument is worse than the block it
replaced. It returns its entries as separate results, so vars/main.yml uses
query(); plain lookup() would join them into one comma-separated string.
tests/lint/ansible/jinja/test_compose_extra_hosts_use_lookup.py fails on any
literal extra_hosts: key in a role template, so the invariant survives the next
author who needs a pin. It follows test_compose_ports_use_container_ports_lookup:
the raw construct is forbidden, the lookup is not mandated per service -- a
service with no pins is correct without it, and the swarm branch raises when
facts are absent, which would turn a no-op into a failure surface for containers
that never speak OIDC.
Equivalence checked by rendering rather than by reading: under the trim_blocks +
lstrip_blocks the stack-host action forces for compose.yml.j2, both former call
site shapes (indented, and matrix's column 0) emit byte-identical text to the
new form when the block applies. The no-op case differs -- a whitespace-only
line where the include emitted none -- and yaml.safe_load returns the same
document for both.
31 unit tests; make test-lint 325 passed. Does not touch web-app-espocrm, whose
missing pin is a separate claim and gets its own commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The adapter was the only image in the tree pinned by a hand-maintained digest,
and the pin was never recomputed by anything. It drifted twice: run 31374959721
lost seven of eight jobs to mirror/docker.io/python@sha256:6771159c... not
found, and the renewal to the newest upstream digest earlier today would have
reproduced it, because builds resolve FROM against the GHCR mirror while the
mirror is copied by tag and lags upstream by design.
The base is now FROM <image>:<version>, taken from the sidecar service in
meta/services.yml - the only file iter_role_images scans, so the tag the build
asks for is by construction the tag the mirror was told to copy. The version
moves from the moving 3.13-slim to 3.13.15-slim, which upstream stops rewriting
once published, and which the existing version drift job renews like every other
image.
With the build reading services.yml, the adapter block's digest, image and
version had no consumer left. They are removed together with the schema check
for the digest form, the warn-only drift test, and the manifest_digest helper
that existed only for it. What remains in the block describes the adapter rather
than the image it is built from.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each provider builds its own sidecar from the shared Dockerfile and
supplies the base image itself, so renaming the build argument had to
reach all ten call sites. Without this the argument is undefined, the
FROM line resolves to a bare image with no digest, and no adapter builds.
Ten applications with no MCP server of their own get one without a line of code
inside them: qdrant, checkmk, fider, jellyfin, listmonk, pretix, prometheus,
snipe-it, zammad and libretranslate.
The sidecar is confined rather than trusted. It runs read-only, drops every
capability, forbids privilege escalation, publishes no port and joins only its
own provider's network, carrying neither that provider's alias nor the consumer
networks the provider itself joined. Its tool surface is a JSON contract, not
code: which tools exist, which upstream path each maps to and what shape the
arguments take. Adding a tool is a line of JSON, not a new image.
The image is built locally from the audited adapter source with the base image
as a build argument, so all ten share one reviewed build instead of ten upstream
dependencies. Each role provisions its own read-only identity in the language
its application speaks, and every adapter include is guarded by the role's MCP
flag, so switching it off leaves no container behind.
Blocking a persona removes the only end-to-end proof that a role's auth chain
works. docs/contributing/artefact/files/role/playwright.specs.js.md:146 has
always required each flag to carry a one-line nocheck stating the role contract
that justifies it, plus a matching paragraph in the role's README or TODO.
Nothing enforced it, so 89 flags across 46 roles carried neither.
The lint mirrors test_has_spec's shape and uses is_suppressed_at with
mode="line-above", so the marker must sit directly above its flag -- a blank
line between them breaks the association, which is what the persona helper's
reader will look at. A literal =false blocks nothing and is exempt; the
sso-conditional ternary is not, because it blocks in every variant that pins
services.sso.enabled false.
Writing 89 reasons turned out to be the point. Five roles audited in one pass
blocked for five entirely different mechanisms, all of which read as the same
shrug behind a bare =true: pgAdmin's login id is the account e-mail rather than
the Keycloak username; phpMyAdmin authenticates as the MariaDB root account;
GitLab's seeded password belongs to the built-in root, not the administrator;
Mastodon's tootctl accounts create never sets a password; phpLDAPadmin binds
with an LDAP DN and a different secret. Further out: MediaWiki removes the form
that would accept the Keycloak secret ($wgPluggableAuth_EnableLocalLogin=false)
and bounces anonymous requests back into the IdP, so the unauthenticated
landing the contract asserts can never be observed; Mattermost Team Edition has
no native OIDC, so Keycloak rides the GitLab OAuth slot behind a relabelled
link; MinIO's console returns redirectRules: null and renders no SSO button at
all; XWiki's administrator is an XWikiUsers object with no password property.
Two flags turned out to be unjustifiable and are gone rather than annotated.
web-app-mailu gated PERSONA_GUEST_BLOCKED on services.dashboard.enabled, but
runGuestFlow does one anonymous goto and asserts no signed-in marker -- it
never touches the dashboard. A leftover from a helper version that entered
through the dashboard account menu. And web-app-mig's rationale claimed the
role is an in-app module of web-app-dashboard; it has its own compose stack,
healthcheck and canonical domain, and dashboard is merely a tile consumer.
Nine of the reasons say plainly that the block is not a role property but a gap
in the shared persona helpers -- decidim's sign-out is a data-method link the
generic click cannot activate, fider's SSO entry is a two-step modal, taiga,
wordpress, xwiki, odoo and penpot bind OIDC on a path the personas never visit,
jenkins blocks unconditionally although its OIDC variant looks runnable, and
opentalk's own spec proves both personas work. That list did not exist before;
it is the actual yield here, not the 89 comment lines.
45 pre-existing comments came into the comment lint's scope because this sweep
touched their files -- it only inspects files that differ from HEAD. 28 of them
were the banner "Service gate flags rendered for closure.", every one sitting
above a line that is not a service gate flag at all, orphaned by the
alphabetical re-sort these templates once got.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A native-metrics app on a node-local force_bridge network cannot be joined or resolved by a swarm-wide prometheus: the stack deploy fails with the network scope error and the scrape job would flap as permanently down. native_metrics_apps now drops those apps at the source (DEPLOYMENT_MODE swarm + networks.local.force_bridge), covering all four call sites (network precreate, compose service networks, external-networks block, scrape include) without per-template guards. DEPLOYMENT_MODE is templated without a default so a missing value fails loudly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New roles/sys-svc-container/templates/env.j2 sets TZ={{ HOST_TIMEZONE }} and LANG=C.UTF-8. Every role env.j2 includes it at the top; it renders via stack_host_template, the same engine that resolves the includes in compose.yml.j2, so it works in both compose and swarm. TZ/LANG are removed from the individual role env.j2 so each value stays single-source.
New lint tests/lint/ansible/test_env_base_include.py verifies every role env.j2 includes the base and never re-sets a base-owned key (keys derived from the base file).
mattermost and gitea meta/volumes.yml drop the /etc/timezone host-path bind: swarm hard-rejects a task whose bind source is missing on the node (Debian-trixie nodes have no /etc/timezone). TZ now comes from the base env. /etc/localtime bind also dropped; the container timezone follows TZ.
Non-directive comments were removed from the touched env.j2 (banners, restatements, reference URLs, narration) so the changed files pass the comment-lint; only tool directives remain.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every compose.yml.j2 now declares services as '{{ service_name }}:' with
its '{% set %}' knobs on two-space lines above the key, includes at four
spaces and if/for tags aligned to the content they wrap. The layout
renders correctly because stack_host_template enables lstrip_blocks for
compose.yml.j2; indent(4, True) in the container base and '{%+' guards
in whitespace-sensitive scalars keep the shared fragments identical
under both render modes. A new lint test enforces the skeleton.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The healthz, media, upload and ws location templates called proxy_pass
without internal_port, so in swarm they fell back to
services.<entity>.ports.internal.http. Apps whose entity service has no
internal http port (e.g. bluesky) hard-failed the nginx config render
with resolve_upstream: swarm upstream needs
services.<entity>.ports.internal.http. Forward internal_port (default '')
like html.conf.j2 already does; no-op where it is not passed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In swarm the sibling exporters (alertmanager, cadvisor, node-exporter, blackbox) and the healthz backend are unreachable on the compose container_name / host.docker.internal / 127.0.0.1 addresses, so prometheus marked every target down and /healthz/ready returned 502. Add a scrape_target lookup emitting <host>:<port> per mode (swarm tasks.<entity>_<service>:<internal_port>, compose container_name/compose_host), reading the internal port from services.<svc>.ports.internal; declare those ports on the four exporters; route both healthz probes through the proxy_pass lookup via one set variable. Compose output stays byte-identical.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The swarm openresty restart used 'container service update --force --detach' (fire-and-forget), so the deploy + e2e proceeded before openresty reconverged with prometheus.conf. The per-vhost log_by_lua then saw a nil metric_requests ('attempt to index global metric_requests') and the prometheus metricz/scrape e2e failed. Drop --detach so the update waits for convergence (bounded by the existing timeout 600). Also guard the per-vhost log_by_lua against a nil metric so any residual pre-init window can't throw.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Take the host IP as an optional third element of each term
([service, protocol, host_ip]) instead of an ip= kwarg, and build the whole
ports: block from declared ports.local/public (host) + ports.internal
(container). Convert every compose template across 84 roles, folding the
remaining raw var/numeric/specific-ip mappings (including mailu's per-interface
mail ports bound to MAILU_IP4_PUBLIC). Multi-term calls render one term per line.
Add the compose-ports-must-use-container-ports lint: it forbids any raw
"<host>:<published>:<container>" publish item under a ports: block and requires
every term to declare ports.internal. Remove
tests/lint/roles/test_swarm_internal_ports.py, now subsumed.
Fix a container_depends_on rendering regression. The lookup self-indented
(default indent=4), so the added 4-space placement + | indent(4) triple-indented
the block. Flip the lookup default to column 0 (consistent with
container_networks), make every call use the 4-space + | indent(4) form, and add
a lint enforcing it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rewrite every hardcoded literal container port in a compose.yml.j2 publish
mapping (- "<host>:<...ports.local|public.kind...>:<LITERAL>") to read the
declared ports.internal.<kind> via lookup, across 22 roles, so the published
(local/public) port and the container (internal) port are a single declared
source of truth. Mappings whose container already references the lookup or a
variable are left as-is.
This surfaced and fixed a real swarm bug: web-app-openproject declared
ports.internal.http: 8080 while the container actually listens on 80, so the
swarm resolve_upstream pointed openproject at the wrong port. Reconciled to 80
(the mapping's real container port).
Extend tests/lint/roles/test_swarm_internal_ports.py with a second check: a
published mapping's container segment must come from ports.internal via a lookup
(or a variable referencing it), never a hardcoded literal.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DEPLOYMENT_MODE is binary (swarm / compose), so a negated `!= 'swarm'` gate
always means `== 'compose'`. Replace every negated gate across roles/ and
group_vars with the positive form for readability, and add a lint
(tests/lint/ansible/variables/test_no_deployment_mode_negation.py) that forbids
`DEPLOYMENT_MODE !=` in .yml/.yaml/.j2 with a `# nocheck:
deployment-mode-negation` escape.
The compose-verb / chdir / jinja-gate lint guidance messages now suggest the
positive form too; the gate matcher (utils/annotations/task_gate.py) and its
test inputs keep recognising the legacy `!= 'swarm'` so a nocheck-marked
exception is still detected as compose-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
matomo: force-restart the swarm service after patching config.ini.php DB creds (the compose-up handler is a no-op in swarm), extracted into 10_resolve_host_cid. nextcloud: gate the redis PONG wait on the engine being local. prometheus: config 0444 (swarm config objects are root) and gate the host-monitor binds plus privileged cadvisor/node-exporter to compose only. dashboard: probe the public icon URL in swarm.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Give each reverse-proxied app a swarm-resolvable upstream: add services.<entity>.ports.internal.http to the role metadata and rename the main compose service from the generic 'application' to the entity name, so its short DNS alias on the shared overlay is unique. Thread the service name through compose, vars (*_SERVICE_NAME), volume mounts and the oauth2 origin host. Consolidate the split image/ports blocks in bridgy-fed and postmarks into the entity block.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Convert the two registry-driven networks templates into pure-Python
rendering functions plus thin Ansible lookup plugins:
roles/sys-svc-compose/templates/networks.yml.j2 -> {{ lookup('compose_networks') }}
roles/sys-svc-container/templates/networks.yml.j2 -> {{ lookup('container_networks') | indent(4) }}
The rendering logic lives in utils/networks/render.py as pure-Python
functions that take a registry + closure-style config/database lookups,
making the logic directly unit-testable without a Jinja stub harness.
The lookup plugins are thin wrappers that resolve application_id,
DEPLOYMENT_MODE, and swarm.network.encryption from the templar, build
the service_registry, and forward to the render functions.
Migrate 233 include lines across 85 production templates. Convert
web-app-matrix's hand-coded MDAD networks block (default + conditional
postgres + conditional openldap) to the generic lookup; this adds an
openresty attachment when services.sso.enabled (harmless extra
interface, pinned in test_matrix_mdad_consumer_attaches_postgres_ldap_sso).
Mark local-DB / redis sidecars (sys-svc-rdbms/templates/services/,
svc-db-redis/templates/service.yml.j2) with `# nocheck: networks-literal`
because they intentionally attach to default-only.
Extend the lint to scan ALL roles/*/templates/**/*.yml.j2 (not only
compose.yml.j2) and to forbid the legacy include form; the marker
above is the documented escape hatch.
Coverage:
- tests/unit/utils/networks/test_render.py (30 cases incl. encrypted=false literal pin)
- tests/unit/plugins/lookup/test_compose_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/unit/plugins/lookup/test_container_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/integration/infrastructure/compose/test_networks_render.py (10 snapshot scenarios)
- tests/integration/infrastructure/compose/test_network_includes.py (3 lint rules)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add plugins/lookup/container_image.py: returns the full `image: "<ref>"`
line so templates drop the redundant image: prefix and surrounding
quotes. Supports `custom=True` to auto-construct `<entity_name>_custom`
for locally-built images, plus explicit `image=`/`version=` overrides
for exotic cases (taiga back-auth/front-oidc variants, decidim infinito
suffix, etc.).
* 25 unit tests for the lookup cover compose vs swarm prefix, custom
flag, overrides, tag_only=True, missing version warnings.
* New strict lint (tests/lint/.../test_no_raw_image_in_compose_templates)
fails on any raw `image:` line in roles/*/templates/*.j2 unless
nocheck-suppressed. Legacy `image: "{{ lookup(...) }}"` form no
longer accepted.
* Migrate 159 image refs across ~80 roles to the standalone
`{{ lookup('container_image', application_id, '<service_key>') }}`
form. 26 roles use `custom=True` (mariadb, postgres, openldap,
jenkins, jira, gitlab, nextcloud, mediawiki, shopware, wordpress,
odoo, joomla, openproject, bookwyrm, espocrm, hugo, littlejs,
fediwall, roulette-wheel, openresty, confluence and more).
* Fix svc-db-redis application_id leak: service.yml.j2 is included
into consuming roles where application_id evaluates to the outer
role, so the image lookup needs the hardcoded 'svc-db-redis'.
* Integration tests (test_templates_require_image,
test_build_requires_image) recognise the standalone lookup as an
implicit `image:` at the same indent.
* Mediawiki swarm tasks: add explicit `:{{ MEDIAWIKI_VERSION }}` to
the registry-prefixed image string because the custom var is now
bare.
Only 3 legitimate exotic nocheck-suppressed image lines remain:
matrix-mdad runner (nested services.matrix.runner.image path),
matrix mautrix bridges (per-iteration dynamic) and svc-registry-docker
(chicken-and-egg self-reference).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
