Migrate every role's meta/volumes.yml from ad-hoc dict-of-named-volumes
to a canonical dict-of-dicts schema where the YAML key is the semantic
short name and the entry body carries type / name / source / mounts /
nfs / mode / read_only. The optional 'name:' field is the docker volume
name (defaults to the key); the previous 'docker_name:' field is gone.
What's in this commit (210 paths):
Infrastructure (23)
- New filter plugins/filter/{compose,container}_volumes.py — render the
top-level volumes:/configs:/secrets: section and per-service mount
blocks from the canonical meta. container_volumes uses the SEMANTIC
name as per-service mount source so it resolves against compose_volumes'
top-level dict key (docker_name as source produced 'undefined volume'
on swarm).
- New lookup plugins/lookup/{compose,container,volume}_volumes.py.
volume(application_id, name) returns a normalised single entry;
volume(application_id) returns the whole canonical dict.
- utils/cache/applications.py keeps a sibling registry
_CANONICAL_VOLUMES_BY_ROLE outside the merged applications payload so
raw Jinja source: strings never enter the templar walk (root-cause of
the 200x cache-miss + recursive get_merged_applications storm).
- utils/roles/applications/mounts.py — schema validator + normaliser;
list-shape is rejected.
- plugins/lookup/nginx.py: _volume_path_by_name reads canonical directly.
- scripts/tests/deploy/swarm/_context.sh: NFS_VOLUMES extraction handles
both legacy dict and canonical dict shapes.
- roles/svc-prx-openresty/handlers/restart_and_validate.yml: gate
compose-only validate+reload behind a single when-block; swarm uses
service-update instead (bind-mounts of a FILE pin source inode at
task-create time, so atomic write-tmp+rename leaves the running task
on a deleted inode).
- roles/sys-svc-compose/tasks/utils/nfs_prep.yml: switch to
lookup('volume', application_id) | dict2items so NFS subdir
pre-creation works under the new cache split.
Tests (12)
- New strict lints flagging inline mount sections in compose templates,
single-file bind mounts, and legacy lookup('config', ..., 'volumes.X.Y')
call sites — all volume-mount surfaces MUST flow through container_volumes
/ compose_volumes / volume lookups.
- Integration: test_volumes_compose_chain + test_volume_constants_are_lookups
accept both the legacy and the new lookup('volume', app, 'X').name forms
during the migration window.
- Unit tests for the new lookup, filter, mounts helper, plus updated
nginx lookup test.
- Prometheus role-wiring test reads canonical meta for the
alert_rules.yml bind mount.
Role migrations (175)
- 62 meta/volumes.yml converted to dict-of-dicts canonical shape
- 63 templates/compose*.yml.j2 replace inline volumes:/configs:/secrets:
with lookup('container_volumes', application_id, '<service>') |
indent(N) per service + a single trailing lookup('compose_volumes',
application_id) for the top-level block
- 50 vars/main.yml drop dead volume constants and refactor 23 remaining
legacy lookup('config', ..., 'volumes.X.Y') to lookup('volume',
application_id, 'X').Y across 9 roles
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`DIR_COMPOSITIONS` resolves to a Jinja expression in swarm mode
(`{{ DIR_VAR_LIB ~ "/compose/" if DEPLOYMENT_MODE == "swarm" else "/opt/compose/" }}`),
but Ansible passes variables un-templated into lookups. container, database,
and compose_file_args now call `templar.template()` when the raw value still
contains `{{`, so the resolved path reaches `get_docker_paths()`.
email's `_render` switches to `_templar_render_best_effort` (TrustedAsTemplate
for Ansible 2.19+) and loops up to 4 passes: nested inventory expressions like
`DOMAIN_PRIMARY="{{ lookup('env','INFINITO_DOMAIN') }}"` need multiple
substitution rounds before reaching the literal domain.
Drops a stale `import textwrap` in test_depends_on.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The mapping from compose-style `docker_restart_policy` (`no`,
`on-failure[:N]`, `always`, `unless-stopped`) to swarm's
`deploy.restart_policy.condition` (`none`, `on-failure`, `any`) was
seven lines of inline Jinja `{% set / if / elif / else %}` in
`roles/sys-svc-container/templates/deploy.yml.j2`. That made the
template noisy at the call site, hid the typo-safety logic from
unit tests, and forced every reviewer to re-derive the mapping table
from a chain of conditionals.
Move the mapping into a dedicated filter plugin so the call site is
a single readable expression and the mapping is exercised by a
focused test suite.
- plugins/filter/swarm_restart_condition.py: pure function, no Jinja
context dependency (the input is the only thing that matters).
Strips + str-casts the input so YAML-literal slop like `" no "`
or a numeric `0` does not defeat the mapping. Output is always a
member of the closed swarm condition set `{none, on-failure, any}`;
unknown / typo inputs map to `any` (safe default -- a misconfigured
long-running service still runs and the operator notices the typo
via the literal `condition: any` in the rendered file, instead of
silently degrading to `none` which would skip respawning a real
service).
- tests/unit/plugins/filter/test_swarm_restart_condition.py: 12
cases. Direct mapping per branch, `on-failure:N` suffix form
(suffix dropped because swarm carries max-attempts in a separate
`max_attempts` key), None / empty / whitespace fallback,
stringification of non-string input, strip semantics, an explicit
unknown-value test that pins the safe-default rationale, and a
property test over twelve inputs that asserts the output is always
inside the closed swarm condition set. A second test class
verifies the filter is registered under the exact name
`swarm_restart_condition` so a typo in the FilterModule shim does
not silently fall back to the identity filter.
- roles/sys-svc-container/templates/deploy.yml.j2: thirteen lines of
inline Jinja (`{% set _drp = … %}` + four `{% set
_swarm_restart_condition = … %}` branches) collapse to a single
filter expression:
condition: {{ docker_restart_policy
| default(DOCKER_RESTART_POLICY)
| swarm_restart_condition }}
The runtime mapping is identical to the pre-refactor inline
version (verified by the new unit tests against the same input
buckets), so nothing changes for existing roles.
The compose/swarm split on `depends_on:` shape (map form with
`condition:` vs. list form, per `docker stack deploy` constraint)
was being hand-coded as a `{% if DEPLOYMENT_MODE == 'swarm' %}` gate
around every block. Repetitive, easy to forget, and templates that
did remember it had to repeat the same six-line pattern per service.
The new `depends_on` lookup encapsulates the shape switch:
{{ lookup('depends_on', {SVC: 'service_healthy', …}) }} # mapping
{{ lookup('depends_on', [SVC_A, SVC_B]) }} # all default
- Compose -> map form with explicit `condition:` per entry.
- Swarm -> list form (just names; swarm ignores depends_on
ordering semantically but accepts the syntax).
- Mapping `None` value uses the default condition (`service_started`,
matching Docker Compose's behaviour when no condition is given).
- Plain list of names uses the default for every entry.
The output is shaped so the call site reads naturally at the
service-member column:
some_service:
image: …
{{ lookup('depends_on', {DB: 'service_healthy'}) }} # at col 4
volumes: …
Concretely the lookup emits line 1 WITHOUT leading whitespace and
lines 2+ with the `indent` kwarg (default 4) plus the YAML-relative
nested indent. Jinja substitution applies the template's leading
spaces to line 1 only, so the column-4 call placement makes
`depends_on:` land at column 4 and the children at 6 / 8. Lines 2+
carry their own indent because Jinja does not propagate template-side
leading whitespace to continuation lines.
Plugin + tests:
- plugins/lookup/depends_on.py: ~190 lines. Mapping / list / empty
input handling, default_condition kwarg, condition validation
against the closed set `{service_started, service_healthy,
service_completed_successfully}`, DEPLOYMENT_MODE resolution via
templar with `mode=` test-only override, indent kwarg semantics
documented in the module docstring.
- tests/unit/plugins/lookup/test_depends_on.py: 27 cases covering
TestInputNormalisation (mapping / list / empty / typo / single
string rejection / non-string sequence rejection / unsupported
type rejection / arity errors), TestModeBehaviour (compose / swarm
/ unknown-falls-to-compose / missing-DEPLOYMENT_MODE-falls-to-compose
/ mode-kwarg override), TestOutputFormatting (line-1 unindented
exact-string assertion, indent=0 edge case, indent=6 edge case,
insertion-order stability for diff-stable output).
Lint:
- tests/lint/ansible/jinja/test_compose_template_uses_depends_on_lookup.py:
rule key `compose-depends-on-must-use-lookup`. Flags three patterns
in any `*compose*.yml.j2` under `roles/`:
1. literal `depends_on:` keyword line
2. flow-form `depends_on: [a, b]`
3. literal `condition: service_*` line
The pre-existing `lookup('container_depends_on', application_id)`
is intentionally not flagged: its call form contains no literal
`depends_on:` token in source. Scan target broadened to
`name.endswith('.yml.j2') and 'compose' in name` so
`compose.override.yml.j2` and `compose-inits.yml.j2` are covered
alongside `compose.yml.j2` and `*.compose.yml.j2`.
Migration (43 depends_on blocks across 17 templates):
- erpnext (6): configurator + backend + websocket + scheduler +
queue-short + queue-long + frontend chains.
- mailu (8): imap + smtp + oletools + antispam + antivirus + webdav +
fetchmail + webmail.
- zammad (4): rails + 3 worker shapes.
- shopware (4): web + worker + scheduler + opensearch, init-gated.
- jitsi (3): three single-key dependencies on JITSI_PROSODY_CONTAINER.
- kix (3): backend + frontend + proxy.
- bigbluebutton/compose.override.yml.j2 (2): bbb-web + etherpad,
inside the existing BBB_COLLABORA_ENABLED gate.
- moodle (2): nginx + cron, both depending on moodle.
- opentalk (2): application + recorder.
- taiga (2): events + gateway.
- matrix (1): loop-driven via `{% set deps = {} %} +
__setitem__` escape hatch over MATRIX_BRIDGES.
- matomo (1): bootstrap service.
- nextcloud (1): whiteboard sidecar.
- openproject (1): proxy.
- funkwhale (1): front.
- magento (1): nginx.
- flowise (1): postgres + redis.
Pre-existing `lookup('container_depends_on', application_id)` calls
(erpnext, nextcloud, moodle, zammad, taiga, matrix, etc.) are left
untouched -- that lookup auto-detects DB / redis from the application
config and emits its own block; it is the app-aware counterpart to
this structural lookup.
Every call placement repositioned to column 4 (standard
compose-service-member indent) so the new line-1-unindented semantic
renders correctly. Eight templates needed the column-4 reposition
(via a mechanical one-shot script that was deleted after).
`make test` green; new lint + 27 unit tests green; independent
re-grep across all `*compose*.yml.j2` finds zero residual raw
`depends_on:` / `condition:` lines.
`docker stack deploy` names services `<stack>_<compose-yaml-key>` and
silently ignores the compose `name:` field (that field maps to the
compose-only `container_name`). The lookup was returning
`<stack>_<services.<key>.name>` instead, which only happened to be
correct when the YAML key and the `name:` field matched.
The matomo bootstrap service is the failure case: services.bootstrap.name
is `matomo-bootstrap` but the YAML key (and therefore the swarm
service name) is `bootstrap`. The lookup returned
`matomo_matomo-bootstrap`, the bootstrap-wait task polled an absent
service, and the wait timed out after 7 empty retries -- root cause
of the act-39 swarm bootstrap stall.
- plugins/lookup/container_service.py: swarm branch returns
`<stack>_<service_key>`; compose branch unchanged.
- tests/unit/plugins/lookup/test_container_service.py: flip the two
swarm assertions to the corrected expectations and rename
`test_swarm_mode_uses_resolved_service_name` ->
`test_swarm_mode_ignores_services_name_field` so the contract is
visible from the test name.
docker stack deploy passes env-file values verbatim, including outer
double quotes - the container reads KEY="value" instead of KEY=value
and DB auth, URL parsers, and int(os.environ['KEY']) consumers break.
Compose keeps stripping the quotes as before.
- plugins/filter/dotenv.py: read DEPLOYMENT_MODE from the render
context via @jinja2.pass_context. compose mode keeps the old quoting
(escape $$, \, "). swarm mode passes the value through verbatim.
Missing or unknown mode falls back to compose quoting so a typo
never silently leaks unquoted values into compose. Direct Python
callers without a Jinja context keep the compose-style default.
- tests/unit/plugins/filter/test_dotenv_quote.py: keep the 12
direct-call cases, add 8 TestDotenvQuoteModeAware cases rendering
the filter through a real jinja2.Environment.
- 4 OIDC_TITLE templates (bookwyrm, confluence, jira, pretix): replace
the manual "{{ X | replace('\"','\\\"') }}" Compose-only escape
with {{ X | dotenv_quote }}.
- 3 env.j2 templates (nextcloud, taiga, wordpress): drop literal
outer quotes that were Compose-only decoration; keep | dotenv_quote
on the password / token assignments.
- sys-svc-rdbms/templates/env/mariadb.env.j2: strip literal quotes
unconditionally - the file name is not env.j2 so the dotenv-quote
integration test does not require the filter and the generated DB
passwords are alphanumeric.
Add a sibling lookup `container_service` (with 12 unit tests) that
returns the addressable form of a service for container service ps,
container service logs, and container ps --filter
label=com.docker.swarm.service.name=... It mirrors `container_address`
but emits the swarm service name (<stack>_<service_key>) instead of
the runtime container ID subshell. Compose mode returns the bare
service name.
Make the matomo manager-side ops swarm-aware without pinning matomo
to the manager. 03_manager_ops splits the wait into two mode-specific
blocks: in swarm a container service ps loop on CurrentState, in
compose the existing container inspect. After convergence the swarm
block resolves the hosting node via container service ps Node and the
compose block sets it to inventory_hostname; both set MATOMO_HOST_NODE
via separate set_fact tasks so the undefined matomo_swarm_node_query
under compose never reaches Jinja. A second unconditional task
resolves the local container ID on the hosting node (compose: bare
name; swarm: container ps with the swarm service-name label filter)
into MATOMO_LOCAL_CID. Downstream 04_install_path, 07_patch_database,
08_patch_trusted_hosts and the health wait all delegate_to
MATOMO_HOST_NODE and use MATOMO_LOCAL_CID for container exec, so the
patches and inspects always hit a local container. 05_bootstrap keeps
the compose-only profile run and adds a swarm path that tails the
in-stack matomo_bootstrap service logs for a hex token, registering
the same matomo_bootstrap_run shape downstream. 09_config delegates
to STACK_HOST with run_once so the uri_retry hits the published
matomo port from the manager regardless of where matomo is scheduled.
Drop the now-unused MATOMO_CONTAINER_ADDRESS from vars/main.yml.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Tier-1 changes verified live in act-12 (mariadb stack converged, postgres
stack started, swarm-readiness gate fired correctly for docker/mariadb/
postgres/openresty).
* `plugins/filter/compose_volumes.py` + `plugins/lookup/compose_volumes.py`:
thread `dir_var_lib` through filter (required keyword, no default)
and lookup (reads from `vars_["DIR_VAR_LIB"]`, raises if missing). Replace
native NFS driver with a bind-mount from the host's already-NFS-mounted
`DIR_VAR_LIB` because native `type: nfs` re-handshakes portmap, which is
unreachable in nested DinD. Drop the FilterModule wrapper so the deprecated
pipe form keeps failing fast per the lookup-only SPOT contract.
* `tests/unit/plugins/{filter,lookup}/test_compose_volumes.py`:
thread `DIR_VAR_LIB` through every fixture (no default-fallbacks).
* `roles/sys-svc-compose/tasks/utils/post_deploy.yml`:
compose-mode db-health probe gated to `DEPLOYMENT_MODE \!= 'swarm'`; new
swarm-mode convergence gate runs the extracted `swarm_stack_ready.sh`
via `ansible.builtin.script` with `STACK` env, addressed by absolute
`playbook_dir`-rooted path so the include resolves from caller roles.
* `roles/sys-svc-compose/files/swarm_stack_ready.sh` (new): probe
`docker stack services` for a single `STACK` and exit non-zero until
every service's running/desired replica count matches.
* `roles/svc-db-mariadb/vars/main.yml`: new `MARIADB_CONTAINER_ADDRESS`
constant wraps the `container_address` lookup so callers reference one
SPOT instead of duplicating the lookup.
* `roles/svc-db-mariadb/tasks/01_core.yml` + `02_init.yml`:
switch root-password probes and bootstrap SQL calls to
`container exec -i {{ MARIADB_CONTAINER_ADDRESS }} sh -c 'mariadb -p"$MARIADB_ROOT_PASSWORD"'`.
Using the container's own `MARIADB_ROOT_PASSWORD` env var mirrors the
healthcheck and matches whatever quoting Docker swarm preserves from
the env_file, sidestepping the compose-vs-swarm env_file quote-strip
divergence.
* `roles/svc-db-postgres/files/set_postgres_superuser_password.sh`:
`eval "container=$container_raw"` so the swarm-mode lookup payload
`"$(/usr/bin/resolve-container-id <stack> <svc>)"` is re-evaluated by
bash and resolves to the live task id at exec time.
* `roles/svc-db-postgres/tasks/01_core.yml`: rename arg to use the new
swarm-aware `POSTGRES_CONTAINER_ADDRESS`.
* `utils/tests/swarm/extend_inventory.py`: when the seed app's primary
service declares `default_placement: manager`, do not add the app to
the worker group, so the swarm test inventory matches the
single-replica placement-pinned semantics.
The oauth2-proxy sidecar set cookie_domains to the app's primary domain only. A role served under several sibling subdomains (e.g. SeaweedFS filer/master/api under one base) therefore could not read its session cookie on the non-primary hosts, looping the auth chain. Add the common_dns_suffix filter and use it so cookie_domains becomes the longest DNS suffix shared by all of the app's domains: a single-domain app keeps its exact host (unchanged), a multi-domain app collapses to the shared parent so one session cookie spans every subdomain.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Switch penpot, nextcloud, gitea, mattermost, openwebui, funkwhale, baserow, openproject, pixelfed, mastodon, bookwyrm, listmonk, peertube and opentalk to the engine-agnostic objstore lookup: the minio service entries are pinned off, a seaweedfs service (gated on the provider being in group_names, public for the media apps) replaces them, and the matrix variants pin the new flag. OpenTalk keeps object storage mandatory through an embedded fallback and provisions its bucket via weed shell.
Add objstore lookup SPOT keys consumed by these roles: endpoint (host:port), public_domain, public_bucket (public S3 host plus bucket) and public_url (scheme plus public_bucket, scheme derived from the provider TLS state). This removes the repeated lookup('config', lookup('objstore', ...), 'server.domains.canonical.api') / host:port concatenations from the consumer templates. Listmonk settings render their JSON values via the to_json filter.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a database-style object-store abstraction: the objstore lookup resolves the active engine (seaweedfs|minio) per consumer and yields a uniform S3 payload, objstore_secret_key is auto-generated like database_password, sys-svc-objstore delegates per-consumer provisioning to the resolved provider's objstore_init entrypoint (lazy-loading the provider when it was not preloaded), and compose/network/volume plumbing supports both a shared central instance and an embedded storage-only instance per consumer.
Add the web-app-seaweedfs provider role: Filer and Master UIs behind the admin-only oauth2-proxy via a role-local nginx and proxy.conf.j2 (S3 API domain stays ungated, admin UIs deny with 403 when SSO is disabled), per-consumer S3 identities from s3.json, idempotent bucket provisioning via weed shell, a /24 on the central-provider subnet range, Playwright coverage for admin/biber/guest plus consumer bucket existence, and unit tests for the objstore lookup and the per-engine compose volumes.
Mark web-app-minio lifecycle: unsupported, move its mc client image into a dedicated services.client entry, align it with the objstore credential contract, and make its consumer provisioning idempotent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In compose mode the `address` key resolved to `instance`, which for a
non-shared local DB is the consumer entity (e.g. "mailu") rather than
the real container_name (`{entity}-database`, e.g. "mailu-database").
Callers like roles/web-app-mailu/tasks/02_wait_for_mariadb.yml then ran
`container exec mailu ...` and crashed the deploy with "No such
container: mailu" (see CI run 27247587073 job 80468402392).
For shared central DBs `container == instance` so this is a no-op
there; the fix only affects the non-shared local-DB path.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a mode-aware `container_address` lookup plugin and a
`resolve-container-id` helper script that resolve swarm task IDs at
exec time (compose mode returns bare names unchanged). Migrates ~25
roles + ~28 task files from `command:` to `shell:` so the lookup's
`$(...)` subshell evaluates correctly, and extends `database` lookup
with an `address` key for cross-role DB exec.
Three new lint tests enforce the contract: every container exec
target goes through the resolver, lives behind an IS_STACK_HOST gate
(walks the include graph one hop deep), and uses `shell:` rather than
`command:` when the target is a Jinja reference. Existing nocheck
suppressions on bare-name DB exec sites (nextcloud redis, listmonk,
moodle OIDC, mailu) are replaced with the lookup-based form.
Fixes the systemd Environment= injection in sys-ctl-cln-disc-space
(it never expanded $(...)) by passing STACK / SERVICE_KEY parts and
resolving inside the script. Extracts shopware admin and mediawiki
extension install bodies into dedicated .sh files invoked via env.
Rewrites password-bearing shell tasks to either `| quote` directly or
the `-e VAR=... | quote` + `sh -c '... ${VAR} ...'` env-bridge pattern.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The lookup read DEPLOYMENT_MODE via str(vars_.get(...)).strip() without
running it through templar.template(). DEPLOYMENT_MODE in
group_vars/all/18_swarm.yml is a Jinja expression
({{ 'swarm' if (groups['svc-swarm']|default([])|length) > 1 else 'compose' }}),
so the untemplated literal never equals 'swarm' and the lookup always
fell through to the compose branch — emitting depends_on as a mapping
(e.g. `redis: {condition: service_healthy}`). docker stack deploy
rejects that with `services.X.depends_on must be a list`, breaking
mastodon, friendica, gitea, onlyoffice and matomo on swarm.
Mirror the templating pattern already used by every sibling lookup
(compose_replicas.py, compose_volumes.py, compose_only.py,
compose_restart.py): capture self._templar, fetch DEPLOYMENT_MODE, and
pass it through templar.template() inside contextlib.suppress(Exception)
before comparing against 'swarm'. Behaviour for already-resolved values
('swarm', 'compose', missing default) is unchanged; only the unresolved
Jinja case is fixed.
Add a regression test (test_unresolved_jinja_deployment_mode_is_templated)
that drives the lookup with the exact group_vars expression plus a stub
templar and asserts the list form is emitted.
`docker stack deploy` rejects compose map-form `depends_on` with
`services.<svc>.depends_on must be a list`; the syntax is only valid
for `docker compose v2`. Until now the lookup always emitted the
map form, so every swarm-deployed app with a local DB or Redis
aborted the stack deploy handler in sys-svc-compose with rc=1
(CI job 79876947270 on friendica).
Branch on `DEPLOYMENT_MODE`:
* compose: keep the map form so health-check ordering still applies.
* swarm: emit the bare service-name list; swarm has no health-check
gated ordering anyway.
Adds two regression tests pinning the emitted shape per mode.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Ansible 2.19 introduced a `_trust_check` in the template engine that
silently returns Python-constructed strings unchanged when they lack the
TrustedAsTemplate tag (see ansible/_internal/_templating/_engine.py:288).
`self._templar.template("{{ IS_STACK_HOST | bool }}")` therefore returned
its own input every time, _to_bool rejected the literal, and every
nginx-tree conf_template task aborted on swarm-mgr-01.
Wrap the expression with `ansible.template.trust_as_template` before
handing it to the templar. Adds a unit test that uses `is_trusted_as_template`
to guard against future regressions that drop the tag.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous fix (`self._templar.available_variables = task_vars`) was
not enough in the Ansible version the swarm CI ran on: templar.template()
still returned the literal Jinja string and _to_bool rejected it. Build
a fresh Templar(loader, variables=task_vars) so the variable scope is
explicit and version-independent.
Also drop `| default(false)` from the gate expression. IS_STACK_HOST is
defined unconditionally in group_vars/all/18_swarm.yml; the default was
dead code that would silently mask a future regression where the SPOT
gets removed. Without it, an undefined IS_STACK_HOST raises loudly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CI run 27044129926 failed every nginx-templating job with
"The value '{{ IS_STACK_HOST | default(false) | bool }}' is not a
valid boolean". Action plugins must explicitly expose task_vars to
their templar before calling template(); without it the renderer has
no variable scope and returns the Jinja string verbatim, which
_to_bool then rejects.
Sets self._templar.available_variables = task_vars in run(), matching
the standard pattern for action plugins that template against play
vars. Adds a unit test that holds the FakeTemplar's available_variables
mirror in place so any future regression that drops the seed call
fails the suite.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds plugins/action/conf_template.py, a drop-in `template:` replacement
that returns `skipped: true` whenever IS_STACK_HOST evaluates to false.
Routes the rendered Jinja through ansible.module_utils.parsing.convert_bool
so both Python bool and string "True"/"False" templar outputs resolve
correctly. Inherits every standard template module argument.
Migrates seven nginx-tree consumers that previously aborted on workers
with "Destination directory does not exist": sys-front-tls,
svc-db-openldap, sys-svc-webserver-core (nginx.conf + default_server.conf),
web-app-matrix (synapse vhost), web-app-peertube (per-domain vhost),
web-opt-rdr-domains (redirect vhost), and web-app-prometheus.
Tags twelve legitimate bare `template:` callers with
`# nocheck: conf-template`: per-host system configs (locale, nix, systemd
manager defaults, journald, msmtprc) and per-app compose-dir configs
(magento x2, mailu x2, moodle, odoo, taiga) that intentionally bypass the
project's nginx tree.
Adds tests/lint/ansible/modules/test_conf_template_migration.py as the
drift guard: any future `template:` task whose `src:` ends in `.conf.j2`
must adopt conf_template or carry the nocheck marker. Adds seven unit
tests for the plugin covering skip semantics, dispatch, mode defaults,
mode overrides, full-arg passthrough, and the string-bool templar path.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add the compose_replicas lookup plugin as the single source of truth for
swarm replica calculation. Default emits `replicas: len(groups[application_id])`
(topology-driven); override term forces an explicit value. In compose mode
the lookup returns the empty string because docker-compose ignores
deploy.replicas.
Refactor deploy.yml.j2 to use the lookup. The previous service_replicas
default in defaults/main.yml is no longer needed and is dropped.
Add a lint test that walks every roles/*/templates/compose*.yml.j2 and
verifies each service entry under the top-level services: declares
replica handling via one of three valid paths:
1. Include sys-svc-container/templates/base.yml.j2 (full inheritance).
2. Include sys-svc-container/templates/deploy.yml.j2 directly (escape
hatch for services that own their restart/logging/env_file and
therefore can't use base.yml.j2 without duplicate-key conflicts).
3. Call lookup('compose_replicas', ...) explicitly for handcrafted
deploy: blocks.
Fix the 7 web-app compose templates flagged by the new lint (bluesky,
dashboard, magento, matrix, moodle, nextcloud, opentalk) by including
deploy.yml.j2 directly and setting service_name as a SPOT before each
service entry, so the service name in the yaml key and the resource_filter
lookups in deploy.yml.j2 both flow from one variable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New ``cli.administration.deploy.swarm`` package wraps the strict
``cli.administration.deploy.dedicated`` runner with a layered closure
so swarm deploys never require operators (or test scripts) to
enumerate every infra role:
effective --id = seed (operator --id, else inventory_role_groups)
∪ dep-walk(seed via applications_if_group_and_all_deps)
∪ default-placement safety net
The dep-walker is the same SPOT
(``utils.roles.applications.in_group_deps.applications_if_group_and_all_deps``)
the runtime constructor consumes via ``lookup('applications_current_play')``,
so the CLI-side closure and the in-play role set stay in sync.
The dedicated CLI keeps strict explicit-only semantics: ``main`` was
split into ``parse_args`` + ``run_from_args`` so the swarm wrapper
can share parser, mode wiring, validator and runner verbatim and
only mutate ``args.id``.
Meta-side: ``svc-db-postgres`` gains ``default_placement: manager``
(semantic symmetry with mariadb). ``default_placement`` is now a
PLACEMENT hint only ("if deployed, run on the manager"); deploy
decisions come from the dep-walk. Apps that declare ``mariadb:``
pull svc-db-mariadb; apps that declare ``postgres:`` pull
svc-db-postgres; apps that declare neither get no central DB.
Shared utils extracted so the four consumers stay aligned:
* ``utils.inventory.groups.inventory_has_group``: YAML + INI static
inventory group probe (extracted from ValidDeployId).
* ``utils.roles.meta_lookup.{get_role_default_placement,
iter_roles_with_default_placement}``: single source of truth for
``default_placement`` discovery (lookup plugin is now a thin wrapper).
Test pipeline (``scripts/tests/deploy/swarm/*``):
* Heredoc-Python shells converted to proper modules under
``utils.tests.swarm.*`` and invoked via ``python -m``:
- extend_inventory.py (was 08_extend_inventory.sh)
- write_extras.py (was 09_write_extras.sh dynamic half)
- derive_includes.py (replaces 06's hardcoded includes array)
- write_hosts_entries.py (replaces 02's hardcoded /etc/hosts heredoc;
derives every canonical + www-redirect domain from APP_ID via
canonical_domains_map + generate_all_domains, so the test
works for any app in the matrix without per-app maintenance)
* 01_create_network.sh deleted (single docker command inlined in workflow).
* 11_deploy.sh deleted (swarm CLI now called directly in workflow).
* 09_write_extras.sh: static half moved to inventories/development/swarm.yml.
* Remaining shell scripts renumbered 01-13 without gaps.
* 08_wait_converge.sh: case block on DB_DEP replaced with single
conditional (container name == DB_DEP).
* 06_provision_inventory.sh: hardcoded includes array replaced with
`mapfile -t includes < <(python -m utils.tests.swarm.derive_includes)`.
* Workflow YAML step references updated to new script names + module
invocations.
New tests (39 covering the closure surface):
* tests/unit/cli/administration/deploy/swarm/test_closure.py (14)
* tests/unit/utils/inventory/test_groups.py (11)
* tests/unit/utils/roles/test_meta_lookup.py (+3 for default_placement)
* tests/unit/utils/roles/validation/test_deploy_id.py (unchanged
8 tests still pass after refactor onto the shared util).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Single SPOT lookup `plugins/lookup/compose_volumes.py` absorbs the
applications-lookup pipe step and auto-wires `DEPLOYMENT_MODE` plus
the `storage` group var from the templating context. Callers only
pass what genuinely varies per service (`application_id` and an
optional `extra_volumes` dict).
Behavior change (bug fix): the filter was always invoked with the
default `deployment_mode='compose'` because no caller passed it
explicitly, which silently skipped the swarm-NFS driver-opts
injection path. The lookup reads `DEPLOYMENT_MODE` from globals,
so swarm-NFS volumes now render the `driver: local` / `driver_opts:
type: nfs` block correctly. Callers can still override via
`deployment_mode=` / `storage=` kwargs when a template needs
non-default rendering.
Other coordinated changes:
- 81 template call sites migrated from pipe form to lookup form
(single-line and multi-line shapes).
- `FilterModule` registration in plugins/filter/compose_volumes.py
removed so the deprecated pipe form fails fast at render time.
The underlying `compose_volumes()` function stays exported and is
imported by the new lookup plus the existing 21-test filter unit
test module.
- 10 new lookup unit tests pin the contract (term shape, kwargs
pass-through, auto-wiring, override precedence, error cases).
- Integration test `test_compose_volumes_call_required.py` updated:
asserts the new lookup form is present in every `compose.yml.j2`
and adds a second test forbidding the deprecated pipe form
anywhere under `roles/**/templates/`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace per-key `compose_container_name` lookup with a generic
`compose_only(key, value)` SPOT that emits `key: "<value>"` only in
compose mode and nothing in swarm. Same gating logic now serves
container_name (157 call sites) and pull_policy (17 call sites)
without duplicate plugin code.
Why this consolidation:
- The earlier `build-key` lint rule was misdirected. Swarm tolerates
`build:` with a harmless warning when `image:` is pre-pushed (via
the existing handlers/swarm_registry_push.py flow in
roles/sys-svc-compose). The actual swarm-rejected key is
`pull_policy:` ("Additional property pull_policy is not allowed"),
and 10 templates were silently broken because they declared inline
`pull_policy: never` outside the existing `{% if DEPLOYMENT_MODE
\!= 'swarm' %}` gate in roles/sys-svc-container/templates/build.yml.j2.
- A single generic lookup is cleaner than N per-key wrappers when
the gating logic is identical (emit-key-or-nothing).
Lint changes in tests/lint/ansible/jinja/test_mode_incompatible_keys.py:
- New `pull-policy-key` rule, regex `^\s*pull_policy\s*:`, remediation
points at compose_only.
- Existing `container-name` rule keeps its raw-key regex; remediation
now also points at compose_only.
- Broken-split / broken-embed regexes rekeyed on the `compose_only`
lookup name (was `compose_container_name`).
- `build-key` rule and its 12 nocheck annotations across templates
removed.
Coordinated changes:
- 86 templates rewritten (157 container_name calls migrated;
17 inline pull_policy lines replaced).
- 1 redundant `{% if DEPLOYMENT_MODE \!= 'swarm' %}` guard dropped in
build.yml.j2 (the lookup self-gates).
- 12 unit tests pin the compose_only contract.
- Catalog (suppression.md) updated: `pull-policy-key` row added,
`container-name` row remediation updated.
- Old `plugins/lookup/compose_container_name.py` and its test module
deleted.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mirror compose_container_name pattern: plugins/lookup/compose_restart.py
emits `restart: <policy>` only in compose mode. In swarm the key is
silently ignored (replaced by deploy.restart_policy) and produces
"Ignoring unsupported options" warnings at docker stack deploy time.
Sweep 19 raw restart: sites across 10 templates to the no-arg form
`{{ lookup('compose_restart') }}` (the lookup falls back to
DOCKER_RESTART_POLICY from ansible-vars).
base.yml.j2 keeps the explicit-arg form
`{{ lookup('compose_restart', docker_restart_policy | default(DOCKER_RESTART_POLICY)) }}`
so the 5 templates that override the policy via Jinja-scope
`{% set docker_restart_policy = ... %}` (erpnext, zammad, shopware,
matomo, openproject) retain their override path. Jinja-scope bindings
are visible only in the rendering template, never in the Ansible
variable context that the lookup sees, so the caller must resolve them.
Generalize the container-name lint into a data-driven multi-rule
scanner at tests/lint/ansible/jinja/test_mode_incompatible_keys.py
covering both container-name and restart-key. Adding a new rule is one
ModeRule row sharing the file-walk / suppression / line-grouping
scaffolding.
Also gate roles/svc-db-redis/templates/service.yml.j2's
resource.yml.j2 include with DEPLOYMENT_MODE \!= 'swarm'. Mirrors
base.yml.j2's existing pattern; prevents cpus / mem_limit /
mem_reservation / pids_limit from leaking into swarm output where
deploy.resources.* in deploy.yml.j2 already handles them.
11 unit tests pin the compose_restart lookup contract. Both rules
registered in docs/contributing/actions/testing/suppression.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
container_name is illegal in swarm mode when replicas > 1 (docker rejects
with "services.deploy.replicas: can't set container_name and X as container
name must be unique"). All compose.yml.j2 templates now route through a
single lookup that suppresses the key in swarm mode and emits it in compose
mode.
SPOT: plugins/lookup/compose_container_name.py
Reads DEPLOYMENT_MODE from the templating context, returns
'container_name: "<name>"' under compose and an empty string under swarm.
Call sites read:
{{ lookup('compose_container_name', MY_CONTAINER) }}
No explicit DEPLOYMENT_MODE pass-through needed.
Sweep: 82 templates rewritten across roles/web-app-*, roles/web-svc-*,
roles/svc-*, plus 4 partial includes (sso_proxy, redis, mariadb, postgres).
For container names assembled from multiple Jinja fragments the args use
tilde concat (X ~ '-' ~ Y) so the lookup receives a single resolved string
and the call stays a single well-formed Jinja expression.
Lint: tests/lint/ansible/jinja/test_no_raw_container_name.py blocks
(a) raw 'container_name:' YAML keys in any roles/*/templates/**/*.j2,
(b) split-Jinja regression where the outer braces close mid-call:
{{ lookup('compose_container_name', A }}-{{ B) }},
(c) embedded-Jinja regression inside a string argument:
lookup(..., "{{ X }}_suffix") (removed in ansible-core 2.23).
The (b) and (c) regexes accept either single- or double-quoted lookup
name and scan the full file so multi-line shapes also surface. Suppress
per-site with '# nocheck: container-name' on the same or preceding line.
Tests: tests/unit/plugins/lookup/test_compose_container_name.py pins the
lookup contract: compose vs swarm output, whitespace tolerance, default-to
compose when DEPLOYMENT_MODE is absent, AnsibleError on missing or excess
terms, non-string name coercion.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Roles that declare default_placement: manager in meta/services.yml are now
auto-added to their own inventory group when the operator did not list them
explicitly. Both the runtime constructor add_host loop and the CLI
pre-flight inventory extender use the new roles_with_default_placement
lookup so swarm and CLI-validation stay consistent.
In compose-mode (no svc-docker-swarm-manager group), svc-prx-openresty
auto-joins the first play host. run_once + ansible_play_hosts | first
keeps that deterministic in multi-host compose deploys so the openresty
vhost view does not fragment.
The svc-cache-registry / svc-docker-registry entries previously hardcoded
in scripts/tests/deploy/swarm/08_extend_inventory.sh fall out of the new
dynamic loop; net inventory is unchanged.
ansible_play_hosts joins the magic-var allowlist in
test_variable_definitions so the constructor add_host passes integration
lint.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drop the "web-opt-rdr-domains in deployed" gate in the
current_play_redirect_domains lookup. CURRENT_PLAY_REDIRECT_DOMAINS now
always carries DOMAIN_PRIMARY to DOMAIN_HOMEPAGE when both are set and
differ. This triggers sys-stk-front-base to auto-load web-opt-rdr-domains
so openresty actually renders a vhost matching its declared canonical, and
the bare apex stops hitting the default-server's ssl_reject_handshake.
CSP probing skips those redirect-source domains: pure 301 vhosts have no
CSP header to validate.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the svc-docker-swarm, svc-docker-swarm-manager, svc-storage-nfs-server
and svc-storage-nfs-client roles, the compose_volumes filter NFS support,
the test-deploy-swarm-nfs CI workflow with its swarm-nfs-pilot script set,
and the deploy_mode-aware compose/swarm wiring across mediawiki, mariadb
and sys-svc-* helpers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TC003: move `pathlib.Path` into `TYPE_CHECKING` blocks across the
new `requirements/archive` package and its completeness lint test.
* TC001: same treatment for `VaultHandler` in
`credentials/emit.py` (annotation-only).
* SIM105/S110: replace `try/except/pass` in
`current_play_redirect_domains.py` with
`contextlib.suppress(Exception)`.
* `archive/__main__.py`: mark `README_PATH.read_text(...)` with
`# nocheck: cache-read` — CLI rewrites the README in the same flow,
mirroring the `changelog/archive` pattern.
* Drive-by ruff-format condensations in the archive unit tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The plugin's previous
domain_primary = str(vars_.get("DOMAIN_PRIMARY") or "")
domain_homepage = str(vars_.get("DOMAIN_HOMEPAGE") or "")
read inventory values that may carry Ansible 2.18+'s
``_AnsibleTaggedStr`` with the ``TrustedAsTemplate`` tag — exactly
the shape ``inventories/development/default.yml:20`` produces with
DOMAIN_PRIMARY: "{{ lookup('env', 'INFINITO_DOMAIN') }}"
A bare ``str(value)`` casts the tag away and leaves the literal Jinja
text in the plugin output. The output then flows into
``CURRENT_PLAY_REDIRECT_DOMAINS`` via the set_fact in
``tasks/stages/01_constructor.yml`` and lands in
``roles/web-opt-rdr-domains/tasks/main.yml`` loop items as
item={'source': "{{ lookup('env','INFINITO_DOMAIN') }}", 'target': "{{ … }}"}
The ``lookup('tls', domain, 'enabled')`` call in
``roles/web-opt-rdr-domains/tasks/redirect-domain.yml:5`` then
receives the literal Jinja string and raises
The lookup plugin 'tls' failed: tls: application_id
'{{ lookup('env', 'INFINITO_DOMAIN') }}' not found in domains mapping
(job 77797371504 on run 26428080957). That CI ran on SHA d1410c0be,
which already shipped this plugin — so the upstream replacement of
the inline two-stage set_fact merges with the lookup plugin did not
fix the underlying trust-tag-strip bug; it just relocated it from
Jinja to Python.
Add ``_resolve_str`` that runs the value through ``self._templar``
before string-coercion. Idempotent for already-resolved strings,
exception-safe (falls back to ``str(raw)`` if the templar is
unavailable, e.g. on hosts without rendered vars). Both
``DOMAIN_PRIMARY`` and ``DOMAIN_HOMEPAGE`` route through it.
Defense-in-depth with the existing ``plugins/lookup/tls.py``
defensive templating (commit f32266d53): even if a future code path
hands ``lookup('tls', …)`` an untemplated string, the tls plugin
itself now templates the first term. Either layer alone suffices for
the rdr-domains failure; both together guarantee the symptom cannot
re-surface from a single regression elsewhere.
Coverage: two regression tests in
``tests/unit/plugins/lookup/test_current_play_redirect_domains.py``
— ``test_jinja_tagged_domain_primary_is_templated`` proves the
templar call happens and produces the resolved domain, and
``test_templar_failure_falls_back_to_raw_value`` proves a raising
templar does not crash the plugin. The 10 existing tests stay green.
``make test`` green across all 5 targets.
Class follow-up: 11 further lookup plugins read inventory values
without ``templar.template`` first (see git log for ``vars_.get``
call sites). Not in scope for this fix; logged for a separate
audit pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CI job 77797371504 on run 26428080957 crashed with:
tls lookup: application_id '{{ lookup('env', 'INFINITO_DOMAIN') }}'
not found in domains mapping
`DOMAIN_PRIMARY` is defined in inventories/development/default.yml:20
as `"{{ lookup('env', 'INFINITO_DOMAIN') }}"`. On Ansible 2.18+ that
string carries a `TrustedAsTemplate` tag (it is an
`_AnsibleTaggedStr`). The constructor at tasks/stages/01_constructor.yml:64
wrapped it in `[ DOMAIN_PRIMARY ] | flatten | map('string') | first`
— the explicit `str()` cast strips the trust tag. The resulting plain
string flowed into `CURRENT_PLAY_REDIRECT_DOMAINS.source` as the
literal text `{{ lookup('env', 'INFINITO_DOMAIN') }}`, was passed
through `domain: "{{ item.source }}"` in
roles/web-opt-rdr-domains/tasks/main.yml:14 (one render pass that
produces untrusted content), and finally reached `as_str(terms[0])`
in plugins/lookup/tls.py:39 unchanged. The tls plugin never templated
its arg, so `resolve_term` looked up the literal Jinja string in the
domains mapping and raised.
Two layers, both necessary:
1. plugins/lookup/tls.py:39 — defensively template `terms[0]` through
`self._templar` before string-coercion. Mirrors the idiom in
plugins/lookup/unit_name.py:33 and plugins/lookup/email.py:61. Any
future caller that hands the plugin an unresolved expression now
gets it resolved before the domains-map lookup.
2. tasks/stages/01_constructor.yml:64-65 — drop the
`[ X ] | flatten | map('string') | first` ritual for
`DOMAIN_PRIMARY` and `DOMAIN_HOMEPAGE`. Both are scalar strings in
every known inventory (group_vars/all/00_general.yml:41,43 +
inventories/development/default.yml:20); the over-defensive
wrapper was the trust-tag stripper. Direct reference preserves the
tag through `CURRENT_PLAY_REDIRECT_DOMAINS` and every downstream
consumer.
Downstream consumers audited; all transparent to `_AnsibleTaggedStr`
(it inherits from `str`):
- merge_mapping filter: content-based dict-key compare
- domain_mappings filter: takes `DOMAIN_PRIMARY` directly, no cast
- items2dict(key_name='source', value_name='source'): hashable
- generate_all_domains filter: `isinstance(item, str)` matches
- sys-stk-front-base `when: ... | length > 0`: only counts
- web-opt-rdr-domains -> tls lookup (the failing path): now templates
Coverage: new `test_term_with_unresolved_jinja_is_templated` in
tests/unit/plugins/lookup/test_tls.py verifies the lookup feeds its
first term through the templar. The 12 existing tls tests stay
green. `make test` green across all 5 targets.
Class follow-up (not addressed here): the
`| string` / `| map('string')` cast still strips `TrustedAsTemplate`
elsewhere if a future commit reintroduces it; 12 other lookup
plugins do not yet defensively template their args. A lint guard and
plugin-wide audit will come in separate PRs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
After 10cb02461 switched `payload._resolve_variant_payloads` from
`get_variants` to `get_variant_overrides_only`, matrix-deploy bakes
only the raw `meta/variants.yml` slice into host_vars (typically just
`services:`). Lookups that read `variables['applications']` raw no
longer see content from `meta/rbac.yml`, `meta/server.yml`, etc.
`rbac_group_path` was the last lookup still going through the raw
host_vars read. For joomla (whose `meta/rbac.yml` declares the
`editor` role) the post-10cb02461 host_vars carried no `rbac` block
at all, the lookup walked an empty `rbac.roles` map, and the deploy
crashed with "rbac_group_path: role 'editor' is not declared under
applications[web-app-joomla].rbac.roles. Declared roles:
['administrator']" (job 77797371433 on run 26428080957).
Route `_get_application` through `get_merged_applications` so the
meta-merged view (services + server + rbac + volumes + schema
credentials + variant overrides) is restored. This mirrors the
existing pattern in plugins/lookup/{config,tls,nginx,domain}.py and
keeps the mirror-override intent of 10cb02461 intact: the bake step
still emits only the variant overrides; the merged cache assembles
the full view at lookup time.
Test updates:
- TestRbacGroupPathLookup.setUp patches get_merged_applications to
read from variables['applications'] so the existing 14 tests stay
hermetic without disk-reading roles/*/meta/*.
- New TestRbacGroupPathReadsMergedApplications regression case
exercises the exact 10cb02461 scenario: host_vars only carry a
variants.yml slice, the merged cache supplies the rbac block, the
lookup returns the correct path.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- web-opt-rdr-www canonical: rdr-www.{DOMAIN_PRIMARY} -> w3redirect.{DOMAIN_PRIMARY}.
- New lookup `www_redirect_domains` (role-scoped): returns full {source, target}
mapping list, resolves DOMAIN_HOMEPAGE target for the role's own canonical via
lookup('domain', 'web-opt-rdr-www'). REDIRECT_WWW_DOMAINS now carries dicts;
tasks/main.yml and _01_cloudflare_edge_redirect.yml updated to consume dicts.
- New lookup `current_play_redirect_domains`: replaces the two inline
set_fact merges in tasks/stages/01_constructor.yml. Merges user-supplied
redirect_domain_mappings, primary-redirect (conditional on web-opt-rdr-domains
in deployed), and per-app domain mappings (inlined from the now-removed
domain_mappings filter).
- domain_redirect_mappings filter removed; web-app-keycloak RedirectUrisTest
pollution of utils.roles.applications.config in sys.modules fixed via
tearDownClass cleanup.
- csp_skip_domains lookup: csp.health.skip opt-out path removed (architecture
no longer needs it).
- Lint test_only_web_app_roles_integrate_dashboard expanded from web-app-* to
web-*; test_role_and_group_vars_used now scans .py files too so vars consumed
only inside plugins (DOMAIN_HOMEPAGE, AUTO_BUILD_ALIASES) are seen.
- New role-scoped variants.yml on web-opt-rdr-www with dashboard on/off.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Eliminate include-style depends_on assembly. The lookup resolves DB/redis
inductively from applications config and returns indented YAML (or "" when
no entries), removing the empty `depends_on:` block bug in libretranslate.
Migrates 43 compose templates from `{% include … dmbs_*.yml.j2 %}` to
`{{ lookup('container_depends_on', application_id) }}`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`status` was too vague for what the lookup returns (apps in current
play). Rename to `deployment` and add two new fields:
* `runtime` — RUNTIME variable value
* `deployed` — runtime-aware authoritative set: equals `running`
(whitelist or groups) for ephemeral CI runtimes (act, github),
falls back to `groups` for persistent runtimes (host, dev). This
matches reality: in CI the container is fresh so only whitelisted
apps exist; in production a partial deploy still leaves prior
apps live, so groups is the right set.
Switch the two webserver/csp healthchecks to `lookup('deployment').deployed`
so they only probe domains for apps that actually exist in the current
deploy. In production behaviour is unchanged.
Update remaining consumers (cli/build/include, web-app-keycloak,
test-e2e-playwright) for the rename. Existing `status` semantics
(`whitelist`, `running`, `groups`, `all`) preserved under the new
name.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fixes Cluster 3a ca-inject YAML parse: when a role variant disables its
DB (enabled=false), the depends_on template emitted ` :` because
the existing `not shared` guard fired for a non-existing DB host.
The new derived flag mirrors `service.required` (= enabled AND shared)
and is consumed in:
- depends_on/dbms_base.yml.j2, dmbs_excl.yml.j2 (the actual fix)
- sys-svc-rdbms postgres/mariadb service templates
- web-app-discourse, web-app-snipe-it (latent same-pattern defensives)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements docs/requirements/021-sso-flavor-migration.md atomically:
* Schema: every role's meta/services.yml replaces the parallel
`oauth2:` / `oidc:` top-level blocks with a single `sso:` block
carrying `enabled`, `shared`, and a `flavor` discriminator
(`oidc` | `oauth2` | `saml`; default `oidc`). Flavor-specific
sub-keys live under `sso.<flavor>.*` (`sso.oauth2.{origin,acl,
allowed_groups}`, `sso.oidc.plugin`).
* Provider role: roles/web-app-oauth2-proxy/ is deleted in its
entirety; its 5 sidecar templates move under
roles/web-app-keycloak/templates/sso_proxy/, and the per-consumer
rendering task lives at roles/web-app-keycloak/tasks/sso_proxy.yml.
web-app-keycloak now declares `provides: sso`.
* Runtime: new resolver utils/roles/applications/services/sso.py
centralises the SSO predicates (`is_enabled`, `is_proxy_gated`,
`is_oidc_native`) and the oauth2 sub-values
(`oauth2_origin_{host,port}`, `oauth2_acl`, `oauth2_allowed_groups`).
Three lookup plugins consume the resolver: `sso` (single-app
predicate access), `sso_proxy_consumers` (enumerates oauth2-flavored
consumers), `sso_oidc_plugin` (renamed from `oidc_flavor`, Nextcloud
plugin selector). sys-stk-backend, sys-svc-compose, sys-svc-proxy
and plugins/filter/compose_volumes.py all gate on
`lookup('sso', ..., 'is_proxy_gated')` instead of manually combining
`enabled` and `flavor`.
* Identifier renames repo-wide: `services.<entity>.ports.local.oauth2`
→ `ports.local.sso`; `PORT_BANDS.local.oauth2` → `PORT_BANDS.local.sso`;
`OAUTH2_PROXY_*` env/Jinja vars → `SSO_PROXY_*`;
`oauth2_proxy_cookie_secret` → `sso_proxy_cookie_secret`;
`OAUTH2_SERVICE_ENABLED` + `OIDC_SERVICE_ENABLED` collapse into
`SSO_SERVICE_ENABLED`; Playwright gating helpers
(isServiceEnabled/skipUnlessServiceEnabled/requireService/safeIsEnabled/
safeSkipUnlessEnabled) move from `oidc`/`oauth2` to `sso`.
* Dual-block resolution: per req's Background-Table, web-app-bookwyrm
is `flavor: oauth2`; web-app-gitea and web-app-friendica keep
`flavor: oauth2` with their existing `oauth2.acl.blacklist`
(/user/login) and `oauth2.acl.whitelist` (federation + discovery
endpoints) — preserving the operative oauth2-proxy gating.
* Tests: the two mutual-exclusion guards
(tests/integration/iam/oauth2_oidc/test_mutual_exclusive.py and
test_acl_mutual_exclusive.py) are deleted per Decision #10;
test_oauth2_contract.py is renamed and rewritten to
test_sso_contract.py, asserting that `flavor: oauth2` roles carry
`oauth2.origin.{host,port}` plus `ports.local.sso`; test_sso.py
(web-app integration) now requires the unified `sso:` block with an
explicit flavor or a `# nocheck: sso` opt-out; new
tests/lint/repository/no_legacy_sso_paths/ guards the source tree
against re-introducing the legacy strings.
* Docs: req 21 itself has every AC checked off, and reqs 006, 008,
009, 013, 015, 016, 019, 020 plus 10+ role READMEs and
docs/contributing/actions/testing/suppression.md are updated to the
unified SSO naming.
Empirically validated end-to-end against the validation app set
(web-app-keycloak, web-app-nextcloud, web-app-bookwyrm, web-app-gitea,
web-app-friendica, web-app-dashboard) in matrix variants 0 and 1; v2
also passes the 4 target roles' Playwright after a documented
pre-existing KC_HOSTNAME-drift workaround (separate concern, not
caused by this refactor).
BREAKING CHANGE: every consumer of `services.oauth2.*`,
`services.oidc.*`, `OAUTH2_PROXY_*`, `OAUTH2_SERVICE_ENABLED`,
`OIDC_SERVICE_ENABLED`, `oauth2_proxy_cookie_secret`,
`oauth2_proxy_application_id`, `ports.local.oauth2`, the
`web-app-oauth2-proxy` role, or the legacy gating helper service names
(`oidc`/`oauth2`) must migrate to the unified `sso` shape documented
in docs/requirements/021-sso-flavor-migration.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Move role_is_invokable / role_has_required_by / role_has_nocheck out
of plugins/filter (where they were flagged as test-only filters) into
utils/roles/required_by_coverage.py. All three use load_yaml_any,
read_text, and ROLE_FILE_META_SERVICES from the canonical SPOTs.
- Refactor the verifier (cli.meta.roles.services.called) the same way:
iterate roles/*/ directly, look up meta/services.yml via the constant,
parse with load_yaml_any, and tolerate a missing roles_dir (unit tests
point at a fake /repo).
- Make the mailu-dependency integration test yaml-aware for meta/
services.yml: walk the parsed tree and skip values inside any
`required_by` subtree. `required_by` is inverse coverage metadata for
the deploy verifier, not a "consumer of mailu" reference, so previous
regex-on-text matching produced false positives for cert-deploy.
- sys-ctl-mtn-cert-renew is gated by `lookup('tls', application_id,
'mode') == 'letsencrypt'`, which the verifier can't express via
role-id/category match — mark nocheck with the reason inline.
- Add README.md for cli/meta/roles/services (CLI-category convention).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a static lint that hard-fails when a non-invokable role's
meta/services.yml declares neither `required_by` nor `nocheck:
required-by-coverage`. This protects the runtime verifier
(cli.meta.roles.services.called) from silently losing coverage when new
infrastructure roles ship without a coverage contract.
New filter plugins (reusable from Ansible + Python tests):
- role_is_invokable: walks categories.yml; true if any node on the
role's category path is `invokable: true`.
- role_has_required_by: true if any entity in meta/services.yml declares
a non-empty `required_by.categories` or `required_by.roles`.
- role_has_nocheck: true if meta/services.yml carries
`# nocheck: <check_id>` (whole-word match).
New test: tests/lint/ansible/roles/meta/test_required_by_for_non_invokable.py
fails CI with a per-file list and a fix recipe (declare required_by OR add
the nocheck marker — both already shipped in Bundle B and Bundle D).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`lookup('image', '<role>', '<svc>', 'ref')` only consulted
`images_overrides.*` plus role defaults, which the mirror pipeline
never populates — the pipeline writes into
`applications.<role>.services.<svc>.{image,version}`. As a result,
`sys-svc-container`'s `BUSYBOX_IMAGE` / `NODE_IMAGE`,
`sys-ctl-hlth-csp`'s `HEALTH_CSP_IMAGE`, and `test-e2e-playwright`'s
`TEST_E2E_PLAYWRIGHT_IMAGE` always resolved to the raw upstream ref
and hit Docker Hub directly. In CI this surfaced as
`toomanyrequests: ...unauthenticated pull rate limit` on
`busybox:1.37`.
Switches every productive `lookup('image', ...)` to
`lookup('config', '<role>', 'services.<svc>.image|version')` — the
same path templates like `web-app-oauth2-proxy` already use, which
resolves through the merged `applications.*` map and so transparently
picks up the GHCR mirror override written by the inventory creator.
Removes the now-unused `plugins/lookup/image.py` plugin and its unit
test (zero remaining consumers across roles, scripts, and tests).
Rewrites the image-origin / image-mirror / dockerfile docs to teach
the `lookup('config', ...)` pattern, dropping references to the
deleted plugin and the dead `images_overrides.*` variable. Doc edits
follow the `documentation.md` rules (no em dashes, one sentence per
line, no historical phrasing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Generalises the postgres image build to be driven by a declarative
extension→library registry instead of a hardcoded POSTGRES_VECTOR_ENABLED
flag. Same Dockerfile.j2 now serves both the central svc-db-postgres
image (libraries aggregated across all deployed consumers) and per-
consumer images for shared:false postgres consumers whose extensions
need libraries not in the postgis/postgis base (e.g. pgvector).
postgres.yml.j2 switches between image: and build: based on whether
custom libraries are needed. New postgres_libraries lookup filters
base-image extensions out and returns recipes for the rest. Dedicated
task path extracted into dedicated.yml + dedicated_postgres.yml so the
block-with-when size limit stays satisfied and the postgres-type check
happens once.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
sys-svc-rdbms now renders an extensions.sql into a per-consumer
.initdb.d/ directory and bind-mounts it into the dedicated postgres
container at /docker-entrypoint-initdb.d/. CREATE EXTENSION runs as
POSTGRES_USER (= bootstrap superuser) on first initdb so non-trusted
extensions succeed. Same declarative source (service_extensions
lookup, services.yml extensions field) drives both shared:true
(02_init.yml loop) and shared:false (initdb.d) paths. database lookup
exposes a new initdb_dir field; service_extensions lookup loses its
shared:true guard (shared:false is now a real path). Refactored
repeated lookups in sys-svc-rdbms tasks into vars/main.yml.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each consumer (bookwyrm/mobilizon/discourse) now declares its required
postgres extensions in its own meta/services.yml under postgres.extensions
and pins shared: true. svc-db-postgres' 02_init.yml drops the hardcoded
postgis-gate and unconditional vector steps and loops over the new
lookup keyed by database_consumer_id. The lookup fast-fails if a role
declares extensions without pinning shared: true.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
OpenResty's inline *_by_lua_block parser is the nginx config parser; it
can't reliably handle Lua long brackets ([=[...]=]) embedded in large
rendered snippets. The prometheus role's javascript.js.j2 (logout-icon
DOM-mutation script) collapses to a ~3.4 KB one-liner wrapped in
head_snippets[#... + 1] = [=[<huge_JS>]=], which trips the parser with
"missing the closing long bracket "]=]", the inlined Lua code may be too
long" — the wording is OpenResty's own hint to switch to the file form.
Rendering the body_filter as an external per-app Lua file routes through
Lua's native parser instead, eliminating the limit.
- plugins/lookup/nginx.py: expose <conf_dir>/lua/ as
directories.configuration.lua + add to directories.ensure so
sys-svc-webserver-core auto-creates it
- roles/sys-front-inj-all/templates/body_filter.lua.j2: new — pure Lua
body (formerly inside body_filter_by_lua_block { ... }) plus the
push_snippets Jinja macro
- roles/sys-front-inj-all/templates/location.lua.j2: replace inline
body_filter_by_lua_block with body_filter_by_lua_file pointing at the
per-app .lua
- roles/sys-front-inj-all/tasks/main.yml: render the per-app Lua file
after the feature include_roles (those set the *_code_one_liner vars
the sub.j2 templates depend on); notify restart openresty
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Reason: the lookup only consulted services.ldap.enabled, so even with OIDC
fully disabled it would still return "sociallogin" (or "oidc_login" when
LDAP was on). Nextcloud's services.yml then enables the matching OIDC
plugin, which redirects to Keycloak — but the Keycloak client's
redirectUris correctly excludes apps with oidc.enabled=false, so the
handoff dies with "Invalid parameter: redirect_uri". Caused all 3
nextcloud variant-1 Playwright errors (`nextcloud login`,
`nextcloud talk admin settings`, `biber logs into nextcloud via OIDC`).
The lookup now short-circuits to "" when oidc.enabled is falsy, which
makes all three plugin gates (oidc_login, sociallogin, user_oidc) compare
false and 06_setup_plugin.yml's disable branch take over. Existing
flavor-selection paths (oidc.enabled=true) are unchanged. Adds two
regression tests covering the new short-circuit and updates the
`_apps` test helper to default oidc.enabled=true so existing
flavor-selection tests stay focused on the ldap/flavor axis.
