Long-string continuation breaks introduced by `make autoformat` on
the friendica addon-sync script command and the matomo bootstrap
failure message. Pure cosmetic, no semantic change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Strip the entire mailu_bootstrap_failed soft-skip pattern from the
mariadb/redis/admin readiness waits. Per the project's hard-fail-when-
upstream-broken convention, the role MUST surface upstream brokenness
loudly rather than mark a flag and skip downstream ops. The known
swarm gap (mailu admin's compose-only DNSSEC + per-service `dns:`
ipv4_address pin) is now documented in the fail message as debugging
context for the operator, not as a justification for silent skip.
Changes:
- 02_wait_for_mariadb.yml: drop file-header soft-skip rationale, restore
nested block/always rescue (log-collect + ansible.builtin.fail) in
both rescues, fix swarm-broken `container exec -T` -> `-i`, drop
outer-block `when: not mailu_bootstrap_failed` gate.
- 03_wait_for_redis.yml: mirror the same nested block/always rescue in
both Redis ping and admin HTTP probes, fix -T -> -i.
- 02b_post_bootstrap_ops.yml (new): host the post-bootstrap chain
(redis wait, user create, cert deploy, DKIM, DNS records) without
any mailu_bootstrap_failed gate.
- 02_manager_ops.yml: drop the coarse include-level gate; the include
always runs and downstream failures propagate.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The `container` wrapper is a thin passthrough to `docker exec`, which
accepts `-i`/`-t` but NOT `-T` (that flag exists only on
`docker compose exec`). Every probe shipped in 1ededacd6 errored with
rc=125 `unknown shorthand flag: 'T' in -T` before reaching the
container, burning the full 30x5s `until: rc == 0` budget on a CLI
parse error.
Affects user/token management and DKIM generation on both compose and
swarm. Sibling files (02_wait_for_mariadb.yml, 03_wait_for_redis.yml)
carry the same -T -> -i fix as part of the next commit's structural
cleanup.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Rewrite compose-only `BIN_COMPOSE exec -T <svc>` invocations in
04_manage_user.yml, 04a_manage_user_token.yml, 04b_create_user_token.yml,
and 05_generate-and-read-dkim.yml to the swarm-aware form
`container exec -i {{ MAILU_<SVC>_CONTAINER_ADDRESS | quote }} ...`,
switch `command:` modules to `ansible.builtin.shell:` where the new exec
form requires it, drop the now-redundant `args.chdir: {{ MAILU_INSTANCE_DIR }}`
(container exec runs inside the container fs), and quote interpolated
arguments. No semantic changes: every register/changed_when/failed_when/
no_log/retries/when stays byte-identical.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Restructures the keycloak role so manager-side operations run on the
hosting node under both deployment models without per-task delegation
noise:
- tasks/02_manager_ops.yml: extracts the manager-side body into
02a_manager_ops_on_host.yml and includes it once with
`apply: delegate_to: KEYCLOAK_HOST_NODE`. Ansible's `apply:` keyword
propagates the delegate to every task in the included file so each
step (kcadm login, client / mail / RBAC / LDAP updates) runs on the
stack host without the inner files needing to know about it.
- tasks/04_login.yml: dual-path discovery. Swarm waits for the
keycloak service to converge via `container service ps`, resolves
the hosting node, and sets KEYCLOAK_HOST_NODE; compose sets it to
inventory_hostname (single-host model). Subsequent LOCAL_CID
resolution is now gated `when: DEPLOYMENT_MODE == 'swarm'` and
scoped to a single block; compose leaves KEYCLOAK_LOCAL_CID unset
and relies on vars/main.yml's
`KEYCLOAK_EXEC_CONTAINER: container exec -i {{ KEYCLOAK_LOCAL_CID
| default(KEYCLOAK_CONTAINER_ADDRESS) }}` fallback. Removes the
brittle inline shell-template else-branch that previously echoed
KEYCLOAK_CONTAINER for compose.
- tasks/05a_realm_import_run.yml: compose path runs the import
in-place via `compose down/up/run`; swarm path dispatches to a new
05b_realm_import_swarm.yml that scales the import service via
`container service update`, tails its logs, and rescales to 0. The
3 mode-gated tasks share a single `when:` wrapper to satisfy the
ansible-repeated-when-run lint.
- tasks/update/05_ldap_shared.yml (new): extracted from the previous
inline LDAP-shared block in the manager-ops handler so the
surrounding block stays under the >3-children limit. Combined
when-guard (KEYCLOAK_LDAP_ENABLED AND KEYCLOAK_LDAP_SHARED) lives
on the include.
- tasks/assert/05_ldap.yml: read-only assertion path; uses
KEYCLOAK_EXEC_KCADM (mode-agnostic).
- handlers/main.yml: drops the comment block that restated
`ignore_errors: true` on the sync-ldap-groups handler. The
ignore_errors + delegate_to + transient-failure semantics remain
intact; the prose was restating the directive, not naming a
trip-wire.
Verified internally coherent + compose-mode-safe via static audit:
include chain main -> 01_core -> 02_manager_ops -> 02a -> {04_login,
05_realm_import -> 05a (compose-branch) | 05b (swarm-branch),
update/{01_client, 02_mail_realm, 03_mail_master,
04_rbac_client_scope, 05_ldap_shared}, assert/05_ldap}. All
include_tasks targets resolve, all KEYCLOAK_* vars are set before
consumption, all kcadm/update/assert tasks are mode-agnostic.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Splits the matomo install path into compose-mode and swarm-mode
manager-side flows so the role works under both deployment models:
- 03_manager_ops.yml: dual `when: DEPLOYMENT_MODE == 'swarm'` /
`\!= 'swarm'` blocks discover the matomo container differently
(`container service ps` for swarm to wait for task convergence +
resolve the hosting node; `container inspect` for compose). Both
blocks set MATOMO_HOST_NODE so the downstream steps `delegate_to:`
the correct node. Container-ID resolution then takes the right
branch via an inline Jinja toggle.
- 05_bootstrap.yml: wraps the compose and swarm bootstrap-launch
tasks in a single `block:` with a paired `rescue:` that loads
06_bootstrap_debug.yml (only under MODE_DEBUG) and then hard-fails
with the captured rc/stdout/stderr from the failed run. Drops the
redundant `| default('')` on `matomo_bootstrap_run.stdout` inside
the swarm `until:` (the lint catches it; the attribute is always
set on a registered task).
- 04_install_path.yml: removes the stale `when: matomo_token_value …`
gate on the api-config include that referenced the deleted
silent-skip rescue. 09_config.yml now runs unconditionally; the
legitimate token-shape assertions live in 05_bootstrap.yml.
- files/patch_trusted_hosts.sh: adds the missing `#\!/bin/sh` shebang
so lint-shellcheck stops flagging SC2148.
Plus a lint bugfix:
- tests/lint/ansible/structure/test_no_oversized_ansible_block.py
was counting `rescue:` / `always:` siblings as direct block
children, double-counting them under the >3 limit. Walker now
stops at `rescue:`/`always:` keys aligned with the `block:`
keyword (each section has its own child list), and computes the
terminator column from the `block:` keyword itself instead of the
regex-captured indent (which is 0 for the `- block:` form because
the dash absorbs into the prefix group).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- tasks/01_core.yml: gate the 03_setup include with
`when: IS_STACK_HOST | bool` so the bootstrap procedure only runs on
the node that hosts the compose project / swarm-stack manager. Worker
nodes skip cleanly.
- tasks/03_setup.yml + tasks/04_wait.yml: replace BIN_COMPOSE-based
exec / chdir patterns with `container exec -i {{ <SVC>_CONTAINER_ADDRESS
| quote }}` so the per-task target resolves correctly under both
compose (single-container name) and swarm (stack-task-id-suffixed
name).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds two structural ansible lints under tests/lint/ansible/structure/:
- ansible-block-oversized: forbid `- block:` constructs with more than
3 direct child tasks. Big block bodies hide intent, accumulate
unrelated guards on the wrapper, and signal that the body wants to
be its own include_tasks sub-file. Extraction keeps the parent
scannable; the sub-file MAY carry the `# include-gated:
when: …` header marker so swarm-compat lints exempt it.
- ansible-repeated-when-run: forbid 3+ consecutive sibling tasks
carrying the SAME `when:` expression. Copy-pasted guards drift,
silently disable each other over time, and read worse than a single
`- name: …` + `when: <expr>` + `block:` wrapper. Per-line opt-out
for runs where wrapping would alter semantics (cross-task register
dependencies).
Both lints use the line-based scanner pattern from existing
tests/lint/ansible/jinja/* + `is_suppressed_at` from
utils.annotations.suppress for the per-line nocheck markers.
Refactors five roles into compliance:
- svc-prx-openresty/tasks/02_assert.yml: heavy rescue body extracted
to 02a_assert_rescue_debug.yml.
- test-e2e-playwright/tasks/01_setup.yml: 3 install-needed siblings
wrapped in a single when-block.
- web-app-erpnext/tasks/02_bench_bootstrap.yml: 3 site-probe-failed
siblings wrapped likewise.
- web-app-odoo/tasks/02_install_modules.yml: 3 OIDC-enabled siblings
wrapped. tasks/03_oidc.yml: dropped redundant single-task wrapper
block (the original wrapper carried name only and would have
required extraction for size — inlining the 23 tasks at file root
keeps the file purpose-narrow and removes one layer of indirection).
- web-app-xwiki/tasks/04_extensions.yml: large block body + rescue
body both extracted to 04a_extensions_rescue.yml +
04b_extensions_execute.yml.
Mixed-scope offenders (web-app-keycloak/tasks/{02_manager_ops,05a,
assert/05_ldap,update/05_ldap_shared,02a} and
web-app-matomo/tasks/{03_manager_ops,05_bootstrap,05a}) deliberately
stay unstaged; they ship with their respective swarm-aware
restructure commits where they belong by topic, not bundled with the
structural-lint cleanup.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Refactors four roles that previously used `# nocheck: compose-chdir-in-task` /
`compose-verb-in-task` / `container-exec-resolver` markers to follow the
canonical swarm-aware pattern:
- web-app-bigbluebutton: adds BBB_POSTGRES_CONTAINER_ADDRESS /
BBB_GREENLIGHT_CONTAINER_ADDRESS vars; 02a_postgres_password_sync.yml +
03_administrator.yml use `container exec -i {{ ..._CONTAINER_ADDRESS | quote }}`;
02_compose.yml's compose-build-phase tasks (generate-compose +
prebuild etherpad) move into the new
02b_bootstrap_compose_project.yml (>3 entries cannot live in a block)
under a single `when: DEPLOYMENT_MODE \!= "swarm"` include guard.
- web-app-baserow: BASEROW_APP_CONTAINER_ADDRESS var + 02_manager_ops.yml
converted to container exec + container restart canonical pattern.
- web-app-matomo: MATOMO_APP_CONTAINER_ADDRESS var; 07_patch_database.yml
swaps MATOMO_LOCAL_CID for the proper resolver-backed address.
- web-app-matrix: 01_docker.yml's two consecutive `compose pull` /
`compose up` tasks extract into 03_compose_pull_and_up.yml (with the new
`# include-gated: when: DEPLOYMENT_MODE \!= "swarm"` header marker) and
the previous 03_webserver.yml renames to 04_webserver.yml to keep the
numeric file sequence contiguous.
Supporting lint infrastructure changes so file-level include-gated
sub-task lists survive the swarm-compat lints:
- utils/annotations/task_gate.py: `is_task_compose_only_gated` now walks
up nested parent blocks (a single `when: DEPLOYMENT_MODE \!= 'swarm'`
on `- block:` exempts every child task underneath) and a new
`is_file_compose_only_by_header` recognises files declaring
`# include-gated: when: DEPLOYMENT_MODE \!= "swarm"` in their first
five lines as compose-only-by-include.
- tests/lint/ansible/shell/test_no_compose_chdir_in_shell_tasks.py and
test_no_compose_verb_in_shell_tasks.py call the header helper before
the per-line scan, so include-only sub-files are exempt.
- tests/unit/utils/annotations/test_task_gate.py covers both new
behaviours (recursive parent walk plus file header) end-to-end.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace compose-mode-specific `{{ BIN_COMPOSE }} exec ... {{ SERVICE }}`
+ `args.chdir: lookup('container', ..., 'directories.instance')`
invocations with `container exec -i {{ ..._CONTAINER_ADDRESS | quote }}`
(plus `--workdir` flag where chdir was load-bearing). The container
exec wrapper resolves the actual runtime container name on both
compose (compose-suffixed name) and swarm (stack-task-id-suffixed
name) via the `container_address` lookup, so the task no longer
assumes the compose-mode service-name address.
Pure-syntax adaptation only — no behavioural change, same register /
when / failed_when / no_log semantics preserved on every task.
Roles touched (12 task files):
- web-app-espocrm (1)
- web-app-friendica (2)
- web-app-listmonk (1)
- web-app-openproject (2)
- web-app-snipe-it (3)
- web-app-taiga (1)
- web-app-wordpress (2)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The hard / soft container-repair roles run `docker compose restart` on
unhealthy containers. That is compose-mode semantics: in swarm-mode
container health is restored via `docker service update` against the
service spec, not by re-issuing compose commands on individual
containers. Adding the role-level guard skips the include cleanly on
every swarm node (incl. the manager) rather than letting the script
spam errors about a missing compose directory or absent engine.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a strict lint that flags every `type: bind` entry in
roles/<role>/meta/volumes.yml whose source or any mount target points
to a single FILE (path ends in a recognised file extension or a
Jinja var with a file-suffix name like *_CONF/*_FILE). Single-file
binds are swarm-fragile: the source path is only present on the node
where the file was rendered, so swarm rejects the task as soon as it
lands on a different node.
Fix is `type: config`: the manager loads the file at `stack deploy`
time and swarm distributes it through raft to every node.
compose-mode honours the same canonical entry (the compose_volumes
filter renders it as a top-level `configs:` block; container_volumes
emits the per-service reference), so the role stays mode-agnostic.
Per-line opt-out: `# nocheck: meta-volume-file-bind-should-be-config`
with a short `# Reason:` for legitimate exceptions (sockets, kernel
interfaces, runtime-rotated files, multi-MB blobs, host writebacks).
This commit migrates 23 file-bind entries across 16 roles to
type:config: web-app-flowise, friendica, kix (2), listmonk, mailu,
matrix (2), mediawiki, mobilizon, nextcloud, pgadmin, pixelfed,
prometheus (4), shopware (2), suitecrm, taiga (2), zammad. All gain a
default mode:"0440"; mount-level read_only:true is dropped (configs
are read-only by default and the schema forbids the field for
type:config).
Updates the prometheus role-wiring test to accept both type:bind and
type:config for the alert_rules mount probe.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Migrate every role's meta/volumes.yml from ad-hoc dict-of-named-volumes
to a canonical dict-of-dicts schema where the YAML key is the semantic
short name and the entry body carries type / name / source / mounts /
nfs / mode / read_only. The optional 'name:' field is the docker volume
name (defaults to the key); the previous 'docker_name:' field is gone.
What's in this commit (210 paths):
Infrastructure (23)
- New filter plugins/filter/{compose,container}_volumes.py — render the
top-level volumes:/configs:/secrets: section and per-service mount
blocks from the canonical meta. container_volumes uses the SEMANTIC
name as per-service mount source so it resolves against compose_volumes'
top-level dict key (docker_name as source produced 'undefined volume'
on swarm).
- New lookup plugins/lookup/{compose,container,volume}_volumes.py.
volume(application_id, name) returns a normalised single entry;
volume(application_id) returns the whole canonical dict.
- utils/cache/applications.py keeps a sibling registry
_CANONICAL_VOLUMES_BY_ROLE outside the merged applications payload so
raw Jinja source: strings never enter the templar walk (root-cause of
the 200x cache-miss + recursive get_merged_applications storm).
- utils/roles/applications/mounts.py — schema validator + normaliser;
list-shape is rejected.
- plugins/lookup/nginx.py: _volume_path_by_name reads canonical directly.
- scripts/tests/deploy/swarm/_context.sh: NFS_VOLUMES extraction handles
both legacy dict and canonical dict shapes.
- roles/svc-prx-openresty/handlers/restart_and_validate.yml: gate
compose-only validate+reload behind a single when-block; swarm uses
service-update instead (bind-mounts of a FILE pin source inode at
task-create time, so atomic write-tmp+rename leaves the running task
on a deleted inode).
- roles/sys-svc-compose/tasks/utils/nfs_prep.yml: switch to
lookup('volume', application_id) | dict2items so NFS subdir
pre-creation works under the new cache split.
Tests (12)
- New strict lints flagging inline mount sections in compose templates,
single-file bind mounts, and legacy lookup('config', ..., 'volumes.X.Y')
call sites — all volume-mount surfaces MUST flow through container_volumes
/ compose_volumes / volume lookups.
- Integration: test_volumes_compose_chain + test_volume_constants_are_lookups
accept both the legacy and the new lookup('volume', app, 'X').name forms
during the migration window.
- Unit tests for the new lookup, filter, mounts helper, plus updated
nginx lookup test.
- Prometheus role-wiring test reads canonical meta for the
alert_rules.yml bind mount.
Role migrations (175)
- 62 meta/volumes.yml converted to dict-of-dicts canonical shape
- 63 templates/compose*.yml.j2 replace inline volumes:/configs:/secrets:
with lookup('container_volumes', application_id, '<service>') |
indent(N) per service + a single trailing lookup('compose_volumes',
application_id) for the top-level block
- 50 vars/main.yml drop dead volume constants and refactor 23 remaining
legacy lookup('config', ..., 'volumes.X.Y') to lookup('volume',
application_id, 'X').Y across 9 roles
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a "Command output logging" section instructing the agent to tee
every non-trivial run (tests, deploys, long pipelines) into a /tmp file
with a monotonically increasing slug index, and to grep that saved log
instead of re-running the command. Reason: re-running make test to find
a failure that scrolled out of context costs ~2 min per cycle; grepping
the saved log costs milliseconds. The rule is unconditional — agents
applied it ad-hoc before; making it explicit prevents the regression.
`DIR_COMPOSITIONS` resolves to a Jinja expression in swarm mode
(`{{ DIR_VAR_LIB ~ "/compose/" if DEPLOYMENT_MODE == "swarm" else "/opt/compose/" }}`),
but Ansible passes variables un-templated into lookups. container, database,
and compose_file_args now call `templar.template()` when the raw value still
contains `{{`, so the resolved path reaches `get_docker_paths()`.
email's `_render` switches to `_templar_render_best_effort` (TrustedAsTemplate
for Ansible 2.19+) and loops up to 4 passes: nested inventory expressions like
`DOMAIN_PRIMARY="{{ lookup('env','INFINITO_DOMAIN') }}"` need multiple
substitution rounds before reaching the literal domain.
Drops a stale `import textwrap` in test_depends_on.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds `act-swarm-zombie app=<id>` which runs the swarm matrix-app test and leaves
the DinD cluster + lab network alive afterwards (INFINITO_KEEP_SWARM_NODES=true
forwarded via the new ACT_ENV passthrough in workflow.sh) so failures can be
post-mortem inspected with `make act-swarm-exec node=… cmd=…`,
`make act-swarm-shell node=…`, and finally released with `make act-swarm-down`.
Consolidates the swarm-test topology (manager / worker / nfs-server names, lab
network, NFS export base) into scripts/tests/deploy/swarm/topology.sh as the
single source of truth: 13_cleanup.sh and shell_node.sh source it directly,
the GitHub Actions workflow ingests it into $GITHUB_ENV via a setup step,
replacing the duplicate env: block entries.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The post-migration restart step has two branches gated on
DEPLOYMENT_MODE. Tighten both:
- Compose branch: keeps the `compose restart …` call, but the
lint suite's swarm-compat audit (rules `compose-verb-in-task`
and `compose-chdir-in-task`) flags it as a compose-only verb /
chdir. The branch IS compose-only by design -- the swarm
branch below carries the equivalent action. Add per-line
`# nocheck:` markers that name the rule and explain *why* the
compose call is legitimate here (so a future reader does not
silence the lint without understanding it).
- Swarm branch: was calling `docker service update --force`
directly with a hand-spelled `<entity-name>_<service-name>`,
re-implementing the swarm naming convention inline. Replace
with the `container` wrapper + `container_service` lookup so
the swarm service name comes from the same single source that
every other swarm-side task uses (and so the recent
`container_service` swarm-naming fix in commit 0a852e5b5 -- which
switched to `<stack>_<service-key>` -- automatically applies
here).
Behaviourally identical for the existing single-host case;
correctness improvement for multi-stack swarm clusters where the
old inline spelling could shadow the entity_name resolution path.
The mapping from compose-style `docker_restart_policy` (`no`,
`on-failure[:N]`, `always`, `unless-stopped`) to swarm's
`deploy.restart_policy.condition` (`none`, `on-failure`, `any`) was
seven lines of inline Jinja `{% set / if / elif / else %}` in
`roles/sys-svc-container/templates/deploy.yml.j2`. That made the
template noisy at the call site, hid the typo-safety logic from
unit tests, and forced every reviewer to re-derive the mapping table
from a chain of conditionals.
Move the mapping into a dedicated filter plugin so the call site is
a single readable expression and the mapping is exercised by a
focused test suite.
- plugins/filter/swarm_restart_condition.py: pure function, no Jinja
context dependency (the input is the only thing that matters).
Strips + str-casts the input so YAML-literal slop like `" no "`
or a numeric `0` does not defeat the mapping. Output is always a
member of the closed swarm condition set `{none, on-failure, any}`;
unknown / typo inputs map to `any` (safe default -- a misconfigured
long-running service still runs and the operator notices the typo
via the literal `condition: any` in the rendered file, instead of
silently degrading to `none` which would skip respawning a real
service).
- tests/unit/plugins/filter/test_swarm_restart_condition.py: 12
cases. Direct mapping per branch, `on-failure:N` suffix form
(suffix dropped because swarm carries max-attempts in a separate
`max_attempts` key), None / empty / whitespace fallback,
stringification of non-string input, strip semantics, an explicit
unknown-value test that pins the safe-default rationale, and a
property test over twelve inputs that asserts the output is always
inside the closed swarm condition set. A second test class
verifies the filter is registered under the exact name
`swarm_restart_condition` so a typo in the FilterModule shim does
not silently fall back to the identity filter.
- roles/sys-svc-container/templates/deploy.yml.j2: thirteen lines of
inline Jinja (`{% set _drp = … %}` + four `{% set
_swarm_restart_condition = … %}` branches) collapse to a single
filter expression:
condition: {{ docker_restart_policy
| default(DOCKER_RESTART_POLICY)
| swarm_restart_condition }}
The runtime mapping is identical to the pre-refactor inline
version (verified by the new unit tests against the same input
buckets), so nothing changes for existing roles.
The compose/swarm split on `depends_on:` shape (map form with
`condition:` vs. list form, per `docker stack deploy` constraint)
was being hand-coded as a `{% if DEPLOYMENT_MODE == 'swarm' %}` gate
around every block. Repetitive, easy to forget, and templates that
did remember it had to repeat the same six-line pattern per service.
The new `depends_on` lookup encapsulates the shape switch:
{{ lookup('depends_on', {SVC: 'service_healthy', …}) }} # mapping
{{ lookup('depends_on', [SVC_A, SVC_B]) }} # all default
- Compose -> map form with explicit `condition:` per entry.
- Swarm -> list form (just names; swarm ignores depends_on
ordering semantically but accepts the syntax).
- Mapping `None` value uses the default condition (`service_started`,
matching Docker Compose's behaviour when no condition is given).
- Plain list of names uses the default for every entry.
The output is shaped so the call site reads naturally at the
service-member column:
some_service:
image: …
{{ lookup('depends_on', {DB: 'service_healthy'}) }} # at col 4
volumes: …
Concretely the lookup emits line 1 WITHOUT leading whitespace and
lines 2+ with the `indent` kwarg (default 4) plus the YAML-relative
nested indent. Jinja substitution applies the template's leading
spaces to line 1 only, so the column-4 call placement makes
`depends_on:` land at column 4 and the children at 6 / 8. Lines 2+
carry their own indent because Jinja does not propagate template-side
leading whitespace to continuation lines.
Plugin + tests:
- plugins/lookup/depends_on.py: ~190 lines. Mapping / list / empty
input handling, default_condition kwarg, condition validation
against the closed set `{service_started, service_healthy,
service_completed_successfully}`, DEPLOYMENT_MODE resolution via
templar with `mode=` test-only override, indent kwarg semantics
documented in the module docstring.
- tests/unit/plugins/lookup/test_depends_on.py: 27 cases covering
TestInputNormalisation (mapping / list / empty / typo / single
string rejection / non-string sequence rejection / unsupported
type rejection / arity errors), TestModeBehaviour (compose / swarm
/ unknown-falls-to-compose / missing-DEPLOYMENT_MODE-falls-to-compose
/ mode-kwarg override), TestOutputFormatting (line-1 unindented
exact-string assertion, indent=0 edge case, indent=6 edge case,
insertion-order stability for diff-stable output).
Lint:
- tests/lint/ansible/jinja/test_compose_template_uses_depends_on_lookup.py:
rule key `compose-depends-on-must-use-lookup`. Flags three patterns
in any `*compose*.yml.j2` under `roles/`:
1. literal `depends_on:` keyword line
2. flow-form `depends_on: [a, b]`
3. literal `condition: service_*` line
The pre-existing `lookup('container_depends_on', application_id)`
is intentionally not flagged: its call form contains no literal
`depends_on:` token in source. Scan target broadened to
`name.endswith('.yml.j2') and 'compose' in name` so
`compose.override.yml.j2` and `compose-inits.yml.j2` are covered
alongside `compose.yml.j2` and `*.compose.yml.j2`.
Migration (43 depends_on blocks across 17 templates):
- erpnext (6): configurator + backend + websocket + scheduler +
queue-short + queue-long + frontend chains.
- mailu (8): imap + smtp + oletools + antispam + antivirus + webdav +
fetchmail + webmail.
- zammad (4): rails + 3 worker shapes.
- shopware (4): web + worker + scheduler + opensearch, init-gated.
- jitsi (3): three single-key dependencies on JITSI_PROSODY_CONTAINER.
- kix (3): backend + frontend + proxy.
- bigbluebutton/compose.override.yml.j2 (2): bbb-web + etherpad,
inside the existing BBB_COLLABORA_ENABLED gate.
- moodle (2): nginx + cron, both depending on moodle.
- opentalk (2): application + recorder.
- taiga (2): events + gateway.
- matrix (1): loop-driven via `{% set deps = {} %} +
__setitem__` escape hatch over MATRIX_BRIDGES.
- matomo (1): bootstrap service.
- nextcloud (1): whiteboard sidecar.
- openproject (1): proxy.
- funkwhale (1): front.
- magento (1): nginx.
- flowise (1): postgres + redis.
Pre-existing `lookup('container_depends_on', application_id)` calls
(erpnext, nextcloud, moodle, zammad, taiga, matrix, etc.) are left
untouched -- that lookup auto-detects DB / redis from the application
config and emits its own block; it is the app-aware counterpart to
this structural lookup.
Every call placement repositioned to column 4 (standard
compose-service-member indent) so the new line-1-unindented semantic
renders correctly. Eight templates needed the column-4 reposition
(via a mechanical one-shot script that was deleted after).
`make test` green; new lint + 27 unit tests green; independent
re-grep across all `*compose*.yml.j2` finds zero residual raw
`depends_on:` / `condition:` lines.
`docker stack deploy` rejects three compose-yml constructs that compose
parses just fine:
- the map form of `depends_on` (`<svc>: condition: service_healthy` /
`service_completed_successfully`) errors out with "Additional property
condition is not allowed",
- `network_mode: host` interacts badly with the swarm routing mesh and
is inconsistent across docker versions,
- `profiles:` is silently ignored, so a profile-gated one-shot deploys
as a regular replicated service and restart-loops.
All three patterns are now split per `DEPLOYMENT_MODE` so compose keeps
the existing behaviour and swarm gets a working equivalent (or a
documented nocheck for cases where there is no per-port equivalent).
depends_on map form -> per-mode list/map split:
- erpnext: backend + websocket depend on the configurator one-shot.
- kix: backend and frontend depend on the DB / backend service.
- matrix: synapse depends on each mautrix bridge.
- moodle: nginx and cron depend on moodle.
- nextcloud: whiteboard sidecar depends on redis.
- opentalk: recorder depends on rabbit + controller.
- shopware: web, worker, scheduler, opensearch depend on init.
- zammad: rails depends on the init service.
network_mode host:
- svc-prx-openresty: real split. Swarm renders per-port
`ports: target: 80/443, mode: host`, compose keeps `network_mode: host`.
- opentalk-livekit and prometheus-node-exporter: marked
`# nocheck: network-mode-host-without-swarm-gate` -- WebRTC media
server and host metrics collection genuinely need host networking
and have no clean swarm equivalent.
profiles:
- funkwhale typesense: gated to compose-only so swarm does not
silently deploy it as a regular service.
The mode-aware dotenv_quote test renders the filter through a real
jinja2.Environment to exercise the @pass_context branch. The filter
produces shell env-file values, not HTML, so autoescape=True would
mangle `&`, `<`, `>`, `'`, and `"` into HTML entities and break every
assertion. Suppressing bandit S701 here is the correct call -- there
is no XSS surface for a test-suite Environment that templates
KEY=VALUE pairs.
The swarm-compat lints (`compose-chdir-in-task`, `compose-verb-in-task`,
`docker-compose-raw-call`) currently flag every offending line and only
accept per-line / file-head `# nocheck:` markers. That is noisy for
two patterns that are objectively safe:
1. Tasks gated by `when: DEPLOYMENT_MODE \!= 'swarm'` (or
`== 'compose'`) never reach a swarm host - the compose verbs inside
are legitimate.
2. YAML comment lines, handler-trigger list items, and human-readable
metadata fields (`name:`, `msg:`, `title:`, `label:`, `description:`)
contain the word `compose` as documentation, not as a shell call.
The lints now skip both classes automatically.
- utils/annotations/task_gate.py: new helper
`is_task_compose_only_gated(lines, idx)`. Walks the surrounding
task block via a line-based heuristic and matches `when:` against
the compose-only DEPLOYMENT_MODE expressions. Documented as
deliberately shallow (no YAML parse).
- tests/lint/ansible/shell/test_no_compose_chdir_in_shell_tasks.py and
tests/lint/ansible/shell/test_no_compose_verb_in_shell_tasks.py:
add a file-level `is_suppressed_in_head` skip, the
`is_task_compose_only_gated` skip, and -- for the verb lint --
three extra regex exclusions for notify-list items, YAML metadata
fields, and quoted list items.
- tests/lint/ansible/shell/test_no_docker_compose_raw_call.py: skip
YAML comment lines (bluesky and jenkins each had a `# … docker
compose build …` doc-comment that was being flagged as a real call).
- tests/unit/utils/annotations/test_task_gate.py: 21 cases covering
block-bounds + gate detection, including a `TestKnownLimitations`
class that pins the heuristic's blind spots (folded-scalar `when:`,
list-form `when:`, `in`/`not in`, unquoted literal, outer-`when:`
inheritance into `block:` children) so a future YAML-aware refactor
knows exactly which assertions to flip.
`docker stack deploy` names services `<stack>_<compose-yaml-key>` and
silently ignores the compose `name:` field (that field maps to the
compose-only `container_name`). The lookup was returning
`<stack>_<services.<key>.name>` instead, which only happened to be
correct when the YAML key and the `name:` field matched.
The matomo bootstrap service is the failure case: services.bootstrap.name
is `matomo-bootstrap` but the YAML key (and therefore the swarm
service name) is `bootstrap`. The lookup returned
`matomo_matomo-bootstrap`, the bootstrap-wait task polled an absent
service, and the wait timed out after 7 empty retries -- root cause
of the act-39 swarm bootstrap stall.
- plugins/lookup/container_service.py: swarm branch returns
`<stack>_<service_key>`; compose branch unchanged.
- tests/unit/plugins/lookup/test_container_service.py: flip the two
swarm assertions to the corrected expectations and rename
`test_swarm_mode_uses_resolved_service_name` ->
`test_swarm_mode_ignores_services_name_field` so the contract is
visible from the test name.
docker stack deploy passes env-file values verbatim, including outer
double quotes - the container reads KEY="value" instead of KEY=value
and DB auth, URL parsers, and int(os.environ['KEY']) consumers break.
Compose keeps stripping the quotes as before.
- plugins/filter/dotenv.py: read DEPLOYMENT_MODE from the render
context via @jinja2.pass_context. compose mode keeps the old quoting
(escape $$, \, "). swarm mode passes the value through verbatim.
Missing or unknown mode falls back to compose quoting so a typo
never silently leaks unquoted values into compose. Direct Python
callers without a Jinja context keep the compose-style default.
- tests/unit/plugins/filter/test_dotenv_quote.py: keep the 12
direct-call cases, add 8 TestDotenvQuoteModeAware cases rendering
the filter through a real jinja2.Environment.
- 4 OIDC_TITLE templates (bookwyrm, confluence, jira, pretix): replace
the manual "{{ X | replace('\"','\\\"') }}" Compose-only escape
with {{ X | dotenv_quote }}.
- 3 env.j2 templates (nextcloud, taiga, wordpress): drop literal
outer quotes that were Compose-only decoration; keep | dotenv_quote
on the password / token assignments.
- sys-svc-rdbms/templates/env/mariadb.env.j2: strip literal quotes
unconditionally - the file name is not env.j2 so the dotenv-quote
integration test does not require the filter and the generated DB
passwords are alphanumeric.
Across 12 task files (gitea x5, shopware x5, wordpress install, mattermost
setup), strip the leftover `chdir: "{{ lookup('container', application_id,
'directories.instance') }}"` arg from shell tasks that already drive the
container via `container exec` through the wrapper. The chdir points at
the compose project root, which is only useful when calling docker-compose
CLI verbs. With `container exec` the cwd is the container's own working
directory; the host-side chdir does nothing. Empty `args:` blocks left
behind by the removal are collapsed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Across 23 web-app env.j2 templates, drop the surrounding "..." from
plain KEY="value" entries. compose silently strips them at env_file
parse time, but docker stack deploy preserves them literally, so the
container then sees KEY="value" with the quotes embedded and parsers
that call int(os.environ['KEY']) or split URLs/DSNs break under swarm.
The change is mechanical: every removed line was `KEY="value"` and
every added line is the same `KEY=value` without surrounding quotes.
Lines that use `| dotenv_quote` are untouched because that filter
deliberately produces escaped values for env_file consumers; the
existing password-quote integration test continues to require it for
sensitive variables.
Affected: bigbluebutton, bluesky, bookwyrm, bridgy-fed, chess,
confluence, decidim, espocrm, friendica, funkwhale, gitea, jira,
magento, mailu, mobilizon, openproject, phpmyadmin, pixelfed, pretix,
shopware, snipe-it, suitecrm, xwiki.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three follow-up improvements to the swarm-compatibility lint suite
introduced earlier on this branch:
1. compose-verb-in-task: skip lines that are Ansible handler triggers
(`notify: compose up`) or YAML comment lines before the regex
match. Without these guards every `notify: compose up` directive
in the repo tripped the lint, drowning the real findings.
2. compose.yml.j2 template lints (depends-on-condition,
network-mode-host): replace the terse "wrap in
{% if DEPLOYMENT_MODE \!= 'swarm' %}" message with concrete Jinja
examples showing the if/else swap and the swarm-native replacement
(list-form depends_on plus ansible-side wait; ports.mode=host plus
deploy.mode=global). Reviewers no longer have to reverse-engineer
the expected fix.
3. env-file-literal-quoted-value: replace an em-dash in the rule
docstring and fail message with a hyphen to satisfy the house
no-em-dash style.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Both the branding-asset HTTPS probe and the dashboard HTTP endpoint
probe in 02_manager_ops.yml use uri_retry, which defaults to running
on the ansible controller. Under the swarm-NFS DinD test rig the
controller is the act-runner, which has no network path to openresty
or the dashboard port inside the simulated cluster. Without
delegate_to: STACK_HOST the waits fail with Connection refused, the
manager host drops out of play_hosts, and downstream apps run only on
workers (matomo loses its manager-side ops and the play aborts).
Mirrors the same delegate_to + run_once pattern already applied to
web-app-matomo 09_config and svc-prx-openresty restart handler. In
compose mode STACK_HOST resolves to inventory_hostname so delegation
is a no-op.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add a meta/services.yml lint that flags any service carrying
default_placement: manager which is not paired with a per-line
`# nocheck: default-placement-manager` opt-out. Pinning to the manager
removes the worker pool from scheduling and defeats the benefit of
swarm: the service runs only on one node and worker capacity goes
unused.
Default policy: drop the line. Services should run anywhere so swarm
distributes them. Where additional capacity is needed, pair with
compose_replicas or deploy.mode: global instead.
Annotate the six legitimate exceptions with the required nocheck
marker and a short architectural justification above the line:
- svc-db-mariadb / svc-db-postgres / svc-db-openldap: central data
stores single-noded until replication is in scope
- svc-prx-openresty: edge TLS terminator at a known IP; the Lua
injection pipeline cannot ride the overlay yet
- svc-registry-cache / svc-registry-docker: registries that back
docker stack deploy itself; cannot run on a worker that still
needs to pull custom images from them
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add seven lint tests that identify roles which still rely on
compose-only patterns and therefore break under docker stack deploy.
Each test fails with concrete fix instructions (alternative syntax
plus example) and accepts a per-line nocheck opt-out so genuinely
compose-only roles can mark themselves explicitly:
- compose-verb-in-task: {{ BIN_COMPOSE }} or bare compose <verb>
(exec/run/restart/logs/up/down/ps/config) outside sys-svc-compose
- compose-chdir-in-task: chdir to lookup('container', ..., directories.instance)
in shell tasks (the compose project root is compose-only)
- docker-compose-raw-call: docker compose <verb> plugin form
- env-file-literal-quoted-value: KEY="..." in *.env.j2 without
dotenv_quote (swarm preserves env-file quotes literally)
- compose-only-depends-on-condition: condition: service_healthy /
service_completed_successfully under depends_on in compose.yml.j2
without a compose-only Jinja gate
- network-mode-host-without-swarm-gate: network_mode: host in
compose.yml.j2 without a compose-only Jinja gate
- compose-profiles-without-swarm-gate: profiles: in compose.yml.j2
without a compose-only Jinja gate
The three Jinja-template tests share an if/else stack walker that
correctly inverts the active condition inside {% else %} branches so
the swarm-vs-compose split pattern matomo uses does not false-positive.
Violations are intentionally left unresolved; this commit only
surfaces the backlog of roles needing swarm refactoring.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add a sibling lookup `container_service` (with 12 unit tests) that
returns the addressable form of a service for container service ps,
container service logs, and container ps --filter
label=com.docker.swarm.service.name=... It mirrors `container_address`
but emits the swarm service name (<stack>_<service_key>) instead of
the runtime container ID subshell. Compose mode returns the bare
service name.
Make the matomo manager-side ops swarm-aware without pinning matomo
to the manager. 03_manager_ops splits the wait into two mode-specific
blocks: in swarm a container service ps loop on CurrentState, in
compose the existing container inspect. After convergence the swarm
block resolves the hosting node via container service ps Node and the
compose block sets it to inventory_hostname; both set MATOMO_HOST_NODE
via separate set_fact tasks so the undefined matomo_swarm_node_query
under compose never reaches Jinja. A second unconditional task
resolves the local container ID on the hosting node (compose: bare
name; swarm: container ps with the swarm service-name label filter)
into MATOMO_LOCAL_CID. Downstream 04_install_path, 07_patch_database,
08_patch_trusted_hosts and the health wait all delegate_to
MATOMO_HOST_NODE and use MATOMO_LOCAL_CID for container exec, so the
patches and inspects always hit a local container. 05_bootstrap keeps
the compose-only profile run and adds a swarm path that tails the
in-stack matomo_bootstrap service logs for a hex token, registering
the same matomo_bootstrap_run shape downstream. 09_config delegates
to STACK_HOST with run_once so the uri_retry hits the published
matomo port from the manager regardless of where matomo is scheduled.
Drop the now-unused MATOMO_CONTAINER_ADDRESS from vars/main.yml.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Renumber the tasks/ tree to a flat NN_ scheme (drop the 02b / 03a / 03b
suffixes) and update all include_tasks references; main.yml -> 01_core
stays unchanged. Lift the hardcoded matomo bootstrap timezone label
into a global HOST_TIMEZONE_LABEL constant in group_vars/all so other
apps can share it; default to "United Kingdom" because GB has a single
IANA timezone and matomo's getTimezoneName collapses to the country
name in that case. Drop unconditional double quotes from env.j2 and
bootstrap.env.j2 since docker stack deploy preserves env-file quotes
literally and was feeding matomo "matomo" instead of matomo as the
DB name. Add MATOMO_ADMIN_EMAIL via the users lookup so the bootstrap
template can reference a typed variable instead of constructing the
address inline. Switch 08_patch_trusted_hosts to ansible.builtin.shell
with the script body piped in via stdin from lookup('file', ...) and
shrink the script file to just the inner sh body. Strip section-banner
and restate-the-code comments from vars/main.yml, env.j2 and the
extracted script; keep the two genuine trip-wire comments (matrix
deploy round-2 DB creds, include_role-chain role_vars pollution) in a
shorter Exception form.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The svc-storage-nfs-server host is the export source and never joins
the svc-storage-nfs-client group, so the assert tripped when msmtp
was loaded on the NFS host. Accept that group as a third pass
condition and clarify the fail message.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace compose exec / compose restart with the container wrapper
addressed by lookup('container_address', ...), so migrate + admin
bootstrap resolve to the swarm task ID (or the bare service name in
compose). Split the post-migrate restart into a compose path
(compose restart) and a swarm path (docker service update --force).
Use Jinja | quote on env values and the embedded bootstrap script.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace BIN_COMPOSE exec + instance-dir chdir with container exec
against WORDPRESS_CONTAINER_ADDRESS so plugin install/configure
tasks resolve correctly under swarm. Gate plugin block tasks on
IS_STACK_HOST to keep them single-execution. Drop the blanket
failed_when:false on the OIDC option-add task (the || in the shell
script already handles the idempotent case).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
