The mount pipeline was split across two layers: compose_volumes and
container_volumes lived as filter modules that ansible's filter loader
rejected on every start ("has no attribute 'FilterModule'", two warnings
per run), while the same-named lookups wrapped them thinly. The filter
bodies now live inside their lookups (the tor_socks precedent), the
filter files are gone, and the render tests moved next to the lookup
tests.
As lookups they see the templating context, so the application_id term
every call site passed back to its own role is gone too: both lookups
read application_id from context and fail loudly when it is missing.
107 templates and nfs_prep.yml drop the term; the calls collapse to
lookup('compose_volumes') and lookup('container_volumes', service_name).
The container_volumes call is now canonical - four leading spaces,
service_name in the service position, | indent(4) - enforced by a new
lint (test_container_volumes_call_shape). Normalizing the 85 literal
call sites surfaced that erpnext keyed its meta/volumes.yml mounts by
container name instead of compose service key, the only role to do so;
its meta now uses the service keys. wordpress's column-0 call, which
rendered its first line at top level, is indented like every other.
Template logic that only restated meta/volumes.yml is deleted: mailu fed
compose_volumes an extra_volumes dict whose eight names were looked up
from the very meta file compose_volumes reads itself; zammad's init and
railsserver extra mounts were the volumes.yml entries minus the two
services, which the meta now lists (the when-gated OIDC binds included).
Conditionality that meta cannot express stays in code, but in role-local
lookup plugins instead of template loops: matrix_bridge_mounts renders
the per-bridge registration mounts, nextcloud_extra_volumes the
feature-gated whiteboard/recording volumes (type: volume entries are
declared unconditionally by compose_volumes, so gating must happen
before it).
Parity is proven, not assumed: every mechanically rewritten template is
byte-identical to HEAD after the transformation regexes (95/95), and the
semantic roles render identically through the real functions against
HEAD's meta - zammad across six sso/tls polarities, all seven erpnext
services, mailu's top level. make test-lint passes complete.
Also: suitecrm's compose template pins container_extra_hosts with
host_alias (the SOCKS proxy URL host.docker.internal never resolved in
compose+tor), and the five disabled mautrix bridge addons use the
empty-default config lookup so templating the applications dict stops
emitting 125 errors per batch in every CI job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The email SPOT turns tls off for an .onion host -- Tor already carries the encryption and no authority issues a certificate for a v3 address -- and env.j2 renders TLS_FLAVOR=notls from that same value. Mailu's front then opens no implicit-TLS listener, while compose kept publishing 465, 993 and 995 anyway.
The swarm reachability probe reads the published set straight out of docker service inspect and demands a listener per port, so it failed on exactly those three while 25, 110, 143, 587, 4190 and 8007 answered. The stack was advertising endpoints it does not serve; the probe was right.
A role-local container_ports_mailu builds the block instead: protocols come from MAILU_PORTS, so declaring a port in the role meta is enough, and the implicit-TLS three drop out when tls is off. The terms are assembled in Python because a lookup call cannot spread a built list into positional terms -- writing the set out twice, once per TLS flavor, would let the two copies drift.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
extra_hosts is the one compose key a service may not carry twice. The onion SSO
back-channel pin was emitted by a Jinja partial that six roles included, while
six other services wrote the key by hand; nothing stopped a service from
receiving both. YAML then either rejects the document or keeps one list and
drops the other, silently.
The partial becomes plugins/lookup/container_extra_hosts.py, and every pin now
goes through it. Callers add their own entries with extra_hosts=[...], the way
container_volumes already takes extra_volumes=, and the lookup merges them after
the automatic pin and deduplicates in first-seen order. With nothing to emit it
returns an empty string.
The swarm address reasoning from 7b1839681 is preserved and now written down in
the plugin rather than only in that commit: the pin must NOT be host-gateway in
swarm, because that alias resolves per node while openresty publishes :80 with
mode: host under node.role == manager. It carries the stack host's address,
which reaches every task because the compose file is rendered once with the
manager as inventory host. The mode is read as compose_mode_force or
DEPLOYMENT_MODE, not the bare cluster mode, so a role with an override does not
lose its pin and its overlay at once.
Two roles kept a pin that now collides with the automatic one, and both are
resolved the same way -- the hand-written entry stays for clearnet and yields
the .onion case to the lookup, which picks the address per mode. In
web-app-mailu the three conditional entries move into a role-local
mailu_extra_hosts lookup exposed as MAILU_EXTRA_HOSTS, because an eight-line
conditional expression in the template argument is worse than the block it
replaced. It returns its entries as separate results, so vars/main.yml uses
query(); plain lookup() would join them into one comma-separated string.
tests/lint/ansible/jinja/test_compose_extra_hosts_use_lookup.py fails on any
literal extra_hosts: key in a role template, so the invariant survives the next
author who needs a pin. It follows test_compose_ports_use_container_ports_lookup:
the raw construct is forbidden, the lookup is not mandated per service -- a
service with no pins is correct without it, and the swarm branch raises when
facts are absent, which would turn a no-op into a failure surface for containers
that never speak OIDC.
Equivalence checked by rendering rather than by reading: under the trim_blocks +
lstrip_blocks the stack-host action forces for compose.yml.j2, both former call
site shapes (indented, and matrix's column 0) emit byte-identical text to the
new form when the block applies. The no-op case differs -- a whitespace-only
line where the include emitted none -- and yaml.safe_load returns the same
document for both.
31 unit tests; make test-lint 325 passed. Does not touch web-app-espocrm, whose
missing pin is a separate claim and gets its own commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every compose.yml.j2 now declares services as '{{ service_name }}:' with
its '{% set %}' knobs on two-space lines above the key, includes at four
spaces and if/for tags aligned to the content they wrap. The layout
renders correctly because stack_host_template enables lstrip_blocks for
compose.yml.j2; indent(4, True) in the container base and '{%+' guards
in whitespace-sensitive scalars keep the shared fragments identical
under both render modes. A new lint test enforces the skeleton.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
mailu_admin uniquely runs on public DNSSEC resolvers, so internal service names
are pinned to overlay VIPs via extra_hosts. That pinning assumed a central redis
stack; with a local redis sidecar MAILU_SWARM_REDIS_HOST is the bare name
'redis', and pinning it to an absent central VIP shadowed stack DNS, so admin
blackholed its first redis dial and was SIGKILLed unhealthy in a loop (blocking
chess and mini-qr). Gate the redis VIP inspect, the fact, and the extra_hosts
line on the redis engine not being local. The shared-redis path is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Take the host IP as an optional third element of each term
([service, protocol, host_ip]) instead of an ip= kwarg, and build the whole
ports: block from declared ports.local/public (host) + ports.internal
(container). Convert every compose template across 84 roles, folding the
remaining raw var/numeric/specific-ip mappings (including mailu's per-interface
mail ports bound to MAILU_IP4_PUBLIC). Multi-term calls render one term per line.
Add the compose-ports-must-use-container-ports lint: it forbids any raw
"<host>:<published>:<container>" publish item under a ports: block and requires
every term to declare ports.internal. Remove
tests/lint/roles/test_swarm_internal_ports.py, now subsumed.
Fix a container_depends_on rendering regression. The lookup self-indented
(default indent=4), so the added 4-space placement + | indent(4) triple-indented
the block. Flip the lookup default to column 0 (consistent with
container_networks), make every call use the 4-space + | indent(4) form, and add
a lint enforcing it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rewrite every hardcoded literal container port in a compose.yml.j2 publish
mapping (- "<host>:<...ports.local|public.kind...>:<LITERAL>") to read the
declared ports.internal.<kind> via lookup, across 22 roles, so the published
(local/public) port and the container (internal) port are a single declared
source of truth. Mappings whose container already references the lookup or a
variable are left as-is.
This surfaced and fixed a real swarm bug: web-app-openproject declared
ports.internal.http: 8080 while the container actually listens on 80, so the
swarm resolve_upstream pointed openproject at the wrong port. Reconciled to 80
(the mapping's real container port).
Extend tests/lint/roles/test_swarm_internal_ports.py with a second check: a
published mapping's container segment must come from ports.internal via a lookup
(or a variable referencing it), never a hardcoded literal.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
DEPLOYMENT_MODE is binary (swarm / compose), so a negated `!= 'swarm'` gate
always means `== 'compose'`. Replace every negated gate across roles/ and
group_vars with the positive form for readability, and add a lint
(tests/lint/ansible/variables/test_no_deployment_mode_negation.py) that forbids
`DEPLOYMENT_MODE !=` in .yml/.yaml/.j2 with a `# nocheck:
deployment-mode-negation` escape.
The compose-verb / chdir / jinja-gate lint guidance messages now suggest the
positive form too; the gate matcher (utils/annotations/task_gate.py) and its
test inputs keep recognising the legacy `!= 'swarm'` so a nocheck-marked
exception is still detected as compose-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Consume central redis and unbound (drop embedded sidecars). Swarm: resolve central mariadb/redis overlay VIPs into admin extra_hosts so the DNSSEC resolver does not NXDOMAIN internal names; pre-create nginx/certs bind dirs plus a placeholder cert before stack-up; recreate the API token when a fresh cluster lacks it; wait for antispam via container exec; disable OIDC SSL verify under self-signed TLS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Set a dash-style name on every addressed sub-service in meta (admin, antispam, redis, imap, smtp, oletools, antivirus, webdav, fetchmail, webmail, resolver, front), expose each as MAILU_<SVC>_CONTAINER_ADDRESS in vars, and drive the compose container_name from those vars so container_address matches the real container.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Convert the two registry-driven networks templates into pure-Python
rendering functions plus thin Ansible lookup plugins:
roles/sys-svc-compose/templates/networks.yml.j2 -> {{ lookup('compose_networks') }}
roles/sys-svc-container/templates/networks.yml.j2 -> {{ lookup('container_networks') | indent(4) }}
The rendering logic lives in utils/networks/render.py as pure-Python
functions that take a registry + closure-style config/database lookups,
making the logic directly unit-testable without a Jinja stub harness.
The lookup plugins are thin wrappers that resolve application_id,
DEPLOYMENT_MODE, and swarm.network.encryption from the templar, build
the service_registry, and forward to the render functions.
Migrate 233 include lines across 85 production templates. Convert
web-app-matrix's hand-coded MDAD networks block (default + conditional
postgres + conditional openldap) to the generic lookup; this adds an
openresty attachment when services.sso.enabled (harmless extra
interface, pinned in test_matrix_mdad_consumer_attaches_postgres_ldap_sso).
Mark local-DB / redis sidecars (sys-svc-rdbms/templates/services/,
svc-db-redis/templates/service.yml.j2) with `# nocheck: networks-literal`
because they intentionally attach to default-only.
Extend the lint to scan ALL roles/*/templates/**/*.yml.j2 (not only
compose.yml.j2) and to forbid the legacy include form; the marker
above is the documented escape hatch.
Coverage:
- tests/unit/utils/networks/test_render.py (30 cases incl. encrypted=false literal pin)
- tests/unit/plugins/lookup/test_compose_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/unit/plugins/lookup/test_container_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/integration/infrastructure/compose/test_networks_render.py (10 snapshot scenarios)
- tests/integration/infrastructure/compose/test_network_includes.py (3 lint rules)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add plugins/lookup/container_image.py: returns the full `image: "<ref>"`
line so templates drop the redundant image: prefix and surrounding
quotes. Supports `custom=True` to auto-construct `<entity_name>_custom`
for locally-built images, plus explicit `image=`/`version=` overrides
for exotic cases (taiga back-auth/front-oidc variants, decidim infinito
suffix, etc.).
* 25 unit tests for the lookup cover compose vs swarm prefix, custom
flag, overrides, tag_only=True, missing version warnings.
* New strict lint (tests/lint/.../test_no_raw_image_in_compose_templates)
fails on any raw `image:` line in roles/*/templates/*.j2 unless
nocheck-suppressed. Legacy `image: "{{ lookup(...) }}"` form no
longer accepted.
* Migrate 159 image refs across ~80 roles to the standalone
`{{ lookup('container_image', application_id, '<service_key>') }}`
form. 26 roles use `custom=True` (mariadb, postgres, openldap,
jenkins, jira, gitlab, nextcloud, mediawiki, shopware, wordpress,
odoo, joomla, openproject, bookwyrm, espocrm, hugo, littlejs,
fediwall, roulette-wheel, openresty, confluence and more).
* Fix svc-db-redis application_id leak: service.yml.j2 is included
into consuming roles where application_id evaluates to the outer
role, so the image lookup needs the hardcoded 'svc-db-redis'.
* Integration tests (test_templates_require_image,
test_build_requires_image) recognise the standalone lookup as an
implicit `image:` at the same indent.
* Mediawiki swarm tasks: add explicit `:{{ MEDIAWIKI_VERSION }}` to
the registry-prefixed image string because the custom var is now
bare.
Only 3 legitimate exotic nocheck-suppressed image lines remain:
matrix-mdad runner (nested services.matrix.runner.image path),
matrix mautrix bridges (per-iteration dynamic) and svc-registry-docker
(chicken-and-egg self-reference).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Migrate every role's meta/volumes.yml from ad-hoc dict-of-named-volumes
to a canonical dict-of-dicts schema where the YAML key is the semantic
short name and the entry body carries type / name / source / mounts /
nfs / mode / read_only. The optional 'name:' field is the docker volume
name (defaults to the key); the previous 'docker_name:' field is gone.
What's in this commit (210 paths):
Infrastructure (23)
- New filter plugins/filter/{compose,container}_volumes.py — render the
top-level volumes:/configs:/secrets: section and per-service mount
blocks from the canonical meta. container_volumes uses the SEMANTIC
name as per-service mount source so it resolves against compose_volumes'
top-level dict key (docker_name as source produced 'undefined volume'
on swarm).
- New lookup plugins/lookup/{compose,container,volume}_volumes.py.
volume(application_id, name) returns a normalised single entry;
volume(application_id) returns the whole canonical dict.
- utils/cache/applications.py keeps a sibling registry
_CANONICAL_VOLUMES_BY_ROLE outside the merged applications payload so
raw Jinja source: strings never enter the templar walk (root-cause of
the 200x cache-miss + recursive get_merged_applications storm).
- utils/roles/applications/mounts.py — schema validator + normaliser;
list-shape is rejected.
- plugins/lookup/nginx.py: _volume_path_by_name reads canonical directly.
- scripts/tests/deploy/swarm/_context.sh: NFS_VOLUMES extraction handles
both legacy dict and canonical dict shapes.
- roles/svc-prx-openresty/handlers/restart_and_validate.yml: gate
compose-only validate+reload behind a single when-block; swarm uses
service-update instead (bind-mounts of a FILE pin source inode at
task-create time, so atomic write-tmp+rename leaves the running task
on a deleted inode).
- roles/sys-svc-compose/tasks/utils/nfs_prep.yml: switch to
lookup('volume', application_id) | dict2items so NFS subdir
pre-creation works under the new cache split.
Tests (12)
- New strict lints flagging inline mount sections in compose templates,
single-file bind mounts, and legacy lookup('config', ..., 'volumes.X.Y')
call sites — all volume-mount surfaces MUST flow through container_volumes
/ compose_volumes / volume lookups.
- Integration: test_volumes_compose_chain + test_volume_constants_are_lookups
accept both the legacy and the new lookup('volume', app, 'X').name forms
during the migration window.
- Unit tests for the new lookup, filter, mounts helper, plus updated
nginx lookup test.
- Prometheus role-wiring test reads canonical meta for the
alert_rules.yml bind mount.
Role migrations (175)
- 62 meta/volumes.yml converted to dict-of-dicts canonical shape
- 63 templates/compose*.yml.j2 replace inline volumes:/configs:/secrets:
with lookup('container_volumes', application_id, '<service>') |
indent(N) per service + a single trailing lookup('compose_volumes',
application_id) for the top-level block
- 50 vars/main.yml drop dead volume constants and refactor 23 remaining
legacy lookup('config', ..., 'volumes.X.Y') to lookup('volume',
application_id, 'X').Y across 9 roles
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The compose/swarm split on `depends_on:` shape (map form with
`condition:` vs. list form, per `docker stack deploy` constraint)
was being hand-coded as a `{% if DEPLOYMENT_MODE == 'swarm' %}` gate
around every block. Repetitive, easy to forget, and templates that
did remember it had to repeat the same six-line pattern per service.
The new `depends_on` lookup encapsulates the shape switch:
{{ lookup('depends_on', {SVC: 'service_healthy', …}) }} # mapping
{{ lookup('depends_on', [SVC_A, SVC_B]) }} # all default
- Compose -> map form with explicit `condition:` per entry.
- Swarm -> list form (just names; swarm ignores depends_on
ordering semantically but accepts the syntax).
- Mapping `None` value uses the default condition (`service_started`,
matching Docker Compose's behaviour when no condition is given).
- Plain list of names uses the default for every entry.
The output is shaped so the call site reads naturally at the
service-member column:
some_service:
image: …
{{ lookup('depends_on', {DB: 'service_healthy'}) }} # at col 4
volumes: …
Concretely the lookup emits line 1 WITHOUT leading whitespace and
lines 2+ with the `indent` kwarg (default 4) plus the YAML-relative
nested indent. Jinja substitution applies the template's leading
spaces to line 1 only, so the column-4 call placement makes
`depends_on:` land at column 4 and the children at 6 / 8. Lines 2+
carry their own indent because Jinja does not propagate template-side
leading whitespace to continuation lines.
Plugin + tests:
- plugins/lookup/depends_on.py: ~190 lines. Mapping / list / empty
input handling, default_condition kwarg, condition validation
against the closed set `{service_started, service_healthy,
service_completed_successfully}`, DEPLOYMENT_MODE resolution via
templar with `mode=` test-only override, indent kwarg semantics
documented in the module docstring.
- tests/unit/plugins/lookup/test_depends_on.py: 27 cases covering
TestInputNormalisation (mapping / list / empty / typo / single
string rejection / non-string sequence rejection / unsupported
type rejection / arity errors), TestModeBehaviour (compose / swarm
/ unknown-falls-to-compose / missing-DEPLOYMENT_MODE-falls-to-compose
/ mode-kwarg override), TestOutputFormatting (line-1 unindented
exact-string assertion, indent=0 edge case, indent=6 edge case,
insertion-order stability for diff-stable output).
Lint:
- tests/lint/ansible/jinja/test_compose_template_uses_depends_on_lookup.py:
rule key `compose-depends-on-must-use-lookup`. Flags three patterns
in any `*compose*.yml.j2` under `roles/`:
1. literal `depends_on:` keyword line
2. flow-form `depends_on: [a, b]`
3. literal `condition: service_*` line
The pre-existing `lookup('container_depends_on', application_id)`
is intentionally not flagged: its call form contains no literal
`depends_on:` token in source. Scan target broadened to
`name.endswith('.yml.j2') and 'compose' in name` so
`compose.override.yml.j2` and `compose-inits.yml.j2` are covered
alongside `compose.yml.j2` and `*.compose.yml.j2`.
Migration (43 depends_on blocks across 17 templates):
- erpnext (6): configurator + backend + websocket + scheduler +
queue-short + queue-long + frontend chains.
- mailu (8): imap + smtp + oletools + antispam + antivirus + webdav +
fetchmail + webmail.
- zammad (4): rails + 3 worker shapes.
- shopware (4): web + worker + scheduler + opensearch, init-gated.
- jitsi (3): three single-key dependencies on JITSI_PROSODY_CONTAINER.
- kix (3): backend + frontend + proxy.
- bigbluebutton/compose.override.yml.j2 (2): bbb-web + etherpad,
inside the existing BBB_COLLABORA_ENABLED gate.
- moodle (2): nginx + cron, both depending on moodle.
- opentalk (2): application + recorder.
- taiga (2): events + gateway.
- matrix (1): loop-driven via `{% set deps = {} %} +
__setitem__` escape hatch over MATRIX_BRIDGES.
- matomo (1): bootstrap service.
- nextcloud (1): whiteboard sidecar.
- openproject (1): proxy.
- funkwhale (1): front.
- magento (1): nginx.
- flowise (1): postgres + redis.
Pre-existing `lookup('container_depends_on', application_id)` calls
(erpnext, nextcloud, moodle, zammad, taiga, matrix, etc.) are left
untouched -- that lookup auto-detects DB / redis from the application
config and emits its own block; it is the app-aware counterpart to
this structural lookup.
Every call placement repositioned to column 4 (standard
compose-service-member indent) so the new line-1-unindented semantic
renders correctly. Eight templates needed the column-4 reposition
(via a mechanical one-shot script that was deleted after).
`make test` green; new lint + 27 unit tests green; independent
re-grep across all `*compose*.yml.j2` finds zero residual raw
`depends_on:` / `condition:` lines.
Single SPOT lookup `plugins/lookup/compose_volumes.py` absorbs the
applications-lookup pipe step and auto-wires `DEPLOYMENT_MODE` plus
the `storage` group var from the templating context. Callers only
pass what genuinely varies per service (`application_id` and an
optional `extra_volumes` dict).
Behavior change (bug fix): the filter was always invoked with the
default `deployment_mode='compose'` because no caller passed it
explicitly, which silently skipped the swarm-NFS driver-opts
injection path. The lookup reads `DEPLOYMENT_MODE` from globals,
so swarm-NFS volumes now render the `driver: local` / `driver_opts:
type: nfs` block correctly. Callers can still override via
`deployment_mode=` / `storage=` kwargs when a template needs
non-default rendering.
Other coordinated changes:
- 81 template call sites migrated from pipe form to lookup form
(single-line and multi-line shapes).
- `FilterModule` registration in plugins/filter/compose_volumes.py
removed so the deprecated pipe form fails fast at render time.
The underlying `compose_volumes()` function stays exported and is
imported by the new lookup plus the existing 21-test filter unit
test module.
- 10 new lookup unit tests pin the contract (term shape, kwargs
pass-through, auto-wiring, override precedence, error cases).
- Integration test `test_compose_volumes_call_required.py` updated:
asserts the new lookup form is present in every `compose.yml.j2`
and adds a second test forbidding the deprecated pipe form
anywhere under `roles/**/templates/`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace per-key `compose_container_name` lookup with a generic
`compose_only(key, value)` SPOT that emits `key: "<value>"` only in
compose mode and nothing in swarm. Same gating logic now serves
container_name (157 call sites) and pull_policy (17 call sites)
without duplicate plugin code.
Why this consolidation:
- The earlier `build-key` lint rule was misdirected. Swarm tolerates
`build:` with a harmless warning when `image:` is pre-pushed (via
the existing handlers/swarm_registry_push.py flow in
roles/sys-svc-compose). The actual swarm-rejected key is
`pull_policy:` ("Additional property pull_policy is not allowed"),
and 10 templates were silently broken because they declared inline
`pull_policy: never` outside the existing `{% if DEPLOYMENT_MODE
\!= 'swarm' %}` gate in roles/sys-svc-container/templates/build.yml.j2.
- A single generic lookup is cleaner than N per-key wrappers when
the gating logic is identical (emit-key-or-nothing).
Lint changes in tests/lint/ansible/jinja/test_mode_incompatible_keys.py:
- New `pull-policy-key` rule, regex `^\s*pull_policy\s*:`, remediation
points at compose_only.
- Existing `container-name` rule keeps its raw-key regex; remediation
now also points at compose_only.
- Broken-split / broken-embed regexes rekeyed on the `compose_only`
lookup name (was `compose_container_name`).
- `build-key` rule and its 12 nocheck annotations across templates
removed.
Coordinated changes:
- 86 templates rewritten (157 container_name calls migrated;
17 inline pull_policy lines replaced).
- 1 redundant `{% if DEPLOYMENT_MODE \!= 'swarm' %}` guard dropped in
build.yml.j2 (the lookup self-gates).
- 12 unit tests pin the compose_only contract.
- Catalog (suppression.md) updated: `pull-policy-key` row added,
`container-name` row remediation updated.
- Old `plugins/lookup/compose_container_name.py` and its test module
deleted.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
container_name is illegal in swarm mode when replicas > 1 (docker rejects
with "services.deploy.replicas: can't set container_name and X as container
name must be unique"). All compose.yml.j2 templates now route through a
single lookup that suppresses the key in swarm mode and emits it in compose
mode.
SPOT: plugins/lookup/compose_container_name.py
Reads DEPLOYMENT_MODE from the templating context, returns
'container_name: "<name>"' under compose and an empty string under swarm.
Call sites read:
{{ lookup('compose_container_name', MY_CONTAINER) }}
No explicit DEPLOYMENT_MODE pass-through needed.
Sweep: 82 templates rewritten across roles/web-app-*, roles/web-svc-*,
roles/svc-*, plus 4 partial includes (sso_proxy, redis, mariadb, postgres).
For container names assembled from multiple Jinja fragments the args use
tilde concat (X ~ '-' ~ Y) so the lookup receives a single resolved string
and the call stays a single well-formed Jinja expression.
Lint: tests/lint/ansible/jinja/test_no_raw_container_name.py blocks
(a) raw 'container_name:' YAML keys in any roles/*/templates/**/*.j2,
(b) split-Jinja regression where the outer braces close mid-call:
{{ lookup('compose_container_name', A }}-{{ B) }},
(c) embedded-Jinja regression inside a string argument:
lookup(..., "{{ X }}_suffix") (removed in ansible-core 2.23).
The (b) and (c) regexes accept either single- or double-quoted lookup
name and scan the full file so multi-line shapes also surface. Suppress
per-site with '# nocheck: container-name' on the same or preceding line.
Tests: tests/unit/plugins/lookup/test_compose_container_name.py pins the
lookup contract: compose vs swarm output, whitespace tolerance, default-to
compose when DEPLOYMENT_MODE is absent, AnsibleError on missing or excess
terms, non-string name coercion.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The compose templates appended additional depends_on entries (e.g. taiga
rabbitmq services, openproject cache/seeder, mailu resolver/front) on
column 6 directly after the lookup call. When the lookup returned ""
(no DB and no redis), those entries dangled without a depends_on header
and broke YAML parsing.
Move every such trailing block into the lookup's `extra` kwarg so the
header is always emitted as soon as any entry exists. Adds a regression
test for the extras-only case.
Affected: taiga, mailu, magento, pixelfed, openproject, espocrm.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Eliminate include-style depends_on assembly. The lookup resolves DB/redis
inductively from applications config and returns indented YAML (or "" when
no entries), removing the empty `depends_on:` block bug in libretranslate.
Migrates 43 compose templates from `{% include … dmbs_*.yml.j2 %}` to
`{{ lookup('container_depends_on', application_id) }}`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add tests/lint/ansible/roles/test_compose_no_raw_image.py (strict mode):
every `image:` line in roles/<role>/templates/compose.yml.j2 must
resolve through a Jinja substitution so the underlying registry /
repository / tag is declared in the role's meta/services.yml instead of
being pinned in the compose template.
Fix the 8 pre-existing violations the lint surfaces:
- web-svc-coturn: gateway nginx → services.gateway.{image,version}
- web-svc-simpleicons: local build tag → services.application.{image,version}
- web-app-mailu: clamav antivirus → services.antivirus.{image,version}
- web-app-taiga (5×): events, async-rabbitmq, events-rabbitmq, protected,
gateway → top-level services.<name>.{image,version}
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Filter no longer takes a `database_volume=` parameter. The volume name is
derived from the central provider's `services.<dbtype>.name` (shared) or
`<entity>_database` (dedicated) — mirroring `plugins/lookup/database.py`'s
`volume_prefix + host` construction. Simultaneous postgres + mariadb on
the same role now raises an AnsibleFilterError with a clear message;
support is deferred per the docstring TODO.
The integration test (renamed from `test_shared_service_volumes.py` to
`test_compose_volumes_call_required.py`) now requires every
`roles/*/templates/compose.yml.j2` to call `compose_volumes`, so the
top-level `volumes:` block is always rendered. Without it, flipping a
shared service to dedicated mode adds a named-volume reference that
Compose rejects with `service "<name>" refers to undefined volume`.
Adds the call to 14 templates that were missing it (dashboard, hugo,
sphinx, bridgy-fed, fediwall, littlejs, mig, mini-qr, navigator,
roulette-wheel, simpleicons, collabora, logout, openresty) and strips
the `database_volume=...` argument from every other role plus Taiga's
now-unused `TAIGA_DATABASE_VOLUME` var.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces inline `applications`/`users` merging with dedicated lookup
plugins and runtime hydration. Introduces `add_csp_hash` filter,
integrates sort into `find_dock_val_by_bkp_entr`, splits the Keycloak
one-shot realm import into `05a_realm_import_run.yml`, and fixes the
Mastodon administrator task YAML. Removes obsolete `csp_hashes` filter,
`sys-token-store` Jinja filter, and empty CLI setup modules.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rename lookup plugin from `docker` to `container`
- Switch all lookups from `lookup('docker', ...)` to `lookup('container', ...)`
- Normalize compose file keys:
- docker_compose → compose
- docker_compose_override → compose_override
- docker_compose_ca_override → compose_ca_override
- Centralize compose path resolution via docker_paths_utils
- Update compose_file_args to always derive paths from DIR_COMPOSITIONS
- Align all roles, templates, and vars with container/compose naming
- Update and rename unit tests accordingly
This prepares the codebase for engine-agnostic container backends
(e.g. Docker ↔ Podman) and removes legacy docker-specific assumptions.
https://chatgpt.com/share/6987a7cc-d374-800f-99ad-bb4c8f92c59e
- Rename all docker-compose.yml templates to compose.yml
- Update override and CA override filenames accordingly
- Adjust compose wrapper, lookup plugins, filters and path helpers
- Update roles, tasks, handlers and scripts to use compose CLI
- Rewrite documentation and admin guides to reference compose
- Update unit and integration tests to expect compose.yml
- Keep internal variable keys for backward compatibility where needed
https://chatgpt.com/share/69870272-b168-800f-84c3-6417398e441f
