The mount pipeline was split across two layers: compose_volumes and
container_volumes lived as filter modules that ansible's filter loader
rejected on every start ("has no attribute 'FilterModule'", two warnings
per run), while the same-named lookups wrapped them thinly. The filter
bodies now live inside their lookups (the tor_socks precedent), the
filter files are gone, and the render tests moved next to the lookup
tests.
As lookups they see the templating context, so the application_id term
every call site passed back to its own role is gone too: both lookups
read application_id from context and fail loudly when it is missing.
107 templates and nfs_prep.yml drop the term; the calls collapse to
lookup('compose_volumes') and lookup('container_volumes', service_name).
The container_volumes call is now canonical - four leading spaces,
service_name in the service position, | indent(4) - enforced by a new
lint (test_container_volumes_call_shape). Normalizing the 85 literal
call sites surfaced that erpnext keyed its meta/volumes.yml mounts by
container name instead of compose service key, the only role to do so;
its meta now uses the service keys. wordpress's column-0 call, which
rendered its first line at top level, is indented like every other.
Template logic that only restated meta/volumes.yml is deleted: mailu fed
compose_volumes an extra_volumes dict whose eight names were looked up
from the very meta file compose_volumes reads itself; zammad's init and
railsserver extra mounts were the volumes.yml entries minus the two
services, which the meta now lists (the when-gated OIDC binds included).
Conditionality that meta cannot express stays in code, but in role-local
lookup plugins instead of template loops: matrix_bridge_mounts renders
the per-bridge registration mounts, nextcloud_extra_volumes the
feature-gated whiteboard/recording volumes (type: volume entries are
declared unconditionally by compose_volumes, so gating must happen
before it).
Parity is proven, not assumed: every mechanically rewritten template is
byte-identical to HEAD after the transformation regexes (95/95), and the
semantic roles render identically through the real functions against
HEAD's meta - zammad across six sso/tls polarities, all seven erpnext
services, mailu's top level. make test-lint passes complete.
Also: suitecrm's compose template pins container_extra_hosts with
host_alias (the SOCKS proxy URL host.docker.internal never resolved in
compose+tor), and the five disabled mautrix bridge addons use the
empty-default config lookup so templating the applications dict stops
emitting 125 errors per batch in every CI job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
e791d2556 centralized the empty-services emit into base.yml.j2 (emit 'services: {}' when no local sidecar). But base.yml.j2 renders the shared 'services:' key that consuming roles nest their own service under (svc-db-mariadb, and every web-app whose backends are all shared). When base emitted 'services: {}' and the role appended ' <service>:' after the include, the result was 'services: {}' followed by a nested mapping -> invalid YAML ('services must be a mapping'), which failed ca-inject for mariadb on compose and the swarm pre-deploy 'compose build' for ~19 apps on swarm (more backends are central there, so more roles hit the empty-sidecar branch). Revert base to a bare 'services:' and restore discourse's local gate -- discourse is the only launcher-based role with no compose service, so it alone needs 'services: {}' when its sidecars are all shared. compose_ca's no-op-on-empty-services and the .ansible-lint-ignore 04_login entry from e791d2556 stay.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
base.yml.j2 now captures the sidecar service includes and emits 'services: {}' when none render, instead of a bare 'services:' which docker compose rejects ('services must be a mapping'). A launcher-based app with all backends shared (web-app-discourse) has no compose services because it runs its container via ./launcher, so its compose was previously invalid. web-app-discourse drops its local _has_local_sidecar gate and just includes base. compose_ca.py (ca-inject) no-ops -- writes an empty {'services': {}} override and exits 0 -- instead of dying with 'No services found after merging' when the merged compose has no services, since there is nothing to CA-inject and the downstream compose up/ps tolerate an empty stack; add test_main_empty_services_writes_noop_override. .ansible-lint-ignore: add the 04_login resolve_host_cid path_join include (a load-failure false-positive that strict ansible-lint profile:min flags). make test is green (lint, integration render, unit).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Discourse's app container runs via the upstream discourse_docker launcher, not compose, so when every backing engine is central the rendered stack has no service of its own and 'services:' is null - the Validate Docker Compose handler then fails with 'services must be a mapping' in swarm. Emit 'services: {}' unless a local sidecar exists. The condition mirrors base.yml.j2's render gates exactly (DB/objstore/redis local via type templates that gate on 'local' = enabled and not shared, plus the oauth2 sso-proxy sidecar).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Convert the two registry-driven networks templates into pure-Python
rendering functions plus thin Ansible lookup plugins:
roles/sys-svc-compose/templates/networks.yml.j2 -> {{ lookup('compose_networks') }}
roles/sys-svc-container/templates/networks.yml.j2 -> {{ lookup('container_networks') | indent(4) }}
The rendering logic lives in utils/networks/render.py as pure-Python
functions that take a registry + closure-style config/database lookups,
making the logic directly unit-testable without a Jinja stub harness.
The lookup plugins are thin wrappers that resolve application_id,
DEPLOYMENT_MODE, and swarm.network.encryption from the templar, build
the service_registry, and forward to the render functions.
Migrate 233 include lines across 85 production templates. Convert
web-app-matrix's hand-coded MDAD networks block (default + conditional
postgres + conditional openldap) to the generic lookup; this adds an
openresty attachment when services.sso.enabled (harmless extra
interface, pinned in test_matrix_mdad_consumer_attaches_postgres_ldap_sso).
Mark local-DB / redis sidecars (sys-svc-rdbms/templates/services/,
svc-db-redis/templates/service.yml.j2) with `# nocheck: networks-literal`
because they intentionally attach to default-only.
Extend the lint to scan ALL roles/*/templates/**/*.yml.j2 (not only
compose.yml.j2) and to forbid the legacy include form; the marker
above is the documented escape hatch.
Coverage:
- tests/unit/utils/networks/test_render.py (30 cases incl. encrypted=false literal pin)
- tests/unit/plugins/lookup/test_compose_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/unit/plugins/lookup/test_container_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/integration/infrastructure/compose/test_networks_render.py (10 snapshot scenarios)
- tests/integration/infrastructure/compose/test_network_includes.py (3 lint rules)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Single SPOT lookup `plugins/lookup/compose_volumes.py` absorbs the
applications-lookup pipe step and auto-wires `DEPLOYMENT_MODE` plus
the `storage` group var from the templating context. Callers only
pass what genuinely varies per service (`application_id` and an
optional `extra_volumes` dict).
Behavior change (bug fix): the filter was always invoked with the
default `deployment_mode='compose'` because no caller passed it
explicitly, which silently skipped the swarm-NFS driver-opts
injection path. The lookup reads `DEPLOYMENT_MODE` from globals,
so swarm-NFS volumes now render the `driver: local` / `driver_opts:
type: nfs` block correctly. Callers can still override via
`deployment_mode=` / `storage=` kwargs when a template needs
non-default rendering.
Other coordinated changes:
- 81 template call sites migrated from pipe form to lookup form
(single-line and multi-line shapes).
- `FilterModule` registration in plugins/filter/compose_volumes.py
removed so the deprecated pipe form fails fast at render time.
The underlying `compose_volumes()` function stays exported and is
imported by the new lookup plus the existing 21-test filter unit
test module.
- 10 new lookup unit tests pin the contract (term shape, kwargs
pass-through, auto-wiring, override precedence, error cases).
- Integration test `test_compose_volumes_call_required.py` updated:
asserts the new lookup form is present in every `compose.yml.j2`
and adds a second test forbidding the deprecated pipe form
anywhere under `roles/**/templates/`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Filter no longer takes a `database_volume=` parameter. The volume name is
derived from the central provider's `services.<dbtype>.name` (shared) or
`<entity>_database` (dedicated) — mirroring `plugins/lookup/database.py`'s
`volume_prefix + host` construction. Simultaneous postgres + mariadb on
the same role now raises an AnsibleFilterError with a clear message;
support is deferred per the docstring TODO.
The integration test (renamed from `test_shared_service_volumes.py` to
`test_compose_volumes_call_required.py`) now requires every
`roles/*/templates/compose.yml.j2` to call `compose_volumes`, so the
top-level `volumes:` block is always rendered. Without it, flipping a
shared service to dedicated mode adds a named-volume reference that
Compose rejects with `service "<name>" refers to undefined volume`.
Adds the call to 14 templates that were missing it (dashboard, hugo,
sphinx, bridgy-fed, fediwall, littlejs, mig, mini-qr, navigator,
roulette-wheel, simpleicons, collabora, logout, openresty) and strips
the `database_volume=...` argument from every other role plus Taiga's
now-unused `TAIGA_DATABASE_VOLUME` var.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces inline `applications`/`users` merging with dedicated lookup
plugins and runtime hydration. Introduces `add_csp_hash` filter,
integrates sort into `find_dock_val_by_bkp_entr`, splits the Keycloak
one-shot realm import into `05a_realm_import_run.yml`, and fixes the
Mastodon administrator task YAML. Removes obsolete `csp_hashes` filter,
`sys-token-store` Jinja filter, and empty CLI setup modules.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Rename all docker-compose.yml templates to compose.yml
- Update override and CA override filenames accordingly
- Adjust compose wrapper, lookup plugins, filters and path helpers
- Update roles, tasks, handlers and scripts to use compose CLI
- Rewrite documentation and admin guides to reference compose
- Update unit and integration tests to expect compose.yml
- Keep internal variable keys for backward compatibility where needed
https://chatgpt.com/share/69870272-b168-800f-84c3-6417398e441f
kevinveenbirkenbachandClaude Opus 5