kevinveenbirkenbachandClaude Opus 5
acc9e25c0f
The mount pipeline was split across two layers: compose_volumes and
container_volumes lived as filter modules that ansible's filter loader
rejected on every start ("has no attribute 'FilterModule'", two warnings
per run), while the same-named lookups wrapped them thinly. The filter
bodies now live inside their lookups (the tor_socks precedent), the
filter files are gone, and the render tests moved next to the lookup
tests.
As lookups they see the templating context, so the application_id term
every call site passed back to its own role is gone too: both lookups
read application_id from context and fail loudly when it is missing.
107 templates and nfs_prep.yml drop the term; the calls collapse to
lookup('compose_volumes') and lookup('container_volumes', service_name).
The container_volumes call is now canonical - four leading spaces,
service_name in the service position, | indent(4) - enforced by a new
lint (test_container_volumes_call_shape). Normalizing the 85 literal
call sites surfaced that erpnext keyed its meta/volumes.yml mounts by
container name instead of compose service key, the only role to do so;
its meta now uses the service keys. wordpress's column-0 call, which
rendered its first line at top level, is indented like every other.
Template logic that only restated meta/volumes.yml is deleted: mailu fed
compose_volumes an extra_volumes dict whose eight names were looked up
from the very meta file compose_volumes reads itself; zammad's init and
railsserver extra mounts were the volumes.yml entries minus the two
services, which the meta now lists (the when-gated OIDC binds included).
Conditionality that meta cannot express stays in code, but in role-local
lookup plugins instead of template loops: matrix_bridge_mounts renders
the per-bridge registration mounts, nextcloud_extra_volumes the
feature-gated whiteboard/recording volumes (type: volume entries are
declared unconditionally by compose_volumes, so gating must happen
before it).
Parity is proven, not assumed: every mechanically rewritten template is
byte-identical to HEAD after the transformation regexes (95/95), and the
semantic roles render identically through the real functions against
HEAD's meta - zammad across six sso/tls polarities, all seven erpnext
services, mailu's top level. make test-lint passes complete.
Also: suitecrm's compose template pins container_extra_hosts with
host_alias (the SOCKS proxy URL host.docker.internal never resolved in
compose+tor), and the five disabled mautrix bridge addons use the
empty-default config lookup so templating the applications dict stops
emitting 125 errors per batch in every CI job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4.7 KiB
Central Engine Services (swarm + compose)
Root-cause architecture for running stateful "engine" dependencies (cache, queue,
search, vector, DNS resolver) the same way central databases already work, so that
application roles stay stateless, unpinned and NFS-shareable in swarm while the
engines are the only placement: manager (node-local) services.
Problem this solves
NFS placement is derived from placement (see
plugins/lookup/compose_volumes.py): an unpinned role's volumes are bound to the
shared NFS mount (survive a reschedule), a pinned role keeps its volumes node-local.
Engine on-disk state (redis/rabbitmq/elasticsearch/sqlite/postfix-queue) cannot
live on NFS (fsync + locking corruption), so a role that embeds such an engine
must be pinned. Pinning whole app stacks to the manager:
- concentrates load on one node, and
- for mailu (13 services, 4 overlay networks, a static-IP unbound resolver) triggers
a swarm VIP-allocation deadlock (
could not find an available IP while allocating VIP+has pending allocations, even on empty /24 networks).
The fix is to externalise the engines into central, pinned svc-db-* / svc-dns-*
roles (exactly like svc-db-postgres / svc-db-mariadb) and have apps connect to
them. Apps then own only user-data volumes (NFS-safe) and need no pin.
Central engine roles
One central role per engine, mirroring svc-db-postgres:
| role | engine | per-consumer isolation |
|---|---|---|
svc-db-postgres (exists) |
PostgreSQL | a database |
svc-db-mariadb (exists) |
MariaDB | a database |
svc-db-redis (convert) |
Redis | ACL user + key-prefix {entity}:* |
svc-db-memcached (convert) |
Memcached | key-prefix only (no native ACL) |
svc-db-rabbitmq (new) |
RabbitMQ | a vhost + user |
svc-db-elasticsearch (new) |
Elasticsearch | index/alias namespace |
svc-db-typesense (new) |
Typesense | API-key scoped collections |
svc-db-qdrant (new) |
Qdrant | per-consumer collection prefix |
svc-dns-unbound (new) |
Unbound | shared resolver (no per-consumer split) |
Each central role provides:
meta/services.yml:placement: manager,enabled,shared: true, image/version/ports/name.templates/compose.yml.j2: its own standalone stack.tasks/00_core.yml: deploy the central stack + wait healthy.tasks/01_init.yml: per-consumer provisioning (create the ACL user / vhost / namespace and its credentials), idempotent, mirroring postgres01_init.- a connection lookup mirroring
database:lookup('<engine>', consumer_id, want)returning host/port/user/password/url/db for the consumer.
shared toggle (compose + swarm, identical)
Mirrors the existing postgres shared flag:
services.<engine>.enabled: true+shared: true-> central: the consumer connects to the central engine over the shared cross-stack network using the<engine>lookup and its own provisioned credentials.services.<engine>.enabled: true+shared: false-> embedded sidecar (current behaviour), for single-host / opt-out.
base.yml.j2 embeds an engine sidecar only {% if enabled and not shared %}.
Cross-stack connectivity
Central engines are shared-net providers (the existing mechanism in
utils/networks/render.py / compose_volumes.py); consumers attach to the engine's
shared overlay and the <engine> lookup resolves the cross-stack address
(resolve-container-id in swarm, service name in compose), exactly as
lookup('database', ...) does today.
mailu root-cause fix
svc-dns-unbound is a single-service, single-network, static-IP, pinned role. mailu's
services drop their own unbound + static IP, attach to the svc-dns-unbound network,
and point dns: at its static IP. The static-IP allocation therefore lives in a
trivial 1-service stack (no deadlock) instead of mailu's 13-service / 4-network stack.
mailu also stops embedding redis (uses central) -> fewer services -> less VIP pressure.
Placement
- Central
svc-db-*/svc-dns-*:placement: manager(single node). - Application roles: no
placement-> distributed + NFS-shared. tests/lint/ansible/roles/meta/test_unpinned_no_engine_volume.pystays as the guard: an unpinned role must not declare an engine on-disk data volume.
Lifecycle + ordering
- Central engine is enabled on demand: deployed only when at least one inventory
role consumes it
shared: true(same discovery the central DB uses). - Deploy order: central engine -> consumer
02_initprovisioning -> consumer (run_after, like postgres).
Scope
Test-fresh: the goal is green compose and swarm deploys on fresh act clusters. No embedded->central data migration path (that is a separate prod-upgrade concern).