mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 10:59:59 +00:00
kevinveenbirkenbachandClaude Opus 5
acc9e25c0f
The mount pipeline was split across two layers: compose_volumes and
container_volumes lived as filter modules that ansible's filter loader
rejected on every start ("has no attribute 'FilterModule'", two warnings
per run), while the same-named lookups wrapped them thinly. The filter
bodies now live inside their lookups (the tor_socks precedent), the
filter files are gone, and the render tests moved next to the lookup
tests.
As lookups they see the templating context, so the application_id term
every call site passed back to its own role is gone too: both lookups
read application_id from context and fail loudly when it is missing.
107 templates and nfs_prep.yml drop the term; the calls collapse to
lookup('compose_volumes') and lookup('container_volumes', service_name).
The container_volumes call is now canonical - four leading spaces,
service_name in the service position, | indent(4) - enforced by a new
lint (test_container_volumes_call_shape). Normalizing the 85 literal
call sites surfaced that erpnext keyed its meta/volumes.yml mounts by
container name instead of compose service key, the only role to do so;
its meta now uses the service keys. wordpress's column-0 call, which
rendered its first line at top level, is indented like every other.
Template logic that only restated meta/volumes.yml is deleted: mailu fed
compose_volumes an extra_volumes dict whose eight names were looked up
from the very meta file compose_volumes reads itself; zammad's init and
railsserver extra mounts were the volumes.yml entries minus the two
services, which the meta now lists (the when-gated OIDC binds included).
Conditionality that meta cannot express stays in code, but in role-local
lookup plugins instead of template loops: matrix_bridge_mounts renders
the per-bridge registration mounts, nextcloud_extra_volumes the
feature-gated whiteboard/recording volumes (type: volume entries are
declared unconditionally by compose_volumes, so gating must happen
before it).
Parity is proven, not assumed: every mechanically rewritten template is
byte-identical to HEAD after the transformation regexes (95/95), and the
semantic roles render identically through the real functions against
HEAD's meta - zammad across six sso/tls polarities, all seven erpnext
services, mailu's top level. make test-lint passes complete.
Also: suitecrm's compose template pins container_extra_hosts with
host_alias (the SOCKS proxy URL host.docker.internal never resolved in
compose+tor), and the five disabled mautrix bridge addons use the
empty-default config lookup so templating the applications dict stops
emitting 125 errors per batch in every CI job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
99 lines
4.7 KiB
Markdown
99 lines
4.7 KiB
Markdown
# Central Engine Services (swarm + compose)
|
|
|
|
Root-cause architecture for running stateful "engine" dependencies (cache, queue,
|
|
search, vector, DNS resolver) the same way central databases already work, so that
|
|
application roles stay **stateless, unpinned and NFS-shareable** in swarm while the
|
|
engines are the only `placement: manager` (node-local) services.
|
|
|
|
## Problem this solves
|
|
|
|
NFS placement is derived from `placement` (see
|
|
`plugins/lookup/compose_volumes.py`): an unpinned role's volumes are bound to the
|
|
shared NFS mount (survive a reschedule), a pinned role keeps its volumes node-local.
|
|
Engine on-disk state (redis/rabbitmq/elasticsearch/sqlite/postfix-queue) **cannot
|
|
live on NFS** (fsync + locking corruption), so a role that *embeds* such an engine
|
|
must be pinned. Pinning whole app stacks to the manager:
|
|
|
|
- concentrates load on one node, and
|
|
- for mailu (13 services, 4 overlay networks, a static-IP unbound resolver) triggers
|
|
a swarm VIP-allocation deadlock (`could not find an available IP while allocating
|
|
VIP` + `has pending allocations`, even on empty /24 networks).
|
|
|
|
The fix is to **externalise the engines** into central, pinned `svc-db-*` / `svc-dns-*`
|
|
roles (exactly like `svc-db-postgres` / `svc-db-mariadb`) and have apps connect to
|
|
them. Apps then own only user-data volumes (NFS-safe) and need no pin.
|
|
|
|
## Central engine roles
|
|
|
|
One central role per engine, mirroring `svc-db-postgres`:
|
|
|
|
| role | engine | per-consumer isolation |
|
|
| --- | --- | --- |
|
|
| `svc-db-postgres` (exists) | PostgreSQL | a database |
|
|
| `svc-db-mariadb` (exists) | MariaDB | a database |
|
|
| `svc-db-redis` (convert) | Redis | ACL user + key-prefix `{entity}:*` |
|
|
| `svc-db-memcached` (convert) | Memcached | key-prefix only (no native ACL) |
|
|
| `svc-db-rabbitmq` (new) | RabbitMQ | a vhost + user |
|
|
| `svc-db-elasticsearch` (new) | Elasticsearch | index/alias namespace |
|
|
| `svc-db-typesense` (new) | Typesense | API-key scoped collections |
|
|
| `svc-db-qdrant` (new) | Qdrant | per-consumer collection prefix |
|
|
| `svc-dns-unbound` (new) | Unbound | shared resolver (no per-consumer split) |
|
|
|
|
Each central role provides:
|
|
|
|
- `meta/services.yml`: `placement: manager`, `enabled`, `shared: true`,
|
|
image/version/ports/name.
|
|
- `templates/compose.yml.j2`: its own standalone stack.
|
|
- `tasks/00_core.yml`: deploy the central stack + wait healthy.
|
|
- `tasks/01_init.yml`: **per-consumer provisioning** (create the ACL user / vhost /
|
|
namespace and its credentials), idempotent, mirroring postgres `01_init`.
|
|
- a connection lookup mirroring `database`: `lookup('<engine>', consumer_id, want)`
|
|
returning host/port/user/password/url/db for the consumer.
|
|
|
|
## shared toggle (compose + swarm, identical)
|
|
|
|
Mirrors the existing postgres `shared` flag:
|
|
|
|
- `services.<engine>.enabled: true` + `shared: true` -> **central**: the consumer
|
|
connects to the central engine over the shared cross-stack network using the
|
|
`<engine>` lookup and its own provisioned credentials.
|
|
- `services.<engine>.enabled: true` + `shared: false` -> **embedded** sidecar
|
|
(current behaviour), for single-host / opt-out.
|
|
|
|
`base.yml.j2` embeds an engine sidecar only `{% if enabled and not shared %}`.
|
|
|
|
## Cross-stack connectivity
|
|
|
|
Central engines are **shared-net providers** (the existing mechanism in
|
|
`utils/networks/render.py` / `compose_volumes.py`); consumers attach to the engine's
|
|
shared overlay and the `<engine>` lookup resolves the cross-stack address
|
|
(`resolve-container-id` in swarm, service name in compose), exactly as
|
|
`lookup('database', ...)` does today.
|
|
|
|
## mailu root-cause fix
|
|
|
|
`svc-dns-unbound` is a single-service, single-network, static-IP, pinned role. mailu's
|
|
services drop their own unbound + static IP, attach to the `svc-dns-unbound` network,
|
|
and point `dns:` at its static IP. The static-IP allocation therefore lives in a
|
|
trivial 1-service stack (no deadlock) instead of mailu's 13-service / 4-network stack.
|
|
mailu also stops embedding redis (uses central) -> fewer services -> less VIP pressure.
|
|
|
|
## Placement
|
|
|
|
- Central `svc-db-*` / `svc-dns-*`: `placement: manager` (single node).
|
|
- Application roles: **no** `placement` -> distributed + NFS-shared.
|
|
- `tests/lint/ansible/roles/meta/test_unpinned_no_engine_volume.py` stays as the
|
|
guard: an unpinned role must not declare an engine on-disk data volume.
|
|
|
|
## Lifecycle + ordering
|
|
|
|
- Central engine is enabled **on demand**: deployed only when at least one inventory
|
|
role consumes it `shared: true` (same discovery the central DB uses).
|
|
- Deploy order: central engine -> consumer `02_init` provisioning -> consumer
|
|
(`run_after`, like postgres).
|
|
|
|
## Scope
|
|
|
|
Test-fresh: the goal is green compose **and** swarm deploys on fresh act clusters.
|
|
No embedded->central data migration path (that is a separate prod-upgrade concern).
|