compose gave up on `dependency failed to start: container kix-backend is unhealthy`,
and the backend's own log names the reason:
psql -q -h postgres -p 5432 -U kix kix -c 'CREATE EXTENSION ...' -> succeeds
/opt/kix/bin/kix.Console.pl Maint::Config::Rebuild --allow-root -> FATAL:
password authentication failed for user "kix"
Two calls in a row, one working and one not, with the same database. psql takes its
password from PGPASSWORD; the perl layer takes it from /opt/kix/config/01main, and the
entrypoint's own `[ ! -f /opt/kix/config/01main ]` test reports that file missing on
every start.
/opt/kix/config was on no volume. Every recreated container therefore loses 01main,
re-enters the first-install path, and runs it against a database that is already
installed. The credential rotation between the sync and the async pass is only what
made the container get recreated; any redeploy does this.
Everything else was ruled out from the run rather than assumed: the ALTER USER landed
(changed: true, 22:18:44), the rendered env carried the same value (22:18:57), there is
exactly one postgres container and 192.168.200.2 is its address, and it stayed up and
healthy across the whole window.
Persisting the directory alone would trade one failure for another: the entrypoint runs
`ln -s '/opt/kix/conf.d/*' /opt/kix/config/` on every start, unquoted-glob-style but with
the glob quoted, so it creates a single dangling link named `*` -- and the second start
would find it there. The link points at a path with a literal asterisk and nothing reads
it, so clear_stale_conf_link.yml removes it on every node before the stack comes up.
The password patch declaration follows the friendica pattern and needs 2c4e0e18b to work
at all: its groups are extended-regex, which basic-regex sed rejected outright.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A volume that reaches several services repeated its container path once per
mount. Written out each time it has no single point of truth: a rename that
misses one line still parses, still deploys, and mounts one service somewhere
nobody intended. Nothing fails until that service looks for content which is
not there.
YAML already solves this, and 62722a6ab used it here first -- web-app-openproject
anchors await_migrations_target and aliases it for the second service. Nineteen
roles now follow, and tests/lint/repository/spot/test_volumes_target_single_spot.py
fails on any target spelled out twice. An alias carries no path, so the rule
cannot see it, which is the point.
The conversion was mechanical and is provable rather than reviewed: every file
was rewritten and then reloaded, and any file whose yaml.safe_load result
differed from the original was discarded instead of kept. Nineteen written,
none discarded. Anchors and aliases are a serialisation form; the loader
resolves them before any consumer sees a document.
validate_volumes_meta over all 81 meta/volumes.yml reports no violation.
make test-lint: 328 passed.
web-app-prometheus carries two unrelated deletions: comments that predate this
change and fell under the comment lint once the file was touched. They explained
a 0444 mode and a compose-only bind; both facts are visible in the entries
themselves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The export subdir was created run_once on the NFS server only; a worker
that had already readdir'd the export kept a negative dentry and swarm
rejected every replicas>1 task with 'readdirent .../_data: no such file
or directory' for the whole converge window. Materialize the bind-device
dir through each node's own mount, guarded by a per-node mountpoint
assert so an unmounted node fails the deploy loudly instead of receiving
a local mkdir that splits the brain and trips the nfs-client
has-local-data assert on the next full deploy.
compose_volumes popped the volume 'nfs' metadata before nfs_prep could
read it, so declared uid/gid/mode never took effect and every subdir
fell back to root 0755; keep the mapping under the compose-spec
extension key x-infinito-nfs (ignored by docker, verified against stack
config and compose config). KIX declares uid/gid 110010100 mode 0770
through it, fixing the backend_api_token EACCES crash on its shared
volume.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The manager pin existed for the embedded kix-db container (private registry image with node-local pgdata). Remove that container and wire KIXDB_* through the database lookup: central svc-db-postgres (pg_trgm pre-activated via services.postgres.extensions) or the auto-rendered sidecar when no central provider is in the inventory. The proxy service floats and scales to the topology default; backend stays a pinned singleton (it bundles the kix.Daemon scheduler) and frontend too (websocket notifications without cross-instance pub/sub). Variants pin postgres.shared once true/once false. Existing embedded-postgres data is not migrated (fresh-install semantics for the DB move).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
resource_filter now falls back service_name -> entity_name -> hard
default, so entity-level limits keep applying to per-service compose
entries. Compose service names drop the entity prefix (kix, jitsi,
zammad, bluesky); the docker container_name keeps it via compose_only.
A new lint forbids '<entity>-'-prefixed service names.
Service names and internal ports are declared once: vars/main.yml
holds <ROLE>_<SERVICE>_SERVICE constants (container names derived via
entity_name), meta/services.yml declares ports.internal consumed
through config lookups. sso-proxy upstreams stack-qualify the origin
host in swarm because short aliases are ambiguous on shared overlays.
Stale proxy.service_key entries fixed for erpnext/shopware/nextcloud.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Swarm config objects mount root:root; a 0440 mode is unreadable by a non-root container app user and crashloops the service (jenkins JCasC AccessDeniedException, exit 5). docker compose never hits this since there the file is a host bind-mount inheriting host perms.
New lint test_config_object_world_readable flags type:config modes lacking o+r (type:secret exempt, per-line # nocheck escape). Widen the 22 offending 0440 configs across 18 roles to 0444: adds read only, cannot break a reader, and fixes latent non-root cases (e.g. synapse, pixelfed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a strict lint that flags every `type: bind` entry in
roles/<role>/meta/volumes.yml whose source or any mount target points
to a single FILE (path ends in a recognised file extension or a
Jinja var with a file-suffix name like *_CONF/*_FILE). Single-file
binds are swarm-fragile: the source path is only present on the node
where the file was rendered, so swarm rejects the task as soon as it
lands on a different node.
Fix is `type: config`: the manager loads the file at `stack deploy`
time and swarm distributes it through raft to every node.
compose-mode honours the same canonical entry (the compose_volumes
filter renders it as a top-level `configs:` block; container_volumes
emits the per-service reference), so the role stays mode-agnostic.
Per-line opt-out: `# nocheck: meta-volume-file-bind-should-be-config`
with a short `# Reason:` for legitimate exceptions (sockets, kernel
interfaces, runtime-rotated files, multi-MB blobs, host writebacks).
This commit migrates 23 file-bind entries across 16 roles to
type:config: web-app-flowise, friendica, kix (2), listmonk, mailu,
matrix (2), mediawiki, mobilizon, nextcloud, pgadmin, pixelfed,
prometheus (4), shopware (2), suitecrm, taiga (2), zammad. All gain a
default mode:"0440"; mount-level read_only:true is dropped (configs
are read-only by default and the schema forbids the field for
type:config).
Updates the prometheus role-wiring test to accept both type:bind and
type:config for the alert_rules mount probe.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Migrate every role's meta/volumes.yml from ad-hoc dict-of-named-volumes
to a canonical dict-of-dicts schema where the YAML key is the semantic
short name and the entry body carries type / name / source / mounts /
nfs / mode / read_only. The optional 'name:' field is the docker volume
name (defaults to the key); the previous 'docker_name:' field is gone.
What's in this commit (210 paths):
Infrastructure (23)
- New filter plugins/filter/{compose,container}_volumes.py — render the
top-level volumes:/configs:/secrets: section and per-service mount
blocks from the canonical meta. container_volumes uses the SEMANTIC
name as per-service mount source so it resolves against compose_volumes'
top-level dict key (docker_name as source produced 'undefined volume'
on swarm).
- New lookup plugins/lookup/{compose,container,volume}_volumes.py.
volume(application_id, name) returns a normalised single entry;
volume(application_id) returns the whole canonical dict.
- utils/cache/applications.py keeps a sibling registry
_CANONICAL_VOLUMES_BY_ROLE outside the merged applications payload so
raw Jinja source: strings never enter the templar walk (root-cause of
the 200x cache-miss + recursive get_merged_applications storm).
- utils/roles/applications/mounts.py — schema validator + normaliser;
list-shape is rejected.
- plugins/lookup/nginx.py: _volume_path_by_name reads canonical directly.
- scripts/tests/deploy/swarm/_context.sh: NFS_VOLUMES extraction handles
both legacy dict and canonical dict shapes.
- roles/svc-prx-openresty/handlers/restart_and_validate.yml: gate
compose-only validate+reload behind a single when-block; swarm uses
service-update instead (bind-mounts of a FILE pin source inode at
task-create time, so atomic write-tmp+rename leaves the running task
on a deleted inode).
- roles/sys-svc-compose/tasks/utils/nfs_prep.yml: switch to
lookup('volume', application_id) | dict2items so NFS subdir
pre-creation works under the new cache split.
Tests (12)
- New strict lints flagging inline mount sections in compose templates,
single-file bind mounts, and legacy lookup('config', ..., 'volumes.X.Y')
call sites — all volume-mount surfaces MUST flow through container_volumes
/ compose_volumes / volume lookups.
- Integration: test_volumes_compose_chain + test_volume_constants_are_lookups
accept both the legacy and the new lookup('volume', app, 'X').name forms
during the migration window.
- Unit tests for the new lookup, filter, mounts helper, plus updated
nginx lookup test.
- Prometheus role-wiring test reads canonical meta for the
alert_rules.yml bind mount.
Role migrations (175)
- 62 meta/volumes.yml converted to dict-of-dicts canonical shape
- 63 templates/compose*.yml.j2 replace inline volumes:/configs:/secrets:
with lookup('container_volumes', application_id, '<service>') |
indent(N) per service + a single trailing lookup('compose_volumes',
application_id) for the top-level block
- 50 vars/main.yml drop dead volume constants and refactor 23 remaining
legacy lookup('config', ..., 'volumes.X.Y') to lookup('volume',
application_id, 'X').Y across 9 roles
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5