compose gave up on `dependency failed to start: container kix-backend is unhealthy`,
and the backend's own log names the reason:
psql -q -h postgres -p 5432 -U kix kix -c 'CREATE EXTENSION ...' -> succeeds
/opt/kix/bin/kix.Console.pl Maint::Config::Rebuild --allow-root -> FATAL:
password authentication failed for user "kix"
Two calls in a row, one working and one not, with the same database. psql takes its
password from PGPASSWORD; the perl layer takes it from /opt/kix/config/01main, and the
entrypoint's own `[ ! -f /opt/kix/config/01main ]` test reports that file missing on
every start.
/opt/kix/config was on no volume. Every recreated container therefore loses 01main,
re-enters the first-install path, and runs it against a database that is already
installed. The credential rotation between the sync and the async pass is only what
made the container get recreated; any redeploy does this.
Everything else was ruled out from the run rather than assumed: the ALTER USER landed
(changed: true, 22:18:44), the rendered env carried the same value (22:18:57), there is
exactly one postgres container and 192.168.200.2 is its address, and it stayed up and
healthy across the whole window.
Persisting the directory alone would trade one failure for another: the entrypoint runs
`ln -s '/opt/kix/conf.d/*' /opt/kix/config/` on every start, unquoted-glob-style but with
the glob quoted, so it creates a single dangling link named `*` -- and the second start
would find it there. The link points at a path with a literal asterisk and nothing reads
it, so clear_stale_conf_link.yml removes it on every node before the stack comes up.
The password patch declaration follows the friendica pattern and needs 2c4e0e18b to work
at all: its groups are extended-regex, which basic-regex sed rejected outright.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/schema.yml became meta/secrets.yml, but its content still landed at
applications.<app>.credentials — the one meta file whose payload did not sit
under a root named after the file. Every other topic obeys that rule:
meta/volumes.yml goes to volumes, meta/rbac.yml to rbac. Credentials now go to
secrets.credentials.
The path exists exactly once, in utils/manager/credential_key.py, and
OVERRIDE_SECTION is derived from its two segments rather than spelled beside
them, so the dotted form used by --set and the nested form used by the
inventory cannot drift. Six consumers read it: the loader, the generator, the
CLI override parser, the two emit helpers, and the four lookup validators.
--set now accepts exactly one key shape,
applications.<app>.secrets.credentials.<key>. The three short forms are
rejected instead of guessed at: the same short key names different credentials
on different apps. That also retired primary_app_id, which only existed to
disambiguate them.
Three defects surfaced while wiring this up. _apply_one_role_schema still wrote
to the old root, so an app's own credentials would have moved while its
providers' stayed behind. web-app-pgadmin reads through
lookup('applications', ...).credentials.<key>, an access path no regex over
lookup('config', ...) could see, and would have deployed with an empty postgres
password. And _META_NON_GENERIC_TOPICS still listed 'schema' — a bare topic
name, invisible to any path search — so the generic file-root loop assigned the
raw schema under secrets and the follow-up write landed inside the cached
document, truncating keycloak's four credentials to one for every later
reader.
tests/integration/ansible/lookups/config/test_credentials_roundtrip.py pins both
directions of the seam and the cache invariant that last defect broke.
Existing inventories still carry the pre-rename root; migrating them is
deliberately out of scope here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The manager pin existed for the embedded kix-db container (private registry image with node-local pgdata). Remove that container and wire KIXDB_* through the database lookup: central svc-db-postgres (pg_trgm pre-activated via services.postgres.extensions) or the auto-rendered sidecar when no central provider is in the inventory. The proxy service floats and scales to the topology default; backend stays a pinned singleton (it bundles the kix.Daemon scheduler) and frontend too (websocket notifications without cross-instance pub/sub). Variants pin postgres.shared once true/once false. Existing embedded-postgres data is not migrated (fresh-install semantics for the DB move).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Container-name vars derived their value inconsistently (hard-coded literals, entity_name composites) instead of the role-owned SPOT. Add a lint requiring every *_CONTAINER var to resolve through lookup('config', app, 'services.<x>.name'), the bare entity_name form, or the engine lookup; give each multi-container worker a name: in meta/services.yml (with realistic resource limits) and switch its var to the lookup. Extend the engine lookup with a shared-aware 'container' field (central provider container when shared, embedded <entity>-<engine> otherwise) and gate zammad's embedded memcached sidecar on not-shared so its container_name never collides with the central memcached.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
resource_filter now falls back service_name -> entity_name -> hard
default, so entity-level limits keep applying to per-service compose
entries. Compose service names drop the entity prefix (kix, jitsi,
zammad, bluesky); the docker container_name keeps it via compose_only.
A new lint forbids '<entity>-'-prefixed service names.
Service names and internal ports are declared once: vars/main.yml
holds <ROLE>_<SERVICE>_SERVICE constants (container names derived via
entity_name), meta/services.yml declares ports.internal consumed
through config lookups. sso-proxy upstreams stack-qualify the origin
host in swarm because short aliases are ambiguous on shared overlays.
Stale proxy.service_key entries fixed for erpnext/shopware/nextcloud.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The KIX backend runs a FRESH_INSTALL schema bootstrap that outlives the default
150x2s converge budget on loaded CI runners (db reaches 1/1 while the backend
stays 0/1 Starting), so the swarm converge gate timed out. Make the gate's
retries overridable (default unchanged at 150) and raise it to 600 for kix.
Swarm-only gate; compose is unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add plugins/lookup/container_image.py: returns the full `image: "<ref>"`
line so templates drop the redundant image: prefix and surrounding
quotes. Supports `custom=True` to auto-construct `<entity_name>_custom`
for locally-built images, plus explicit `image=`/`version=` overrides
for exotic cases (taiga back-auth/front-oidc variants, decidim infinito
suffix, etc.).
* 25 unit tests for the lookup cover compose vs swarm prefix, custom
flag, overrides, tag_only=True, missing version warnings.
* New strict lint (tests/lint/.../test_no_raw_image_in_compose_templates)
fails on any raw `image:` line in roles/*/templates/*.j2 unless
nocheck-suppressed. Legacy `image: "{{ lookup(...) }}"` form no
longer accepted.
* Migrate 159 image refs across ~80 roles to the standalone
`{{ lookup('container_image', application_id, '<service_key>') }}`
form. 26 roles use `custom=True` (mariadb, postgres, openldap,
jenkins, jira, gitlab, nextcloud, mediawiki, shopware, wordpress,
odoo, joomla, openproject, bookwyrm, espocrm, hugo, littlejs,
fediwall, roulette-wheel, openresty, confluence and more).
* Fix svc-db-redis application_id leak: service.yml.j2 is included
into consuming roles where application_id evaluates to the outer
role, so the image lookup needs the hardcoded 'svc-db-redis'.
* Integration tests (test_templates_require_image,
test_build_requires_image) recognise the standalone lookup as an
implicit `image:` at the same indent.
* Mediawiki swarm tasks: add explicit `:{{ MEDIAWIKI_VERSION }}` to
the registry-prefixed image string because the custom var is now
bare.
Only 3 legitimate exotic nocheck-suppressed image lines remain:
matrix-mdad runner (nested services.matrix.runner.image path),
matrix mautrix bridges (per-iteration dynamic) and svc-registry-docker
(chicken-and-egg self-reference).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Migrate every role's meta/volumes.yml from ad-hoc dict-of-named-volumes
to a canonical dict-of-dicts schema where the YAML key is the semantic
short name and the entry body carries type / name / source / mounts /
nfs / mode / read_only. The optional 'name:' field is the docker volume
name (defaults to the key); the previous 'docker_name:' field is gone.
What's in this commit (210 paths):
Infrastructure (23)
- New filter plugins/filter/{compose,container}_volumes.py — render the
top-level volumes:/configs:/secrets: section and per-service mount
blocks from the canonical meta. container_volumes uses the SEMANTIC
name as per-service mount source so it resolves against compose_volumes'
top-level dict key (docker_name as source produced 'undefined volume'
on swarm).
- New lookup plugins/lookup/{compose,container,volume}_volumes.py.
volume(application_id, name) returns a normalised single entry;
volume(application_id) returns the whole canonical dict.
- utils/cache/applications.py keeps a sibling registry
_CANONICAL_VOLUMES_BY_ROLE outside the merged applications payload so
raw Jinja source: strings never enter the templar walk (root-cause of
the 200x cache-miss + recursive get_merged_applications storm).
- utils/roles/applications/mounts.py — schema validator + normaliser;
list-shape is rejected.
- plugins/lookup/nginx.py: _volume_path_by_name reads canonical directly.
- scripts/tests/deploy/swarm/_context.sh: NFS_VOLUMES extraction handles
both legacy dict and canonical dict shapes.
- roles/svc-prx-openresty/handlers/restart_and_validate.yml: gate
compose-only validate+reload behind a single when-block; swarm uses
service-update instead (bind-mounts of a FILE pin source inode at
task-create time, so atomic write-tmp+rename leaves the running task
on a deleted inode).
- roles/sys-svc-compose/tasks/utils/nfs_prep.yml: switch to
lookup('volume', application_id) | dict2items so NFS subdir
pre-creation works under the new cache split.
Tests (12)
- New strict lints flagging inline mount sections in compose templates,
single-file bind mounts, and legacy lookup('config', ..., 'volumes.X.Y')
call sites — all volume-mount surfaces MUST flow through container_volumes
/ compose_volumes / volume lookups.
- Integration: test_volumes_compose_chain + test_volume_constants_are_lookups
accept both the legacy and the new lookup('volume', app, 'X').name forms
during the migration window.
- Unit tests for the new lookup, filter, mounts helper, plus updated
nginx lookup test.
- Prometheus role-wiring test reads canonical meta for the
alert_rules.yml bind mount.
Role migrations (175)
- 62 meta/volumes.yml converted to dict-of-dicts canonical shape
- 63 templates/compose*.yml.j2 replace inline volumes:/configs:/secrets:
with lookup('container_volumes', application_id, '<service>') |
indent(N) per service + a single trailing lookup('compose_volumes',
application_id) for the top-level block
- 50 vars/main.yml drop dead volume constants and refactor 23 remaining
legacy lookup('config', ..., 'volumes.X.Y') to lookup('volume',
application_id, 'X').Y across 9 roles
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Variant 1 of web-app-kix disables every dynamic service flag
(css, dashboard, email, ldap, logout, matomo, oauth2, prometheus —
meta/variants.yml:32-56). With `services.ldap.enabled=false`,
svc-db-openldap is not in the deploy closure, so the top-level
`LDAP.SERVER.DOMAIN` chain in group_vars/all/12_ldap.yml:14-16
resolves via `lookup('config','svc-db-openldap', ...)` and raises
hard the moment any task touches it.
tasks/01_core.yml rendered `zzz_auth_ldap.xml.j2` unconditionally,
the template referenced `KIX_LDAP_HOST` etc., the lazy `LDAP.*`
chain fired, and the deploy aborted before any companion subapp
ever reached Playwright (job 77602110489 on run 26362676803 has
zero variant-1 artefacts across the entire 14-app bundle, while
gitlab's variant-1 produced at least three).
Introduce `KIX_LDAP_ENABLED` in vars/main.yml as the single source
of truth (matches OPENTALK_OIDC_ENABLED / OPENTALK_TURN_ENABLED in
web-app-opentalk), and gate both consumers on it:
* the LDAP-config render task — skipped when LDAP is off
* the compose mount of the LDAP config file — emitted only when
LDAP is on, so the backend container does not try to mount a
missing path
Matches the persisted operator preference: auth services stay
optional — fix code, not variants.
Static check only (per operator request). Confidence ~70% that
this clears the immediate variant-1 failure point; possible
follow-up issues (oauth2-proxy absence, etc.) would only surface
once the kix log reaches further than the previous abort.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh:
- rename run_sort_claude_settings worker to run_claude (matches the
renamed scripts/lint/claude.sh).
- drop the EXIT trap that wiped status_dir; cleanup happens explicitly
after `wait` returns, so a parallel worker that interrupts the
parent never races a vanished status dir.
- each worker now captures its tool's exit code and writes one of
OK / SKIP / FAIL. Tolerated `expected violations remain`-style exits
(markdownlint-cli2 rc≤1, ansible-lint rc≤2) still count as OK; only
real failures count as FAIL.
- all workers run to completion regardless of individual tool failures
(so the user always sees the full picture); after the summary, the
script exits 1 if any worker reported FAIL.
* scripts/lint/sort_claude_settings.sh -> scripts/lint/claude.sh
(history preserved via git mv); update test_claude_settings_sorted.py
fix-hint accordingly.
Plus working-tree drift across roles/, group_vars/, inventories/,
requirements/, and .pre-commit-config.yaml accumulated from the
make-test setup phase regenerating generated artefacts during recent
test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Deploys KIX Start as a first-class web-app-* role behind
sys-stk-front-proxy + web-app-oauth2-proxy + Keycloak realm-level
2FA at kix.<DOMAIN_PRIMARY>. The 4-container upstream stack
(proxy, backend, frontend, db) reuses the shared svc-db-redis for
OAuth2-proxy session storage and KIX cache.
Auth chain:
- OAuth2-proxy `allowed_groups` admits members of either
`/roles/web-app-kix/administrator` or `/roles/web-app-kix/user`
(the user role is declared via `meta/rbac.yml`).
- A custom kix-proxy nginx default.conf is bind-mounted (the
upstream image ships an empty one) and forwards everything
through the frontend Node server, plus copies the OAuth2-proxy
X-Forwarded-User into a Remote-User upstream header.
- KIX `Auth::LDAP` + `Auth::Sync::LDAP` against svc-db-openldap
with one backend per role group (admin GroupDN and user GroupDN).
KIX UsageContext must be a single-string value; comma-list
silently disables the backend.
Validated end-to-end via deploy-fresh-purged-apps APPS=web-app-kix
FULL_CYCLE=true (Pass 1 + Pass 2 both failed=0). Playwright 3/3
in both passes:
- TLS+HSTS at the canonical URL.
- administrator full flow: dashboard -> KIX card -> OAuth2-proxy
-> Keycloak -> KIX-LDAP login form -> KIX UI -> universal logout
-> re-anonymous redirect back to the OAuth2-proxy gate.
- biber full flow: same, with a Keycloak admin REST API fixture
that puts biber into /roles/web-app-kix/user (mirrors what an
operator does manually via the Keycloak UI).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5