Adds 7 new Playwright scenarios covering helpdesk-core workflows and
infrastructure regressions:
- test-csp-headers — CSP response + meta-tag parity
- test-alias-domain — server-name alias serves 2xx, not 30x
- test-rest-api-ticket-create — POST /api/v1/tickets via Basic auth
- test-ticket-create-as-customer — biber SPA → fill form → submit
- test-ticket-reply-as-agent — API-seeded ticket; admin posts a reply
- test-websocket-realtime — open ticket; API appends article;
expect realtime push into the open SPA
- test-mail-to-ticket — raw-socket SMTP to Mailu helpdesk
mailbox; poll Zammad until ticket
appears
Auto-configures Zammad's Email channel (inbound IMAP + outbound SMTP)
against Mailu via a new post-bootstrap rails-runner step. The Ruby for
both bootstrap steps lives in standalone files and is piped through
`container exec -i ... bundle exec rails r -`; the previous wrapper
shell scripts (`apply_*.sh`) are removed in favour of inline `shell:`
calls in tasks/01_core.yml. Env vars are forwarded into the container
via `container exec -e VAR ...` so the Ruby uses `ENV.fetch` and stays
value-free / portable.
vars/main.yml grows three constants:
- ZAMMAD_OMNIAUTH_INIT_CONTAINER_PATH — bind-mount target inside the
container (was hard-coded four times in compose.yml.j2)
- ZAMMAD_HELPDESK_USER / _EMAIL / _PASS — derived from the `helpdesk`
mail-bot user provisioned via meta/users.yml
- ZAMMAD_MAIL / ZAMMAD_EMAIL_ENABLED — derived from the central email
lookup + the services.email gate
playwright.env.j2 grows MAIL_SMTP_HOST/PORT/USER/PASS + HELPDESK_EMAIL
for the SMTP injection in the mail-to-ticket scenario, sorted ASC per
the env-key lint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Playwright spec split into per-test partials, mirroring the web-app-kix
layout (lint-conform: top-level `playwright.spec.js` requires the
partials and shared module):
- `files/playwright/_shared.js` — env vars, `signInViaZammadOidc`,
`zammadLogout`, `beforeEach`
- `files/playwright/test-landing.js`
- `files/playwright/test-login-administrator.js`
- `files/playwright/test-login-biber.js`
- `files/playwright/test-guest-persona.js`
Variable dedup in `vars/main.yml`:
- `ZAMMAD_VOLUMES_DIR` — single base path
- `ZAMMAD_AUTO_WIZARD_HOST_PATH` — derived
- `ZAMMAD_OMNIAUTH_INIT_HOST_PATH` — derived (was repeated 4x in compose)
- `ZAMMAD_OIDC_ENABLED` — single source for the OIDC gate
All consumers in `tasks/01_core.yml`, `templates/compose.yml.j2`,
`templates/auto_wizard.json.j2`, `templates/playwright.env.j2` now read
the constants instead of inlining the `lookup(...)` expressions.
Drive-by: `meta/main.yml` description narrowed to describe Zammad itself
(role-specific text moved out of the upstream-facing galaxy_info field).
`make autoformat` clean, `make test` green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`make autoformat` line-wraps the api_token credential description in
meta/schema.yml at the standard yamllint column. No semantic change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`web-app-mailu/tasks/03_manage_user.yml` and `03b_create_user_token.yml`
run `compose exec admin flask mailu user <...>` and `compose exec admin
curl http://localhost:8080/api/v1/token` immediately after the existing
wait blocks in `02_wait_for_database.yml` (mariadb domain table + admin
flask CLI ready). Under CI load the Mailu admin container is reachable
on the compose network, but two further runtime preconditions are not
guaranteed when those tasks fire:
1. The flask CLI imports succeed (`flask routes` returns 0) before
Mailu's admin code has finished establishing the lazy Redis
connection that `flask mailu user` needs to commit the user.
Symptom (run 26480944120 job 77980186049, infinito-nexus/core
run 26480946718 job 77980074367, web-app-suitecrm):
redis/connection.py:639 ConnectionRefusedError: [Errno 111]
during `flask mailu user no-reply infinito.example -- <pw>`.
2. The admin gunicorn HTTP server on :8080 is not yet listening when
`03b_create_user_token.yml` curls `http://localhost:8080/api/v1/
token`. Symptom (run 26480946718 job 77980074254, web-app-fediwall):
curl: (7) Failed to connect to localhost port 8080 after 0 ms.
Add two further wait blocks after the existing flask-routes wait, both
following the same probe/rescue pattern (300-line container log dump
on timeout) the file already establishes:
* Redis: `compose exec -T redis redis-cli ping` until stdout
contains "PONG". Matches Mailu's own healthcheck command verbatim
(svc-db-redis/templates/service.yml.j2 line 24) and depends on no
extra auth -- Mailu's env.j2 declares no REDIS_PASSWORD and the
injected svc-db-redis service runs with no `--requirepass`.
* Admin HTTP: `compose exec -T admin curl -sS -o /dev/null -w
'%{http_code}' http://localhost:8080/sso/login` until rc=0 and the
captured status code matches `^[1-5][0-9][0-9]$`. Any HTTP response
(200 / 302 / 404 / 5xx) proves gunicorn is bound; the regex form
makes the check robust against an upstream rename of the SSO
endpoint.
Restructure for clarity:
* Split the original `02_wait_for_database.yml` into
`02_wait_for_mariadb.yml` (mariadb domain table + flask CLI ready;
flask init touches the DB) and `03_wait_for_redis.yml` (Redis PING
+ admin HTTP :8080; the HTTP API requires Redis-backed sessions).
* Bump every subsequent task file by one step
(03_manage_user -> 04_manage_user, 04_generate-... -> 05_..., etc.)
so the directory order still encodes execution order at a glance.
* Update the cross-`include_tasks` references in `01_core.yml`,
`04_manage_user.yml` and `04a_manage_user_token.yml`. Also bump
the trailing comment in `01_core.yml` that pointed at
`03_manage_user.yml` to the new `04_manage_user.yml`.
* Hoist the recurring `lookup('container', application_id,
'directories.instance')` (12 occurrences plus a 6x indirection via
a per-include `MAILU_DOCKER_DIR` alias previously defined in
`01_core.yml`) into a single `MAILU_INSTANCE_DIR` constant in
`vars/main.yml`, matching the existing `MAILU_VOLUMES_DIR` shape.
All `chdir:` values now resolve to the same string with no extra
indirection.
`make autoformat` and `make test` (188 tests, 158s) pass on the
restructured tree.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds roles/web-app-zammad/ — a brand-new web-app role deploying Zammad as
the Infinito.Nexus helpdesk with direct OpenID Connect SSO against the
shared Keycloak client. Tracks requirement 022.
Stack: ghcr.io/zammad/zammad 6.5.0 (init / railsserver / websocket /
scheduler / nginx, all driven by upstream command modes), bundled
elasticsearch 8.13.4 + bundled memcached (no central svc role exists),
central svc-db-postgres reused via lookup('database', application_id).
Variant matrix mirrors web-app-kix: V1 oidc+ldap, V2 all-false, V3 ldap-only.
Bootstrap flow (tasks/01_core.yml):
- Stage auto_wizard.json into the role's volumes dir AFTER sys-stk-full
creates the directory; bind-mount it into zammad-init so the upstream
rake zammad:setup:auto_wizard bypasses the setup UI on first boot.
- Stage zz_omniauth_allow_get.rb initializer + bind-mount into rails /
websocket / scheduler / nginx, since the Zammad SPA's SSO link is a
plain GET and omniauth 2.x forbids GET by default.
- ALTER ROLE <zammad-db-user> WITH CREATEDB so Zammad's unconditional
`rake db:create` doesn't trip on insufficient privilege when the
database is pre-created by svc-db-postgres.
- Apply post-bootstrap settings via apply_oidc_settings.sh
(rails-runner inside zammad-railsserver): fqdn, http_type so Zammad
emits the correct redirect_uri to Keycloak; auth_openid_connect +
auth_openid_connect_credentials so the SSO button works.
Workarounds embedded in compose.yml.j2:
- zammad-nginx command override that copies /docker-entrypoint.sh to
/tmp and patches `grep "nameserver"` to `grep -E "^nameserver"`,
because Docker writes comment lines like `# Overrides: …` into
/etc/resolv.conf that the upstream grep matches as substrings,
yielding an invalid `resolver … Overrides: …` directive nginx rejects.
- DATABASE_URL pre-populated in env.j2 so post-bootstrap
`container exec rails r` calls (which bypass the entrypoint) can
load Rails without database.yml.
Playwright (files/playwright/playwright.spec.js, single-file lint-conform
layout): landing-reachable + administrator OIDC + biber OIDC + guest
public-landing scenarios. Personas skip-correctly via
skipUnlessServiceEnabled('oidc') under V2 / V3.
V1 end-to-end green: admin + biber Playwright pass through the full
Keycloak SSO round-trip. V2 (all-false) and V3 (ldap-only) green on
baseline reinstall with full_cycle=true.
Follow-ups indexed in roles/web-app-zammad/TODO.md: Zammad LdapSource
auto-create for V3, mail channel auto-config + end-to-end mail-to-ticket
test, Keycloak-group-to-Zammad-role mapping (Zammad doesn't ship a
built-in claim-to-role translator — currently all OIDC logins default to
Customer per Decision #5 fallback), Playwright assertion for the
`zammad.helpdesk.*` server-name alias.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`startMattermostSsoFlow` polled `/login` in a `while (true)` loop with
`waitUntil: "domcontentloaded"` and a 10s `isVisible` window for
`#input_loginId`. Under CI load Mattermost v11+'s React bundle takes
15-30s to parse and mount the login form. Each loop iteration's
`page.goto` aborted the previous in-flight hydration, so the SPA
restarted from scratch on every retry and the form mounted only when
the iteration's 10s window happened to align with a hydration that
slipped through.
Run 26451797037 job 77878640850 (web-app-mattermost ubuntu) timed out
at 300s in this state. The Playwright trace shows biber's loop ran 80+
iterations over 156s before `#input_loginId` finally became visible;
admin's loop then had only 132s of budget left, never broke out, and
the test hit the outer `testTimeout: 300_000`. Both contexts'
`error-context.md` page snapshots confirmed the form was present at
the moment of timeout, ruling out an actual Mattermost-side failure.
Replace the loop with a single navigate sequence:
* `waitUntil: "load"` waits for the full `load` event, giving the
React bundle time to parse and run instead of returning at
`domcontentloaded` before any of it has executed.
* One explicit `page.locator(...).waitFor({ visible, timeout: 60_000 })`
spans the observed 15-30s hydration tail with comfortable headroom.
* The wrapping `for (attempt = 0; attempt < 2; ...)` adds exactly one
retry on transient navigation failures (network blip, mid-deploy
container restart) without reintroducing the abort-self race the
bare loop had — only one in-flight hydration at a time.
The PASS 1 (sync) path that already succeeded in the failing run keeps
working because the new control flow is strictly simpler than the
loop. The failing PASS 2 (async) path now has a deterministic upper
bound: at most two 60s navigations + two 60s `waitFor`s per call site
= 240s ceiling per `startMattermostSsoFlow`, comfortably below the
test-level 300s timeout even with biber + admin both invoking it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`web-opt-rdr-www` generates a redirect vhost
`<NGINX_DIR>/conf.d/servers/{http,https}/www.<domain>.conf` for every
deployed app's canonical and alias domains. Those redirect vhosts are
not part of any app's `server.domains.{canonical,aliases}` list, so the
entity-keyed purge primitive in `utils/cleanup/nginx_vhosts.py` did
not yield them and `iter_vhost_files_for_entity` left them on disk.
Between matrix-deploy rounds this leaked the redirect vhosts of the
previous round's broader app set into the next round's narrower one.
On run 26451828677 job 77879571696 (web-opt-rdr-www on debian) the
round 1 -> round 2 transition
round 1/2 inv=github-0 apps=[15 apps incl. web-svc-cdn,
web-svc-simpleicons, web-svc-css, ...]
round 2/2 inv=github-1 apps=[web-opt-rdr-www]
dropped the bare `cdn.infinito.example.conf` etc. via the existing
purge but kept the matching `www.cdn.infinito.example.conf`,
`www.icon.infinito.example.conf`, ... behind. Round 2 only redeploys
web-opt-rdr-www, whose `www_redirect_domains` lookup now only emits
its own `www.w3redirect.<primary>` redirect, so nothing reissues the
stale www-prefixed vhosts.
`sys-ctl-hlth-csp` discovers domains by scanning .conf filenames in
the nginx servers dir, so it still probed the stale `www.<x>.conf`
hosts. nginx served them with a cert whose SAN no longer covered the
hostname (the per-domain cert path baked into the conf had been
removed alongside the bare vhost), the TLS handshake returned
`net::ERR_SSL_UNRECOGNIZED_NAME_ALERT`, the CSP probe exited 11, and
`sys-service-terminator : Apply state` failed after 10 retries.
Extend `iter_vhost_files_for_entity` to also yield the
`www.<domain>.conf` variant for every domain it already yields,
skipping domains that already begin with `www.` so a role that
declares its canonical with a leading `www.` (web-opt-rdr-www's own
`www.w3redirect.<primary>`) does not produce `www.www.<x>.conf`. The
existing two- and three-step purge primitives (db / compose / dir /
nginx) and the orchestrator in `scripts/container/purge/apps.sh` keep
their entity-keyed contract; only the per-entity yield set widens.
Coverage:
- test_yields_www_redirect_variant: per-app www-prefixed file is
yielded alongside the bare vhost.
- test_www_prefixed_domain_not_double_prefixed: a www-prefixed
canonical does not synthesize `www.www.<x>` even when such a file
exists on disk (defense in depth against a future regression that
prepends unconditionally).
- test_removes_www_redirect_variant: end-to-end purge removes both
bare and www variant, leaves an unrelated www-prefixed file alone.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The include_role vars block referenced `_load_app_id`, a host fact
created by `set_fact` at the top of load_app.yml. Host facts live in
the non-persistent fact cache (vars/manager.py:405) and survive across
include frames, so every nested `include_tasks: utils/load_app.yml`
call overwrote `_load_app_id` for the whole play.
Inside the coturn deploy chain (web-svc-coturn -> sys-stk-full ->
sys-stk-backend -> sys-svc-compose), sys-svc-container.Setup Swapfile
triggers a nested load_app.yml(svc-opt-swapfile) BEFORE
sys-svc-compose/04_files.yml renders coturn's env.j2. The nested call
set_fact'd `_load_app_id='svc-opt-swapfile'` and left it as a host
fact. When env.j2 lazy-evaluated `COTURN_REALM -> domain ->
lookup('domain', application_id)`, the outer include_role's vars block
resolved `application_id` through `{{ _load_app_id }}` and read the
mutated value, yielding
lookup('domain'): failed to resolve domain for 'svc-opt-swapfile':
application_id 'svc-opt-swapfile' not found in domains mapping
(job 77879571693 on run 26451828677, debian, play 3)
The fix is two-step:
1. Switch the include_role to reference `load_app_id` directly.
`load_app_id` is the caller's `include_tasks vars:` argument,
attached to the TaskInclude object as task-scoped vars (Level 414
include params in vars/manager.py). It walks the `_parent` chain
via `task.get_include_params()`, and `_ACTION_ALL_INCLUDES`
(constants.py:101) covers both include_tasks and include_role, so
the outer TaskInclude's vars stay reachable from any depth. Each
nested include_tasks frame has its own TaskInclude.vars and cannot
mutate the outer frame's `load_app_id`.
2. Remove the `set_fact: _load_app_id` step entirely. The host fact
was the only mutation source for this variable in load_app.yml,
and dropping it eliminates the bug class for the whole loader.
`grep` across the repo confirms `_load_app_id` is unused elsewhere
and no role does `set_fact: application_id|service_role|service_item
|load_app_id`, so the include-params resolution chain stays immune
to host-fact races.
The 7 current callers of load_app.yml are verified safe:
- 6 pass literal strings (sys-svc-webserver-core, web-app-bluesky,
sys-stk-front-base, web-opt-rdr-www, sys-svc-container,
web-app-dashboard)
- 1 (sys-service-loader/load_service.yml) passes
`"{{ service_role }}"`, which resolves via outer-loop-scoped
`service_item.role` and is itself only ever set via include-tasks
vars or loop_control, never via set_fact.
Reuses the same defense-in-depth direction as commits f32266d53
(tls lookup defensive templating) and 7b9cb5617
(current_play_redirect_domains defensive templating): close the path
where a mutable side channel feeds into a load-bearing lookup.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New requirement for a `web-app-zammad` role that integrates Zammad as
the Infinito.Nexus helpdesk with OIDC-direct SSO against Keycloak.
Captures 11 operator-confirmed design decisions before implementation:
- Canonical `helpdesk.infinito.example` + true server-name alias
`zammad.helpdesk.infinito.example` (no 301 redirect).
- SSO flavor: OIDC-direct via Zammad's native OmniAuth strategy
(no oauth2-proxy sidecar). Uses the legacy `services.oidc.*` shape
(pre-021); does not block on 021.
- Keycloak OIDC client auto-provisioned via `web-app-keycloak`.
- Keycloak group -> Zammad role mapping (Admin/Agent/Customer) if
Zammad's OIDC strategy supports group claims natively; otherwise
documented limitation.
- Upstream `ghcr.io/zammad/zammad` image pinned to latest stable
semver. Self-built image (a la 015 Moodle) explicitly out of scope.
- Central-service reuse: `svc-db-postgres`, `svc-db-redis`,
`svc-db-memcached`. Elasticsearch bundled in-role (no central
`svc-db-elasticsearch` exists today).
- Email in scope: SMTP outbound via `sys-svc-mail-smtp` /
`web-app-mailu`, IMAP inbound (mail-to-ticket) auto-wired.
- Setup wizard bypassed on first deploy via env-vars + post-start
API call so a fresh deploy lands a ready-to-use instance.
- Playwright biber + administrator personas in scope (per 019).
- `meta/variants.yml` defines three variants mirroring web-app-kix:
V1 oidc+ldap, V2 all-false, V3 ldap-only.
Acceptance criteria are grouped per concern (routing/TLS, role
layout & image, central-service reuse, SSO/OIDC, LDAP, email,
first-admin bootstrap, variants, playwright, health & quality,
documentation) so each item is independently verifiable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ran `python -m cli.contributing.requirements.archive` after marking all
remaining acceptance criteria as `[x]` in:
- 012 - Package Cache (Nexus 3 OSS)
- 013 - Sub-Beta-to-Beta Lifecycle Promotion for `web-*` Roles
- 015 - OpenCloud and OpenTalk IAM integration
- 015 - Self-built Moodle image
Their headings are appended to the `## Archive` block of
`docs/requirements/README.md` and the per-requirement files are deleted.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Ran `python -m cli.contributing.requirements.archive` to move the
following fully-checked-off requirement files into the `## Archive`
block of `docs/requirements/README.md` and delete the per-requirement
files:
- 001 - Nextcloud Talk TURN relay
- 002 - Service Registry Refactoring
- 003 - Reduce `applications` and `users` to lookup-only access
- 004 - Generic RBAC role auto-provisioning via OpenLDAP
- 005 - WordPress Multisite support for OIDC-driven RBAC
- 006 - Service-gated Playwright tests
- 007 - WordPress to Discourse post round-trip
- 015 - KIX Service Management integration
- 016 - web-app-hugo
Cross-references from the still-open requirements (015-moodle,
015-opencloud, 017, 018, 019, 021) rewritten from the deleted file
paths to \`README.md#archive\`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* TC003: move `pathlib.Path` into `TYPE_CHECKING` blocks across the
new `requirements/archive` package and its completeness lint test.
* TC001: same treatment for `VaultHandler` in
`credentials/emit.py` (annotation-only).
* SIM105/S110: replace `try/except/pass` in
`current_play_redirect_domains.py` with
`contextlib.suppress(Exception)`.
* `archive/__main__.py`: mark `README_PATH.read_text(...)` with
`# nocheck: cache-read` — CLI rewrites the README in the same flow,
mirroring the `changelog/archive` pattern.
* Drive-by ruff-format condensations in the archive unit tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* `python -m cli.contributing.requirements.archive` moves the H1 of
every fully-checked `docs/requirements/NNN-*.md` into the
`## Archive` block of the directory README and deletes the file.
Files with any remaining `- [ ]` are skipped. Split SRP-wise into
discovery / inspect / readme / __main__.
* New lint `test_requirements_completeness` warns per unchecked
acceptance criterion (GitHub annotation) and fails when any
requirement is ready for archival, pointing at the CLI. Keeps the
requirements directory small so AI agents process less context.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The plugin's previous
domain_primary = str(vars_.get("DOMAIN_PRIMARY") or "")
domain_homepage = str(vars_.get("DOMAIN_HOMEPAGE") or "")
read inventory values that may carry Ansible 2.18+'s
``_AnsibleTaggedStr`` with the ``TrustedAsTemplate`` tag — exactly
the shape ``inventories/development/default.yml:20`` produces with
DOMAIN_PRIMARY: "{{ lookup('env', 'INFINITO_DOMAIN') }}"
A bare ``str(value)`` casts the tag away and leaves the literal Jinja
text in the plugin output. The output then flows into
``CURRENT_PLAY_REDIRECT_DOMAINS`` via the set_fact in
``tasks/stages/01_constructor.yml`` and lands in
``roles/web-opt-rdr-domains/tasks/main.yml`` loop items as
item={'source': "{{ lookup('env','INFINITO_DOMAIN') }}", 'target': "{{ … }}"}
The ``lookup('tls', domain, 'enabled')`` call in
``roles/web-opt-rdr-domains/tasks/redirect-domain.yml:5`` then
receives the literal Jinja string and raises
The lookup plugin 'tls' failed: tls: application_id
'{{ lookup('env', 'INFINITO_DOMAIN') }}' not found in domains mapping
(job 77797371504 on run 26428080957). That CI ran on SHA d1410c0be,
which already shipped this plugin — so the upstream replacement of
the inline two-stage set_fact merges with the lookup plugin did not
fix the underlying trust-tag-strip bug; it just relocated it from
Jinja to Python.
Add ``_resolve_str`` that runs the value through ``self._templar``
before string-coercion. Idempotent for already-resolved strings,
exception-safe (falls back to ``str(raw)`` if the templar is
unavailable, e.g. on hosts without rendered vars). Both
``DOMAIN_PRIMARY`` and ``DOMAIN_HOMEPAGE`` route through it.
Defense-in-depth with the existing ``plugins/lookup/tls.py``
defensive templating (commit f32266d53): even if a future code path
hands ``lookup('tls', …)`` an untemplated string, the tls plugin
itself now templates the first term. Either layer alone suffices for
the rdr-domains failure; both together guarantee the symptom cannot
re-surface from a single regression elsewhere.
Coverage: two regression tests in
``tests/unit/plugins/lookup/test_current_play_redirect_domains.py``
— ``test_jinja_tagged_domain_primary_is_templated`` proves the
templar call happens and produces the resolved domain, and
``test_templar_failure_falls_back_to_raw_value`` proves a raising
templar does not crash the plugin. The 10 existing tests stay green.
``make test`` green across all 5 targets.
Class follow-up: 11 further lookup plugins read inventory values
without ``templar.template`` first (see git log for ``vars_.get``
call sites). Not in scope for this fix; logged for a separate
audit pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Six ``uri_retry`` tasks under ``roles/web-app-xwiki/tasks/`` carried
their XML payload as inline ``body: |`` blocks (CheckExtension Groovy
page, InstallExtensions Groovy page, the administrator user-page
PUT/POST pair, and the post-Fix-#8 XWikiAdminGroup PUT/POST pair).
Inline XML mixes Ansible YAML indentation with Jinja control flow
(``{% raw %}``-escaping for ``{{groovy}}`` delimiters), defeats
editor XML support, and silently rots when a future contributor
copy/edits the snippet without realising the surrounding ``body: |``
context.
Split by concern:
- ``templates/xml/*.xml.j2`` — six XML files that embed Jinja:
``user_page``, ``user_xwikiusers_object``,
``xwiki_admin_group_page``, ``xwiki_admin_group_member_object``,
``install_extensions_page``, ``main_webhome_page`` (the last one
was already a template; moved here for symmetry with the new five).
- ``files/xml/check_extension_page.xml`` — fully static; rewritten
without ``{% raw %}{{groovy}}{% endraw %}`` because the file is
served via ``lookup('file', …)`` (no Jinja pass), so ``{{groovy}}``
reaches XWiki literally.
Task references migrate to ``body: "{{ lookup('template',
'xml/<name>.xml.j2') }}"`` (or ``lookup('file', 'xml/<name>.xml')``
for the static case), normalising the previously bespoke long-form
``roles/web-app-xwiki/templates/main_webhome_page.xml.j2`` lookup in
the bargain.
SPOT-extract the magic string ``XWikiAdminGroup``:
- ``XWIKI_ADMIN_GROUP_PAGE: "XWikiAdminGroup"`` in vars/main.yml,
documented as the page-name part used by REST URLs, the
XAR-shipped page, the LDAP ``group_mapping`` argument, and the
OIDC ``groups.mapping`` argument.
- Five call sites updated to ``{{ XWIKI_ADMIN_GROUP_PAGE }}`` or
bare ``XWIKI_ADMIN_GROUP_PAGE`` inside list contexts:
``tasks/03_administrator.yml`` (×4 — two task names, the PUT URL,
the POST URL), ``templates/xml/xwiki_admin_group_page.xml.j2``,
``templates/compose.yml.j2`` (LDAP group_mapping), and
``templates/xwiki.properties.j2`` (OIDC groups.mapping, in a
commented example).
Lint guard: new ``tests/lint/ansible/tasks/test_no_inline_xml_body.py``
scans every ``roles/`` and ``tasks/`` YAML for ``body: |`` (or
``body: >``) folded scalars whose first content line starts with
``<`` and fails with a pointer at the matching ``templates/xml/...``
or ``files/xml/...`` alternative. JSON / shell / SQL bodies stay
allowed. Per-task opt-out: ``# nocheck: inline-xml-body`` on the
``body:`` line or the line above. Five unit tests cover the
scanner: matches XML body, ignores JSON body, honours same-line
nocheck, honours line-above nocheck, repo-wide sweep clean after
this refactor.
XWiki v17.10.8 distribution-merge behaviour was cross-checked
against upstream (``AbstractXarEntryType.upgradeType =
THREEWAYS``); the empty ``<member/>`` placeholder in the
XAR-shipped ``XWikiAdminGroup`` cannot overwrite our explicitly-added
member (different GUID / number).
``make test`` green across all 5 targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`XWIKI | Ensure user is in XWikiAdminGroup` POSTed
`<XWiki.XWikiGroups>{<member>XWiki.administrator</member>}</…>` to
`POST /rest/wikis/xwiki/spaces/XWiki/pages/XWikiAdminGroup/objects`,
hit HTTP 404 31 retries in a row, and aborted the deploy (job
77797115264 on run 26428083128, debian/no-SSO variant).
XWiki's REST `…/pages/<name>/objects` resource resolves the parent
page first; a 404 means the page document does not exist yet. The
admin-group page ships in
`xwiki-platform-distribution-ui-base/.../XWikiAdminGroup.xml` and is
materialised by the XWiki Distribution Job on first boot. The
Distribution Job did not finish materialising it before the
administrator-bootstrap step fired in this run — log evidence:
`XWiki.administrator` user-page POST 201 at 08:24:41 (line 487473),
group POST 404 at 10:09:27 (line 487745), no
`XWikiAdminGroup` PUT/import in between.
Mirror the pattern already used two tasks above for the user page:
PUT-create `XWiki.XWikiAdminGroup` (idempotent — returns 200/202 when
the Distribution Job won the race, 201 when we beat it) before POSTing
the membership object. The new PUT + the existing POST live inside a
single `block:` gated by `not (XWIKI_SSO_ENABLED | bool)` — the
existing `XWIKI_SSO_ENABLED` SPOT in vars/main.yml subsumes the
duplicated `not LDAP and not OIDC` clause on the POST task.
Distribution-Job-overwrite risk assessed against XWiki v17.10.8
upstream source:
- `AbstractXarEntryType.upgradeType` defaults to
`UpgradeType.THREEWAYS` (3-way merge, not OVERWRITE/SKIP). Verified
at `xwiki-platform-xar-model/.../type/AbstractXarEntryType.java:38`.
- `DocumentMergeImporter.importDocument` passes the existing document
as merge-current; the XAR template ships an empty `<member/>`
placeholder with a hardcoded GUID
(`46d387f2-4d8d-446d-a618-8bfcb655ba18`), so our REST-created
XWikiGroups object (auto-assigned number, fresh GUID) is treated as
a separate object and survives a later XAR re-install. The empty
placeholder also means the XAR install has no member value to
overwrite with.
- Combined risk: ~5% that a future XWiki release changes the upgrade
type or the XAR-shipped XWikiAdminGroup content; logged for review
but not blocking.
Confidence: ~92%. Remaining ~8% covers (a) future XWiki XAR re-runs
with non-default conflict actions, (b) custom merge-conflict configs
in the deployed flavor, (c) unbedachte XWiki internal listeners.
Variant scope: no-SSO only (`XWIKI_SSO_ENABLED == false`). The OIDC
and LDAP variants delegate group membership to the SSO mapper and
skip this task entirely, so the fix is a no-op there.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The inventory validator at cli/administration/inventory/validate
walked `recursive_keys(host_vars) ⊆ recursive_keys(application_defaults)`
and flagged any extra key with "Missing default for <app>: <key>".
After commit 10cb02461 ("bake only variants.yml overrides into
matrix-deploy --vars") host_vars carry every key the active
variants.yml overlay declares. Since 29377e1bd
(`_build_application_defaults` reads only `meta/<topic>.yml`, not
variants.yml — to keep the variant-0 service-key leak closed), the
defaults' allow-set is artificially narrower than the role contract:
roles/web-opt-rdr-www declares `services.dashboard.{enabled,shared}`
exclusively in `meta/variants.yml`, so the validator rejects the
baked host_vars with three "Missing default" entries before
ansible-playbook even starts (job 77797115291 on run 26428083128,
distro=debian; distro-agnostic per the matrix).
Widen the validator's legal-key set per app to
recursive_keys(defaults[app]) ∪ ⋃ recursive_keys(variants[app][i])
variants.yml IS part of the role contract; the validator's previous
"defaults only" view was incomplete. Runtime
`_build_application_defaults` stays narrow (so 29377e1bd's
variant-0-leak fix stays in place) — only the validator widens. Typos
not declared by any variant remain caught: a new
`test_key_typo_in_variants_still_rejected` unit test pins that.
Package split (KISS/DRY/SPOT), mirroring the credentials/ refactor:
`validate/__main__.py` carried six orthogonal concerns plus
orchestration in 175 LoC. Split:
- `validate/keys.py`: `recursive_keys` (pure helper).
- `validate/loaders.py`: `load_yaml_file` (with vault-block stripping)
and `load_inventory_files` (directory scan).
- `validate/applications.py`: `compare_application_keys` (the widened
variants-aware logic).
- `validate/users.py`: `compare_user_keys`.
- `validate/hosts.py`: `validate_host_keys`.
- `validate/__main__.py`: thin orchestrator — argparse + load defaults/
variants/users + invoke compare_* + exit code. 75 LoC.
`validate/__init__.py` re-exports the public API
(`main`, `compare_application_keys`, `compare_user_keys`,
`load_inventory_files`, `load_yaml_file`, `recursive_keys`,
`validate_host_keys`, `PROJECT_ROOT`). External callers
(`tests/unit/cli/administration/inventory/validate/test_main.py`,
`tests/unit/cli/administration/inventory/validate/test_hosts.py`)
keep working unchanged via this re-export.
`__main__.py` uses ABSOLUTE imports (`from cli.administration.inventory.
validate.applications import ...`) instead of relative because it's
invoked as a subprocess by tests and by the deploy entrypoint; the
sys.path bootstrap at the top of the file makes `cli.…` resolvable
in that mode.
Test layout follows the split — one test file per module:
- `test_keys.py` (6 tests: empty/non-dict/flat/nested/deep/prefix).
- `test_loaders.py` (8 tests: vault stripping, missing file, broken
YAML, top-level + recursive directory scan, empty-block skip).
- `test_applications.py` (9 tests: all-keys-match, unknown app, extra
key, credentials skip, variants legal, variants union, no-variants
fallback, source path in error, typo still rejected).
- `test_users.py` (5 tests: match, unknown user warn, missing default,
password skip, credentials skip).
- `test_hosts.py` (renamed from test_inventory_hosts.py — unit tests
for `validate_host_keys`).
- `test_main.py` (renamed from test_inventory.py — end-to-end
subprocess tests for `main()`, including the
`test_variants_only_key_is_accepted` regression guard for this fix).
Coverage: 36 tests, all green. `make test` green across all 5 targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mattermost v11.7.1's `Root.tsx` redirects every fresh browser context
that lacks `localStorage.__landingPageSeen__` to `/landing#/` (the
"open in app vs view in browser" splash) — verified against upstream
source webapp/channels/src/components/root/root.tsx:146 and the key
name in webapp/channels/src/utils/constants.tsx:808 at tag v11.7.1.
The redirect is purely client-side (React Router `history.push`), so
localStorage stamped via Playwright's `addInitScript` before the very
first navigation suppresses it.
The bounce broke job 77797115305 on run 26428083128 (fedora distro):
`test-biber-dm-administrator` opens two fresh `browser.newContext()`s
that go through `startMattermostSsoFlow`. The first SSO test passes
(default `page` fixture inherits no relevant state either, but the
admin user's first provisioning seemingly papers over the missing
flag for that one context). The DM test's second context bounces to
`/landing#/` indefinitely, the existing 120s deadline + outer 300s
test budget expire across all three retries, and the role-runner
reports `Playwright failed for roles: ['web-app-mattermost']`.
Adds a single `page.addInitScript` in `startMattermostSsoFlow` that
writes `localStorage.__landingPageSeen__ = "true"` on every document
load. The existing bounce-loop stays as a safety net for the case
where Mattermost ever moves the flag — the fix is purely additive.
Verification:
- Flag name `__landingPageSeen__`: `StoragePrefixes.LANDING_PAGE_SEEN`
in upstream webapp/channels/src/utils/constants.tsx:808.
- Redirect gate: `BrowserStore.hasSeenLandingPage()` short-circuits
the redirect at root.tsx:146 — `localStorage.getItem(...)` on the
same key.
- Playwright contract: `addInitScript` evaluates on every page load
before any page script (per Playwright docs).
- Confidence: ~95%. The remaining 5% covers (a) `appDownloadLink`
unset (redirect never fires — irrelevant) and (b) Playwright
ordering edge cases (standard pattern, low risk).
The clean architectural follow-up (Playwright `storageState` reuse
to skip the second SSO round-trip entirely) lives elsewhere; this
commit is the focused unblocker.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The credentials baker at cli/administration/inventory/credentials/__main__.py
walked `schema_creds.items()` one level deep and called `str(value)` on
every leaf. For nested credential groups like
``credentials.recaptcha = {key, secret}`` (declared in
roles/web-app-espocrm/meta/schema.yml and
roles/web-app-listmonk/meta/schema.yml) this collapsed the inner dict
into its Python repr, vault-encrypted that blob, and wrote it as a
single scalar at ``credentials.recaptcha``. The runtime ``config``
lookup then choked on the descent into ``recaptcha.key``:
The lookup plugin 'config' failed: Expected dict for 'key', got
_AnsibleTaggedStr at 'key'
Full path so far: applications['web-app-espocrm'].credentials.recaptcha.key
Current object: "{'key': '{{ CAPTCHA.RECAPTCHA.KEY }}', 'secret':
'{{ CAPTCHA.RECAPTCHA.SECRET }}'}"
(jobs 77797371397 espocrm and 77797371442 listmonk on run 26428080957.)
Introduce ``emit_credentials`` (new module: ``credentials/emit.py``)
that walks the schema recursively. Scalar leaves still vault-encrypt
via ``to_vault_block``; dict nodes recurse into a nested CommentedMap
via ``ensure_map``. The walker also tracks dotted full-key paths so
the existing ``--force``-mode override-apply logic sees the same dotted
keys it always saw.
Auto-healing for previously-corrupted host_vars: ``ensure_map``
overwrites non-CommentedMap placeholders, so the next bake replaces a
stringified ``credentials.recaptcha`` blob with the correct nested
structure. Operators with corrupted entries simply re-run the
credentials generator; no manual cleanup needed.
Package refactor (KISS/DRY/SPOT):
The 450+ LoC ``__main__.py`` was carrying four orthogonal concerns
(vault, --set parsing, recursive emission, prompts) plus orchestration.
Split by concern:
- ``credentials/vault.py``: ``is_vault_encrypted``,
``is_ruamel_vault``, ``to_vault_block`` (+ private helpers
``_vault_body``, ``_make_vault_scalar_from_text``).
- ``credentials/overrides.py``: ``parse_overrides``, ``override_for``.
- ``credentials/emit.py``: ``ensure_map``, ``emit_credentials``.
- ``credentials/prompts.py``: ``ask_for_confirmation``.
- ``credentials/__main__.py``: argparse + ``main`` + two thin mode
handlers (``_run_snippet_mode``, ``_run_default_mode``) + the
force-override helper.
``credentials/__init__.py`` re-exports the public API
(``main``, ``ask_for_confirmation``, ``parse_overrides``, etc.) so
external callers (notably ``tests/unit/cli/administration/inventory/
test_credentials.py``) keep working unchanged.
Test layout follows the split:
- ``tests/unit/cli/administration/inventory/credentials/test_vault.py``
(20 tests: vault detection, body extraction, scalar construction,
to_vault_block paths, plus the VaultHandler subprocess coverage).
- ``tests/unit/cli/administration/inventory/credentials/test_overrides.py``
(13 tests: parse_overrides + override_for incl. dotted nested keys).
- ``tests/unit/cli/administration/inventory/credentials/test_emit.py``
(10 tests: ensure_map placeholder replacement, recursive emission,
none → empty, skip_existing semantics, override precedence,
dotted-path tracking, the on-disk corruption recovery case).
- ``tests/unit/cli/administration/inventory/credentials/test_prompts.py``
(5 tests: ask_for_confirmation yes/no/garbage/case/whitespace).
``tests/unit/cli/administration/inventory/test_credentials.py`` keeps
only the end-to-end ``main()`` integration coverage (including
``test_nested_credentials_dict_recurses_instead_of_stringifying`` —
the explicit regression guard for the original failure mode).
Coverage: 47 new per-module unit tests + the 5 existing main()
integration tests, all green. ``make test`` green across all five
targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CI job 77797371504 on run 26428080957 crashed with:
tls lookup: application_id '{{ lookup('env', 'INFINITO_DOMAIN') }}'
not found in domains mapping
`DOMAIN_PRIMARY` is defined in inventories/development/default.yml:20
as `"{{ lookup('env', 'INFINITO_DOMAIN') }}"`. On Ansible 2.18+ that
string carries a `TrustedAsTemplate` tag (it is an
`_AnsibleTaggedStr`). The constructor at tasks/stages/01_constructor.yml:64
wrapped it in `[ DOMAIN_PRIMARY ] | flatten | map('string') | first`
— the explicit `str()` cast strips the trust tag. The resulting plain
string flowed into `CURRENT_PLAY_REDIRECT_DOMAINS.source` as the
literal text `{{ lookup('env', 'INFINITO_DOMAIN') }}`, was passed
through `domain: "{{ item.source }}"` in
roles/web-opt-rdr-domains/tasks/main.yml:14 (one render pass that
produces untrusted content), and finally reached `as_str(terms[0])`
in plugins/lookup/tls.py:39 unchanged. The tls plugin never templated
its arg, so `resolve_term` looked up the literal Jinja string in the
domains mapping and raised.
Two layers, both necessary:
1. plugins/lookup/tls.py:39 — defensively template `terms[0]` through
`self._templar` before string-coercion. Mirrors the idiom in
plugins/lookup/unit_name.py:33 and plugins/lookup/email.py:61. Any
future caller that hands the plugin an unresolved expression now
gets it resolved before the domains-map lookup.
2. tasks/stages/01_constructor.yml:64-65 — drop the
`[ X ] | flatten | map('string') | first` ritual for
`DOMAIN_PRIMARY` and `DOMAIN_HOMEPAGE`. Both are scalar strings in
every known inventory (group_vars/all/00_general.yml:41,43 +
inventories/development/default.yml:20); the over-defensive
wrapper was the trust-tag stripper. Direct reference preserves the
tag through `CURRENT_PLAY_REDIRECT_DOMAINS` and every downstream
consumer.
Downstream consumers audited; all transparent to `_AnsibleTaggedStr`
(it inherits from `str`):
- merge_mapping filter: content-based dict-key compare
- domain_mappings filter: takes `DOMAIN_PRIMARY` directly, no cast
- items2dict(key_name='source', value_name='source'): hashable
- generate_all_domains filter: `isinstance(item, str)` matches
- sys-stk-front-base `when: ... | length > 0`: only counts
- web-opt-rdr-domains -> tls lookup (the failing path): now templates
Coverage: new `test_term_with_unresolved_jinja_is_templated` in
tests/unit/plugins/lookup/test_tls.py verifies the lookup feeds its
first term through the templar. The 12 existing tls tests stay
green. `make test` green across all 5 targets.
Class follow-up (not addressed here): the
`| string` / `| map('string')` cast still strips `TrustedAsTemplate`
elsewhere if a future commit reintroduces it; 12 other lookup
plugins do not yet defensively template their args. A lint guard and
plugin-wide audit will come in separate PRs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pruned per project convention "no unnecessary code comments":
- module docstring: 23 lines → 1 line (the file name + symbol already
state intent; full incident write-up belongs in the commit log)
- wipe_database_entries docstring: 11 lines → 2 lines
- six inline rationale comments in tests/__main__ removed (test
names already encode the intent)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
After 10cb02461 switched `payload._resolve_variant_payloads` from
`get_variants` to `get_variant_overrides_only`, matrix-deploy bakes
only the raw `meta/variants.yml` slice into host_vars (typically just
`services:`). Lookups that read `variables['applications']` raw no
longer see content from `meta/rbac.yml`, `meta/server.yml`, etc.
`rbac_group_path` was the last lookup still going through the raw
host_vars read. For joomla (whose `meta/rbac.yml` declares the
`editor` role) the post-10cb02461 host_vars carried no `rbac` block
at all, the lookup walked an empty `rbac.roles` map, and the deploy
crashed with "rbac_group_path: role 'editor' is not declared under
applications[web-app-joomla].rbac.roles. Declared roles:
['administrator']" (job 77797371433 on run 26428080957).
Route `_get_application` through `get_merged_applications` so the
meta-merged view (services + server + rbac + volumes + schema
credentials + variant overrides) is restored. This mirrors the
existing pattern in plugins/lookup/{config,tls,nginx,domain}.py and
keeps the mirror-override intent of 10cb02461 intact: the bake step
still emits only the variant overrides; the merged cache assembles
the full view at lookup time.
Test updates:
- TestRbacGroupPathLookup.setUp patches get_merged_applications to
read from variables['applications'] so the existing 14 tests stay
hermetic without disk-reading roles/*/meta/*.
- New TestRbacGroupPathReadsMergedApplications regression case
exercises the exact 10cb02461 scenario: host_vars only carry a
variants.yml slice, the merged cache supplies the rbac block, the
lookup returns the correct path.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Matrix-deploy variant transitions wipe a consumer's compose stack and
its named DB volume (`compose down -v` in
scripts/container/purge/entity/compose.sh) but `databases.csv` at
$FILE_DATABASE_SECRETS survives the purge. When the next round drops
the consumer from its inventory, the stale baudolo-seed row remains
behind. The subsequent backup run reads that row, attempts
`mariadb-dump -u<entity> -p<old_pw>` against the freshly initialised
RDBMS user that no longer exists, and the systemd unit fails with
"Access denied for user 'matomo'@'127.0.0.1' (using password: YES)"
(job 77797356000 on run 26428080957: matomo dump in
svc-bkp-container-2-local variant 1, where round 1 seeded matomo but
round 2 only deploys postgres + mariadb + svc-bkp itself).
Add `utils.cleanup.databases_csv` mirroring the existing
`utils.cleanup.tokens` pattern: parse the baudolo-seed CSV format
(';'-separated, four columns instance/database/username/password, with
header — verified against baudolo.seed.__main__:check_and_add_entry)
and strip every row whose `database` or `username` column matches an
entity derived from the supplied app ids via
utils.roles.entity_name.get_entity_name. Header and blank lines are
preserved verbatim; the file is rewritten only when at least one row
was removed.
Wire the helper into scripts/container/purge/apps.sh right after the
token-store wipe so every matrix-deploy inter-round purge invokes it
alongside the other cleanups. Operator-triggered purges
(`INFINITO_APPS=<app> bash scripts/container/purge/apps.sh`) also get
the cleanup — that is a behaviour change relative to before, where
the row would survive an operator-triggered entity teardown.
Coverage: 12 unit tests in tests/unit/utils/cleanup/test_databases_csv.py
covering the happy path, the multi-app case, the
`database='*'`/username-match case, the unknown-app no-op, header and
blank-line preservation, the all-rows-removed case, malformed short
rows, and the legacy-header-less file. `make test` green across all 5
targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- web-opt-rdr-www canonical: rdr-www.{DOMAIN_PRIMARY} -> w3redirect.{DOMAIN_PRIMARY}.
- New lookup `www_redirect_domains` (role-scoped): returns full {source, target}
mapping list, resolves DOMAIN_HOMEPAGE target for the role's own canonical via
lookup('domain', 'web-opt-rdr-www'). REDIRECT_WWW_DOMAINS now carries dicts;
tasks/main.yml and _01_cloudflare_edge_redirect.yml updated to consume dicts.
- New lookup `current_play_redirect_domains`: replaces the two inline
set_fact merges in tasks/stages/01_constructor.yml. Merges user-supplied
redirect_domain_mappings, primary-redirect (conditional on web-opt-rdr-domains
in deployed), and per-app domain mappings (inlined from the now-removed
domain_mappings filter).
- domain_redirect_mappings filter removed; web-app-keycloak RedirectUrisTest
pollution of utils.roles.applications.config in sys.modules fixed via
tearDownClass cleanup.
- csp_skip_domains lookup: csp.health.skip opt-out path removed (architecture
no longer needs it).
- Lint test_only_web_app_roles_integrate_dashboard expanded from web-app-* to
web-*; test_role_and_group_vars_used now scans .py files too so vars consumed
only inside plugins (DOMAIN_HOMEPAGE, AUTO_BUILD_ALIASES) are seen.
- New role-scoped variants.yml on web-opt-rdr-www with dashboard on/off.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Since 576fad84f (2026-05-10) moved role images from
`defaults/main.yml` into `meta/services.yml`, matrix-deploy's variant
baker has been writing the FULL effective config (services.yml +
variants.yml deep-merged via `get_variants`) into host_vars through
the `--vars` JSON consumed by `apply_vars_overrides`. That populates
`applications.{role}.services.{svc}.image` (and `.version`, .cpus,
etc.) with the upstream value, which then silently disables
`apply_mirror_overrides` for every role with an explicit image
declaration: the default `if_missing` policy short-circuits when the
field is already non-blank, so the rendered compose / Dockerfile FROM
keeps pulling from the upstream registry rather than the GHCR mirror.
The regression went unnoticed because Docker Hub usually serves the
pull, but it eventually surfaced as a BuildKit 429 on
`docker.io/library/php:8.2-apache` during the suitecrm Dockerfile
build (job 77602110531 on run 26362676803). Nextcloud, moodle,
mariadb consumers, postgres consumers, simpleicons, phpmyadmin,
phpldapadmin, jenkins, baserow, fider, ... all share the same latent
defect — only suitecrm happened to hit the rate limit first.
Fix the root cause instead of papering over per-role with
`mirror_policy: force`: introduce `get_variant_overrides_only` next
to `get_variants` and switch payload.py to it. The new function
returns the raw `meta/variants.yml` entries per role — no merge with
`meta/services.yml`. matrix-deploy now bakes only the variant-
specific overrides into host_vars; the role's services.yml fields
flow through the normal Ansible lookup chain at deploy time, leaving
`image`/`version` blank in host_vars so `apply_mirror_overrides`
populates them from `mirrors.yml` under its default `if_missing`
policy.
Behavior preserved for the other `get_variants` consumers:
* utils/manager/inventory.py:104 (schema-driven credentials) keeps
the full merged variant payload.
* planner.py:58 (variant-count only) is unaffected.
* legacy_resolver.py:42 (CombinedResolver runtime services_overrides)
keeps the full merged view.
Coverage:
* 3 new tests in test_applications.py for the new helper (single
empty variant when no meta/variants.yml, raw override without base
merge, cache invariance).
* test_payload.py + test_builder.py mocks repointed from `get_variants`
to `get_variant_overrides_only`.
* `make test` green across all 5 targets (lint, unit, integration,
external, test-lint) — 188 tests total.
Side-effects assessed:
* Repo-wide grep confirms no template / task accesses
`applications.X.services.Y.image` directly via hostvars — every
consumer goes through `lookup('config', …)` which still deep-merges.
* Roles previously pulling upstream now resolve via the GHCR mirror
whenever the image is already synced (the `images-mirror-missing.yml`
workflow keeps the destination populated).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirror the nextcloud / friendica layout so each kix scenario lives in
its own `test-*.js` and `_shared.js` carries only the logic consumed
by more than one file. `playwright.spec.js` becomes a thin orchestrator
that pulls each test module's `register(shared)` hook.
Layout
playwright.spec.js orchestrator (test.use, beforeEach, requires)
_shared.js cross-cutting: env (appBaseUrl,
canonicalDomain, oidcIssuerUrl, logoutUrl,
oauth2Enabled, ldapEnabled), performKixLogin,
runKixLoginLogoutFlow, beforeEach, persona
+ service-gating re-exports
test-kix-root-tls-hsts.js `kix root emits TLS+HSTS`
test-login-administrator.js admin OAuth2-proxy + Keycloak + KIX-LDAP
flow (ADMIN_USERNAME/PASSWORD local)
test-login-biber.js same flow for biber + the Keycloak admin
REST helper `ensureUserInGroup`
(KEYCLOAK_*, KIX_USER_GROUP_PATH,
BIBER_USERNAME/PASSWORD local)
test-guest-persona.js `guest: public-landing → auth chain`
test-biber-persona.js `biber: app → universal logout`
Per the strict shared-vs-local split, only `appBaseUrl`,
`canonicalDomain`, `oidcIssuerUrl`, `logoutUrl` and the two service
flags stay in `_shared.js`; admin credentials, biber credentials and
the Keycloak admin API env live in their consuming `test-*.js`.
`ensureUserInGroup` moves with its sole caller (biber) and keeps its
`api.dispose()` cleanup.
Behavioral delta: env validation is now per-consumer — TLS / persona /
admin-login tests no longer fail in `beforeEach` for missing BIBER_*
or KEYCLOAK_*; they validate the env they actually touch. Matches the
nextcloud/friendica diagnostic pattern.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Variant 1 of web-app-kix disables every dynamic service flag
(css, dashboard, email, ldap, logout, matomo, oauth2, prometheus —
meta/variants.yml:32-56). With `services.ldap.enabled=false`,
svc-db-openldap is not in the deploy closure, so the top-level
`LDAP.SERVER.DOMAIN` chain in group_vars/all/12_ldap.yml:14-16
resolves via `lookup('config','svc-db-openldap', ...)` and raises
hard the moment any task touches it.
tasks/01_core.yml rendered `zzz_auth_ldap.xml.j2` unconditionally,
the template referenced `KIX_LDAP_HOST` etc., the lazy `LDAP.*`
chain fired, and the deploy aborted before any companion subapp
ever reached Playwright (job 77602110489 on run 26362676803 has
zero variant-1 artefacts across the entire 14-app bundle, while
gitlab's variant-1 produced at least three).
Introduce `KIX_LDAP_ENABLED` in vars/main.yml as the single source
of truth (matches OPENTALK_OIDC_ENABLED / OPENTALK_TURN_ENABLED in
web-app-opentalk), and gate both consumers on it:
* the LDAP-config render task — skipped when LDAP is off
* the compose mount of the LDAP config file — emitted only when
LDAP is on, so the backend container does not try to mount a
missing path
Matches the persisted operator preference: auth services stay
optional — fix code, not variants.
Static check only (per operator request). Confidence ~70% that
this clears the immediate variant-1 failure point; possible
follow-up issues (oauth2-proxy absence, etc.) would only surface
once the kix log reaches further than the previous abort.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The biber sso-login spec timed out at 300s per attempt (688.751s
across 3 retries) on job 77602110523 of run 26362676803. Trace
analysis showed a single action accounted for the entire test
budget: `Press "Enter" on input[name=password]` ran for 312.33s.
The race the existing comment already flagged: biber's first OIDC
hop triggers OpenCloud's lazy user provisioning, Keycloak unmounts
the password form sooner than admin's already-existing path, and
Playwright's `.press("Enter")` actionability retry waits for the now
detached input to come back. Without an action-level timeout the
retry inherits the test-level 300s budget, the `.catch(() => {})`
fires too late, and the subsequent banner assertion crashes with
"Target page, context or browser has been closed".
Bound the press to 5s so the catch can swallow the failure
immediately; the `leftKeycloak` promise (60s) then waits for the
real navigation, which the page-snapshot confirms always reaches
the Files view in time. Admin keeps its 2.96s happy path (5s is an
upper bound, not a baseline delay).
Static check only (per operator request).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The biber-LDAP-login test asserted that biber's URL after logout
matches `/realms/.+/protocol/openid-connect/logout`, but its
variant gating (test.skip when `oidc || \!ldap`) means the test only
runs in nextcloud variant 2 — where meta/variants.yml explicitly
sets `services.logout.enabled=false`. Without web-svc-logout in
scope, no universal-logout JS is injected, the native Nextcloud
logout never hits Keycloak's end-session endpoint, and the assertion
times out 100% of the time (job 77602110528 on run 26362676803).
The combination `oidc=false + logout=true` is semantically
impossible: universal-logout coordinates logout across an OIDC realm
and has nothing to do when no OIDC session exists. The Keycloak
end-session assertion is structurally never reachable from this
test's scope.
Mirror the existing post-logout verification in
test-login-biber-oidc.js (which passed in variant 0): navigate to
/login and assert no authenticated Nextcloud shell is visible. That
check is variant-agnostic and captures the real semantic intent —
"after clicking Log out, biber is no longer signed in" — without
depending on the universal-logout injection chain.
Static check only (per operator request).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`_apply_one_role_special_rules()` in utils/manager/inventory.py used
`bool(resolve_database_service_key(...))` to decide whether to
auto-generate a `database_password` credential for a role. The
resolver raises when multiple RDBMS keys are enabled — which is the
correct contract for compose_volumes / container_depends_on /
plugins.lookup.database (they need an unambiguous single key) but the
wrong contract for inventory's credential-generation pass: variant 1
of svc-bkp-container-2-local intentionally enables both mariadb and
postgres to exercise the engine-side dump path against both engines
at once, and the deploy aborted with `ValueError: multiple direct
database services are enabled: mariadb, postgres` during inventory
generation (job 77602095377 on run 26362676803).
Add a non-raising sibling helper `has_single_database_service()` that
returns True iff exactly one RDBMS service is enabled, and switch
inventory's call site to it. Multi-DB roles now skip the auto-
generated `database_password` (which was a phantom credential for
svc-bkp anyway — no schema.yml, no cross-role lookup; the role's own
`04_seed-database-to-backup.yml` receives `database_password` as a
caller-passed task var, not as a self-credential lookup).
`resolve_database_service_key` keeps its strict raise semantics for
the compose_volumes / container_depends_on / database-lookup callers,
which still need a single unambiguous key.
Add unit coverage for both functions (3 + 4 tests) in
tests/unit/utils/roles/applications/services/test_database.py.
Static check only (per operator request).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The biber+admin side-by-side scenario asserted on `a[href*='/logout']`
to verify admin's session survives biber's logout, but in variant 2
(ldap-only, no oauth2-proxy) Friendica's frio theme renders the logout
link as `href="logout"` (relative, no leading slash). The substring
match `/logout` therefore never hit, the assertion timed out, and the
deploy failed at `Fail if there were Playwright failures` (job
77602110488 web-app-friendica plus 77602110466 web-app-fediwall which
re-runs the same spec via variant-2 bundling on run 26362676803).
The trace.zip of the failing assertion shows the page IS fully
authenticated — banner with the Administrator user-menu button, the
New post link, full sidebar navigation — and `id="topbar-first"
class="topbar" role="menubar"` is present in all four captured HTML
snapshots. Variant 0 (oauth2+ldap+oidc) passed with the old selector
because the oauth2-proxy round-trip emits `/logout` substrings in the
HTML that variant 2's direct ldap flow does not.
Switch the assertion to the same composite selector both Friendica
login helpers (loginViaOauth2ProxyAndFriendica, loginViaFriendicaDirect)
already rely on:
"#topbar-first, #navbar-apps-menu, a[href*='/logout']"
`#topbar-first` matches in both variants (verified in trace.zip),
keeps a[href*='/logout'] as a third OR option for forward
compatibility, and uses toBeVisible to match the login helpers'
`waitFor({ state: 'visible' })` semantics. Session-bleed detection
(the real intent of the test) stays intact: if admin's session were
killed by biber's logout, /network would redirect to /login where
none of the three selectors match.
Static check only (per operator request).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
