Migrate every role's meta/volumes.yml from ad-hoc dict-of-named-volumes
to a canonical dict-of-dicts schema where the YAML key is the semantic
short name and the entry body carries type / name / source / mounts /
nfs / mode / read_only. The optional 'name:' field is the docker volume
name (defaults to the key); the previous 'docker_name:' field is gone.
What's in this commit (210 paths):
Infrastructure (23)
- New filter plugins/filter/{compose,container}_volumes.py — render the
top-level volumes:/configs:/secrets: section and per-service mount
blocks from the canonical meta. container_volumes uses the SEMANTIC
name as per-service mount source so it resolves against compose_volumes'
top-level dict key (docker_name as source produced 'undefined volume'
on swarm).
- New lookup plugins/lookup/{compose,container,volume}_volumes.py.
volume(application_id, name) returns a normalised single entry;
volume(application_id) returns the whole canonical dict.
- utils/cache/applications.py keeps a sibling registry
_CANONICAL_VOLUMES_BY_ROLE outside the merged applications payload so
raw Jinja source: strings never enter the templar walk (root-cause of
the 200x cache-miss + recursive get_merged_applications storm).
- utils/roles/applications/mounts.py — schema validator + normaliser;
list-shape is rejected.
- plugins/lookup/nginx.py: _volume_path_by_name reads canonical directly.
- scripts/tests/deploy/swarm/_context.sh: NFS_VOLUMES extraction handles
both legacy dict and canonical dict shapes.
- roles/svc-prx-openresty/handlers/restart_and_validate.yml: gate
compose-only validate+reload behind a single when-block; swarm uses
service-update instead (bind-mounts of a FILE pin source inode at
task-create time, so atomic write-tmp+rename leaves the running task
on a deleted inode).
- roles/sys-svc-compose/tasks/utils/nfs_prep.yml: switch to
lookup('volume', application_id) | dict2items so NFS subdir
pre-creation works under the new cache split.
Tests (12)
- New strict lints flagging inline mount sections in compose templates,
single-file bind mounts, and legacy lookup('config', ..., 'volumes.X.Y')
call sites — all volume-mount surfaces MUST flow through container_volumes
/ compose_volumes / volume lookups.
- Integration: test_volumes_compose_chain + test_volume_constants_are_lookups
accept both the legacy and the new lookup('volume', app, 'X').name forms
during the migration window.
- Unit tests for the new lookup, filter, mounts helper, plus updated
nginx lookup test.
- Prometheus role-wiring test reads canonical meta for the
alert_rules.yml bind mount.
Role migrations (175)
- 62 meta/volumes.yml converted to dict-of-dicts canonical shape
- 63 templates/compose*.yml.j2 replace inline volumes:/configs:/secrets:
with lookup('container_volumes', application_id, '<service>') |
indent(N) per service + a single trailing lookup('compose_volumes',
application_id) for the top-level block
- 50 vars/main.yml drop dead volume constants and refactor 23 remaining
legacy lookup('config', ..., 'volumes.X.Y') to lookup('volume',
application_id, 'X').Y across 9 roles
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The compose/swarm split on `depends_on:` shape (map form with
`condition:` vs. list form, per `docker stack deploy` constraint)
was being hand-coded as a `{% if DEPLOYMENT_MODE == 'swarm' %}` gate
around every block. Repetitive, easy to forget, and templates that
did remember it had to repeat the same six-line pattern per service.
The new `depends_on` lookup encapsulates the shape switch:
{{ lookup('depends_on', {SVC: 'service_healthy', …}) }} # mapping
{{ lookup('depends_on', [SVC_A, SVC_B]) }} # all default
- Compose -> map form with explicit `condition:` per entry.
- Swarm -> list form (just names; swarm ignores depends_on
ordering semantically but accepts the syntax).
- Mapping `None` value uses the default condition (`service_started`,
matching Docker Compose's behaviour when no condition is given).
- Plain list of names uses the default for every entry.
The output is shaped so the call site reads naturally at the
service-member column:
some_service:
image: …
{{ lookup('depends_on', {DB: 'service_healthy'}) }} # at col 4
volumes: …
Concretely the lookup emits line 1 WITHOUT leading whitespace and
lines 2+ with the `indent` kwarg (default 4) plus the YAML-relative
nested indent. Jinja substitution applies the template's leading
spaces to line 1 only, so the column-4 call placement makes
`depends_on:` land at column 4 and the children at 6 / 8. Lines 2+
carry their own indent because Jinja does not propagate template-side
leading whitespace to continuation lines.
Plugin + tests:
- plugins/lookup/depends_on.py: ~190 lines. Mapping / list / empty
input handling, default_condition kwarg, condition validation
against the closed set `{service_started, service_healthy,
service_completed_successfully}`, DEPLOYMENT_MODE resolution via
templar with `mode=` test-only override, indent kwarg semantics
documented in the module docstring.
- tests/unit/plugins/lookup/test_depends_on.py: 27 cases covering
TestInputNormalisation (mapping / list / empty / typo / single
string rejection / non-string sequence rejection / unsupported
type rejection / arity errors), TestModeBehaviour (compose / swarm
/ unknown-falls-to-compose / missing-DEPLOYMENT_MODE-falls-to-compose
/ mode-kwarg override), TestOutputFormatting (line-1 unindented
exact-string assertion, indent=0 edge case, indent=6 edge case,
insertion-order stability for diff-stable output).
Lint:
- tests/lint/ansible/jinja/test_compose_template_uses_depends_on_lookup.py:
rule key `compose-depends-on-must-use-lookup`. Flags three patterns
in any `*compose*.yml.j2` under `roles/`:
1. literal `depends_on:` keyword line
2. flow-form `depends_on: [a, b]`
3. literal `condition: service_*` line
The pre-existing `lookup('container_depends_on', application_id)`
is intentionally not flagged: its call form contains no literal
`depends_on:` token in source. Scan target broadened to
`name.endswith('.yml.j2') and 'compose' in name` so
`compose.override.yml.j2` and `compose-inits.yml.j2` are covered
alongside `compose.yml.j2` and `*.compose.yml.j2`.
Migration (43 depends_on blocks across 17 templates):
- erpnext (6): configurator + backend + websocket + scheduler +
queue-short + queue-long + frontend chains.
- mailu (8): imap + smtp + oletools + antispam + antivirus + webdav +
fetchmail + webmail.
- zammad (4): rails + 3 worker shapes.
- shopware (4): web + worker + scheduler + opensearch, init-gated.
- jitsi (3): three single-key dependencies on JITSI_PROSODY_CONTAINER.
- kix (3): backend + frontend + proxy.
- bigbluebutton/compose.override.yml.j2 (2): bbb-web + etherpad,
inside the existing BBB_COLLABORA_ENABLED gate.
- moodle (2): nginx + cron, both depending on moodle.
- opentalk (2): application + recorder.
- taiga (2): events + gateway.
- matrix (1): loop-driven via `{% set deps = {} %} +
__setitem__` escape hatch over MATRIX_BRIDGES.
- matomo (1): bootstrap service.
- nextcloud (1): whiteboard sidecar.
- openproject (1): proxy.
- funkwhale (1): front.
- magento (1): nginx.
- flowise (1): postgres + redis.
Pre-existing `lookup('container_depends_on', application_id)` calls
(erpnext, nextcloud, moodle, zammad, taiga, matrix, etc.) are left
untouched -- that lookup auto-detects DB / redis from the application
config and emits its own block; it is the app-aware counterpart to
this structural lookup.
Every call placement repositioned to column 4 (standard
compose-service-member indent) so the new line-1-unindented semantic
renders correctly. Eight templates needed the column-4 reposition
(via a mechanical one-shot script that was deleted after).
`make test` green; new lint + 27 unit tests green; independent
re-grep across all `*compose*.yml.j2` finds zero residual raw
`depends_on:` / `condition:` lines.
Across 23 web-app env.j2 templates, drop the surrounding "..." from
plain KEY="value" entries. compose silently strips them at env_file
parse time, but docker stack deploy preserves them literally, so the
container then sees KEY="value" with the quotes embedded and parsers
that call int(os.environ['KEY']) or split URLs/DSNs break under swarm.
The change is mechanical: every removed line was `KEY="value"` and
every added line is the same `KEY=value` without surrounding quotes.
Lines that use `| dotenv_quote` are untouched because that filter
deliberately produces escaped values for env_file consumers; the
existing password-quote integration test continues to require it for
sensitive variables.
Affected: bigbluebutton, bluesky, bookwyrm, bridgy-fed, chess,
confluence, decidim, espocrm, friendica, funkwhale, gitea, jira,
magento, mailu, mobilizon, openproject, phpmyadmin, pixelfed, pretix,
shopware, snipe-it, suitecrm, xwiki.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add a meta lint asserting every web-app role with postgres or mariadb declares a seaweedfs object-store service, exemptable via the seaweedfs-required nocheck marker. Mark the 33 database-backed roles that do not use object storage, and register the rule in the suppression catalog.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a mode-aware `container_address` lookup plugin and a
`resolve-container-id` helper script that resolve swarm task IDs at
exec time (compose mode returns bare names unchanged). Migrates ~25
roles + ~28 task files from `command:` to `shell:` so the lookup's
`$(...)` subshell evaluates correctly, and extends `database` lookup
with an `address` key for cross-role DB exec.
Three new lint tests enforce the contract: every container exec
target goes through the resolver, lives behind an IS_STACK_HOST gate
(walks the include graph one hop deep), and uses `shell:` rather than
`command:` when the target is a Jinja reference. Existing nocheck
suppressions on bare-name DB exec sites (nextcloud redis, listmonk,
moodle OIDC, mailu) are replaced with the lookup-based form.
Fixes the systemd Environment= injection in sys-ctl-cln-disc-space
(it never expanded $(...)) by passing STACK / SERVICE_KEY parts and
resolving inside the script. Extracts shopware admin and mediawiki
extension install bodies into dedicated .sh files invoked via env.
Rewrites password-bearing shell tasks to either `| quote` directly or
the `-e VAR=... | quote` + `sh -c '... ${VAR} ...'` env-bridge pattern.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Resolves Cluster G.2 (2 compose jobs from CI run 27063963773): mailu
user-mgmt commands (`flask mailu user`, `flask mailu password`)
were SIGKILL'd with rc=137 after 4-6s. Flask CLI inside the admin
container hit the 1g cap while executing user-creation / password
ops.
Raising to 2g matches the headroom Mailu's upstream docker-compose
recommends for non-trivial user counts.
Locally validated: mailu variant=0 reinstall completes (23m, no rc=137).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The MODE_DEBUG-gated DinD DNS handler in sys-svc-container spawned a
busybox container with `--dns 1.1.1.1` and expected the mocked
infinito.example to resolve to 127.0.0.1, which Cloudflare obviously
cannot do. Run an in-cluster dnsmasq inside mgr-01 instead of patching
the symptom out:
- Stop and mask systemd-resolved so port 53 is free.
- Bootstrap /etc/resolv.conf with 1.1.1.1 so apt-get can fetch dnsmasq.
- Install dnsmasq with bind-dynamic, no-resolv, address=/${INFINITO_DOMAIN}/127.0.0.1
and 1.1.1.1/8.8.8.8 as upstream forwarders; bind-dynamic catches docker0
when the Ansible-installed dockerd creates it later.
- Switch /etc/resolv.conf to 127.0.0.1 and verify the chain via
`getent ahosts ${INFINITO_DOMAIN}` so the script fails fast on misconfig.
- Pin mgr-01's docker0 to 172.17.0.1/16 via daemon.json `bip`, then point
the swarm.yml networks.internet.dns at that gateway so DinD-spawned
containers reach dnsmasq via their default route.
Also re-stage three pure-formatting tightenings ansible-lint made to
existing `template: # nocheck` lines (double space -> single space).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds plugins/action/conf_template.py, a drop-in `template:` replacement
that returns `skipped: true` whenever IS_STACK_HOST evaluates to false.
Routes the rendered Jinja through ansible.module_utils.parsing.convert_bool
so both Python bool and string "True"/"False" templar outputs resolve
correctly. Inherits every standard template module argument.
Migrates seven nginx-tree consumers that previously aborted on workers
with "Destination directory does not exist": sys-front-tls,
svc-db-openldap, sys-svc-webserver-core (nginx.conf + default_server.conf),
web-app-matrix (synapse vhost), web-app-peertube (per-domain vhost),
web-opt-rdr-domains (redirect vhost), and web-app-prometheus.
Tags twelve legitimate bare `template:` callers with
`# nocheck: conf-template`: per-host system configs (locale, nix, systemd
manager defaults, journald, msmtprc) and per-app compose-dir configs
(magento x2, mailu x2, moodle, odoo, taiga) that intentionally bypass the
project's nginx tree.
Adds tests/lint/ansible/modules/test_conf_template_migration.py as the
drift guard: any future `template:` task whose `src:` ends in `.conf.j2`
must adopt conf_template or carry the nocheck marker. Adds seven unit
tests for the plugin covering skip semantics, dispatch, mode defaults,
mode overrides, full-arg passthrough, and the string-bool templar path.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Across roles/web-app-*, the typical body shape is:
- include_role: sys-stk-full (or sys-svc-compose) cluster prereqs
- <host-fs writes, container exec, compose exec> stack-host only
Under swarm-mode delegation the second group only makes sense on the
host that holds the compose stack (manager), but the role plays on
every host in groups[<role>] so workers abort on missing paths.
Two workflow sweeps (gate-manager-ops + gate-manager-ops-01core, 58
agents total, discover/plan/adversarial-verify per role) plus three
manual fixes surfaced 27 roles to refactor. Each role keeps
sys-stk-full ungated so worker-side prereqs (sys-svc-container, NFS
client) still bootstrap; the trailing manager-only block either moves
into a new tasks/02_manager_ops.yml included with
when: IS_STACK_HOST | bool (SUB_FILE pattern) or gets the gate
inlined on the one or two trailing tasks (INLINE pattern). In
compose mode IS_STACK_HOST stays true so single-host deploys are
unchanged.
Manual catches the workflow agents missed:
- web-app-fediwall: utils/once/flag.yml lives AT THE END of 01_core,
not at the start; preserved verbatim outside the gate so workers
still flip the run_once_<role> fact.
- web-app-xwiki: a pre-existing block(...) wraps sys-stk-full plus
_flush_config.yml; gated only the _flush_config.yml include so
sys-stk-full keeps running on workers.
- web-app-dashboard: 02_javascript.yml (creates /opt/compose/dashboard/
JS dirs) ran BEFORE sys-stk-full and was ungated; added the gate.
Roles touched (INLINE): akaunting, jenkins, magento, mobilizon
Roles touched (SUB_FILE): baserow, dashboard, decidim, erpnext,
fediwall, friendica, gitea, jitsi, keycloak, listmonk, littlejs,
mailu, matomo, moodle, odoo, openproject, opentalk, pixelfed,
shopware, suitecrm, taiga, xwiki, zammad
Single SPOT lookup `plugins/lookup/compose_volumes.py` absorbs the
applications-lookup pipe step and auto-wires `DEPLOYMENT_MODE` plus
the `storage` group var from the templating context. Callers only
pass what genuinely varies per service (`application_id` and an
optional `extra_volumes` dict).
Behavior change (bug fix): the filter was always invoked with the
default `deployment_mode='compose'` because no caller passed it
explicitly, which silently skipped the swarm-NFS driver-opts
injection path. The lookup reads `DEPLOYMENT_MODE` from globals,
so swarm-NFS volumes now render the `driver: local` / `driver_opts:
type: nfs` block correctly. Callers can still override via
`deployment_mode=` / `storage=` kwargs when a template needs
non-default rendering.
Other coordinated changes:
- 81 template call sites migrated from pipe form to lookup form
(single-line and multi-line shapes).
- `FilterModule` registration in plugins/filter/compose_volumes.py
removed so the deprecated pipe form fails fast at render time.
The underlying `compose_volumes()` function stays exported and is
imported by the new lookup plus the existing 21-test filter unit
test module.
- 10 new lookup unit tests pin the contract (term shape, kwargs
pass-through, auto-wiring, override precedence, error cases).
- Integration test `test_compose_volumes_call_required.py` updated:
asserts the new lookup form is present in every `compose.yml.j2`
and adds a second test forbidding the deprecated pipe form
anywhere under `roles/**/templates/`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace per-key `compose_container_name` lookup with a generic
`compose_only(key, value)` SPOT that emits `key: "<value>"` only in
compose mode and nothing in swarm. Same gating logic now serves
container_name (157 call sites) and pull_policy (17 call sites)
without duplicate plugin code.
Why this consolidation:
- The earlier `build-key` lint rule was misdirected. Swarm tolerates
`build:` with a harmless warning when `image:` is pre-pushed (via
the existing handlers/swarm_registry_push.py flow in
roles/sys-svc-compose). The actual swarm-rejected key is
`pull_policy:` ("Additional property pull_policy is not allowed"),
and 10 templates were silently broken because they declared inline
`pull_policy: never` outside the existing `{% if DEPLOYMENT_MODE
\!= 'swarm' %}` gate in roles/sys-svc-container/templates/build.yml.j2.
- A single generic lookup is cleaner than N per-key wrappers when
the gating logic is identical (emit-key-or-nothing).
Lint changes in tests/lint/ansible/jinja/test_mode_incompatible_keys.py:
- New `pull-policy-key` rule, regex `^\s*pull_policy\s*:`, remediation
points at compose_only.
- Existing `container-name` rule keeps its raw-key regex; remediation
now also points at compose_only.
- Broken-split / broken-embed regexes rekeyed on the `compose_only`
lookup name (was `compose_container_name`).
- `build-key` rule and its 12 nocheck annotations across templates
removed.
Coordinated changes:
- 86 templates rewritten (157 container_name calls migrated;
17 inline pull_policy lines replaced).
- 1 redundant `{% if DEPLOYMENT_MODE \!= 'swarm' %}` guard dropped in
build.yml.j2 (the lookup self-gates).
- 12 unit tests pin the compose_only contract.
- Catalog (suppression.md) updated: `pull-policy-key` row added,
`container-name` row remediation updated.
- Old `plugins/lookup/compose_container_name.py` and its test module
deleted.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
container_name is illegal in swarm mode when replicas > 1 (docker rejects
with "services.deploy.replicas: can't set container_name and X as container
name must be unique"). All compose.yml.j2 templates now route through a
single lookup that suppresses the key in swarm mode and emits it in compose
mode.
SPOT: plugins/lookup/compose_container_name.py
Reads DEPLOYMENT_MODE from the templating context, returns
'container_name: "<name>"' under compose and an empty string under swarm.
Call sites read:
{{ lookup('compose_container_name', MY_CONTAINER) }}
No explicit DEPLOYMENT_MODE pass-through needed.
Sweep: 82 templates rewritten across roles/web-app-*, roles/web-svc-*,
roles/svc-*, plus 4 partial includes (sso_proxy, redis, mariadb, postgres).
For container names assembled from multiple Jinja fragments the args use
tilde concat (X ~ '-' ~ Y) so the lookup receives a single resolved string
and the call stays a single well-formed Jinja expression.
Lint: tests/lint/ansible/jinja/test_no_raw_container_name.py blocks
(a) raw 'container_name:' YAML keys in any roles/*/templates/**/*.j2,
(b) split-Jinja regression where the outer braces close mid-call:
{{ lookup('compose_container_name', A }}-{{ B) }},
(c) embedded-Jinja regression inside a string argument:
lookup(..., "{{ X }}_suffix") (removed in ansible-core 2.23).
The (b) and (c) regexes accept either single- or double-quoted lookup
name and scan the full file so multi-line shapes also surface. Suppress
per-site with '# nocheck: container-name' on the same or preceding line.
Tests: tests/unit/plugins/lookup/test_compose_container_name.py pins the
lookup contract: compose vs swarm output, whitespace tolerance, default-to
compose when DEPLOYMENT_MODE is absent, AnsibleError on missing or excess
terms, non-string name coercion.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`web-app-mailu/tasks/03_manage_user.yml` and `03b_create_user_token.yml`
run `compose exec admin flask mailu user <...>` and `compose exec admin
curl http://localhost:8080/api/v1/token` immediately after the existing
wait blocks in `02_wait_for_database.yml` (mariadb domain table + admin
flask CLI ready). Under CI load the Mailu admin container is reachable
on the compose network, but two further runtime preconditions are not
guaranteed when those tasks fire:
1. The flask CLI imports succeed (`flask routes` returns 0) before
Mailu's admin code has finished establishing the lazy Redis
connection that `flask mailu user` needs to commit the user.
Symptom (run 26480944120 job 77980186049, infinito-nexus/core
run 26480946718 job 77980074367, web-app-suitecrm):
redis/connection.py:639 ConnectionRefusedError: [Errno 111]
during `flask mailu user no-reply infinito.example -- <pw>`.
2. The admin gunicorn HTTP server on :8080 is not yet listening when
`03b_create_user_token.yml` curls `http://localhost:8080/api/v1/
token`. Symptom (run 26480946718 job 77980074254, web-app-fediwall):
curl: (7) Failed to connect to localhost port 8080 after 0 ms.
Add two further wait blocks after the existing flask-routes wait, both
following the same probe/rescue pattern (300-line container log dump
on timeout) the file already establishes:
* Redis: `compose exec -T redis redis-cli ping` until stdout
contains "PONG". Matches Mailu's own healthcheck command verbatim
(svc-db-redis/templates/service.yml.j2 line 24) and depends on no
extra auth -- Mailu's env.j2 declares no REDIS_PASSWORD and the
injected svc-db-redis service runs with no `--requirepass`.
* Admin HTTP: `compose exec -T admin curl -sS -o /dev/null -w
'%{http_code}' http://localhost:8080/sso/login` until rc=0 and the
captured status code matches `^[1-5][0-9][0-9]$`. Any HTTP response
(200 / 302 / 404 / 5xx) proves gunicorn is bound; the regex form
makes the check robust against an upstream rename of the SSO
endpoint.
Restructure for clarity:
* Split the original `02_wait_for_database.yml` into
`02_wait_for_mariadb.yml` (mariadb domain table + flask CLI ready;
flask init touches the DB) and `03_wait_for_redis.yml` (Redis PING
+ admin HTTP :8080; the HTTP API requires Redis-backed sessions).
* Bump every subsequent task file by one step
(03_manage_user -> 04_manage_user, 04_generate-... -> 05_..., etc.)
so the directory order still encodes execution order at a glance.
* Update the cross-`include_tasks` references in `01_core.yml`,
`04_manage_user.yml` and `04a_manage_user_token.yml`. Also bump
the trailing comment in `01_core.yml` that pointed at
`03_manage_user.yml` to the new `04_manage_user.yml`.
* Hoist the recurring `lookup('container', application_id,
'directories.instance')` (12 occurrences plus a 6x indirection via
a per-include `MAILU_DOCKER_DIR` alias previously defined in
`01_core.yml`) into a single `MAILU_INSTANCE_DIR` constant in
`vars/main.yml`, matching the existing `MAILU_VOLUMES_DIR` shape.
All `chdir:` values now resolve to the same string with no extra
indirection.
`make autoformat` and `make test` (188 tests, 158s) pass on the
restructured tree.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The compose templates appended additional depends_on entries (e.g. taiga
rabbitmq services, openproject cache/seeder, mailu resolver/front) on
column 6 directly after the lookup call. When the lookup returned ""
(no DB and no redis), those entries dangled without a depends_on header
and broke YAML parsing.
Move every such trailing block into the lookup's `extra` kwarg so the
header is always emitted as soon as any entry exists. Adds a regression
test for the extras-only case.
Affected: taiga, mailu, magento, pixelfed, openproject, espocrm.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Eliminate include-style depends_on assembly. The lookup resolves DB/redis
inductively from applications config and returns indented YAML (or "" when
no entries), removing the empty `depends_on:` block bug in libretranslate.
Migrates 43 compose templates from `{% include … dmbs_*.yml.j2 %}` to
`{{ lookup('container_depends_on', application_id) }}`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements docs/requirements/021-sso-flavor-migration.md atomically:
* Schema: every role's meta/services.yml replaces the parallel
`oauth2:` / `oidc:` top-level blocks with a single `sso:` block
carrying `enabled`, `shared`, and a `flavor` discriminator
(`oidc` | `oauth2` | `saml`; default `oidc`). Flavor-specific
sub-keys live under `sso.<flavor>.*` (`sso.oauth2.{origin,acl,
allowed_groups}`, `sso.oidc.plugin`).
* Provider role: roles/web-app-oauth2-proxy/ is deleted in its
entirety; its 5 sidecar templates move under
roles/web-app-keycloak/templates/sso_proxy/, and the per-consumer
rendering task lives at roles/web-app-keycloak/tasks/sso_proxy.yml.
web-app-keycloak now declares `provides: sso`.
* Runtime: new resolver utils/roles/applications/services/sso.py
centralises the SSO predicates (`is_enabled`, `is_proxy_gated`,
`is_oidc_native`) and the oauth2 sub-values
(`oauth2_origin_{host,port}`, `oauth2_acl`, `oauth2_allowed_groups`).
Three lookup plugins consume the resolver: `sso` (single-app
predicate access), `sso_proxy_consumers` (enumerates oauth2-flavored
consumers), `sso_oidc_plugin` (renamed from `oidc_flavor`, Nextcloud
plugin selector). sys-stk-backend, sys-svc-compose, sys-svc-proxy
and plugins/filter/compose_volumes.py all gate on
`lookup('sso', ..., 'is_proxy_gated')` instead of manually combining
`enabled` and `flavor`.
* Identifier renames repo-wide: `services.<entity>.ports.local.oauth2`
→ `ports.local.sso`; `PORT_BANDS.local.oauth2` → `PORT_BANDS.local.sso`;
`OAUTH2_PROXY_*` env/Jinja vars → `SSO_PROXY_*`;
`oauth2_proxy_cookie_secret` → `sso_proxy_cookie_secret`;
`OAUTH2_SERVICE_ENABLED` + `OIDC_SERVICE_ENABLED` collapse into
`SSO_SERVICE_ENABLED`; Playwright gating helpers
(isServiceEnabled/skipUnlessServiceEnabled/requireService/safeIsEnabled/
safeSkipUnlessEnabled) move from `oidc`/`oauth2` to `sso`.
* Dual-block resolution: per req's Background-Table, web-app-bookwyrm
is `flavor: oauth2`; web-app-gitea and web-app-friendica keep
`flavor: oauth2` with their existing `oauth2.acl.blacklist`
(/user/login) and `oauth2.acl.whitelist` (federation + discovery
endpoints) — preserving the operative oauth2-proxy gating.
* Tests: the two mutual-exclusion guards
(tests/integration/iam/oauth2_oidc/test_mutual_exclusive.py and
test_acl_mutual_exclusive.py) are deleted per Decision #10;
test_oauth2_contract.py is renamed and rewritten to
test_sso_contract.py, asserting that `flavor: oauth2` roles carry
`oauth2.origin.{host,port}` plus `ports.local.sso`; test_sso.py
(web-app integration) now requires the unified `sso:` block with an
explicit flavor or a `# nocheck: sso` opt-out; new
tests/lint/repository/no_legacy_sso_paths/ guards the source tree
against re-introducing the legacy strings.
* Docs: req 21 itself has every AC checked off, and reqs 006, 008,
009, 013, 015, 016, 019, 020 plus 10+ role READMEs and
docs/contributing/actions/testing/suppression.md are updated to the
unified SSO naming.
Empirically validated end-to-end against the validation app set
(web-app-keycloak, web-app-nextcloud, web-app-bookwyrm, web-app-gitea,
web-app-friendica, web-app-dashboard) in matrix variants 0 and 1; v2
also passes the 4 target roles' Playwright after a documented
pre-existing KC_HOSTNAME-drift workaround (separate concern, not
caused by this refactor).
BREAKING CHANGE: every consumer of `services.oauth2.*`,
`services.oidc.*`, `OAUTH2_PROXY_*`, `OAUTH2_SERVICE_ENABLED`,
`OIDC_SERVICE_ENABLED`, `oauth2_proxy_cookie_secret`,
`oauth2_proxy_application_id`, `ports.local.oauth2`, the
`web-app-oauth2-proxy` role, or the legacy gating helper service names
(`oidc`/`oauth2`) must migrate to the unified `sso` shape documented
in docs/requirements/021-sso-flavor-migration.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
scripts/install/venv.sh
-----------------------
INFINITO_VENV_BASE is a static-default convention; the active VENV
may end up outside that base (e.g. \$HOME/.venvs/<name> when
/opt/venvs is not writable, or a wholly custom override). Probing
writability on the legacy base triggered a spurious sudo branch
whenever VENV lived elsewhere -- fatal in no_new_privs sandboxes
where sudo cannot escalate. Switch the writability + mkdir + chown
chain to \$(dirname "\${VENV}") so the script prepares the directory
it is actually about to create the venv in.
tests/lint/ansible/roles/web-app/playwright/test_app_base_url_uses_role.py
-------------------------------------------------------------------------
New lint: every role's templates/playwright.env.j2 must declare
APP_BASE_URL as a reference to the role's own surface. APP_BASE_URL
feeds the shared guest-persona helper and Playwright's global
baseURL; a role that hardcodes a sibling role name (most commonly
'web-app-dashboard') sends the guest test to the wrong site and
breaks any matrix variant where the sibling is not deployed
(ERR_SSL_UNRECOGNIZED_NAME_ALERT).
The 16 web-app role templates that violated the contract switch
from sibling lookups (`lookup('tls', 'web-app-dashboard', ...)`)
to the canonical `lookup('tls', application_id, ...)` form.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rename env/static.env → env/default.env across the entire repo
(string label, doc links, Python file-path constants) — the
`parse_static_env*` function names stay as a stable API surface.
Sort all `*.env` / `*.env.j2` entries alphabetically (52 in
default.env, 4 in ci.env, 46 of 49 j2 templates; 3 templates
with Jinja2 `{% %}` control flow are skipped because conditional
blocks would tear apart under a naive sort). Sort all 83
Makefile target blocks alphabetically; .PHONY group headers
stay in place.
Collapse the two remaining multi-line Makefile recipe comments
(dotenv-force, test-signed) to single-line WHY comments.
Add four lint tests under tests/lint/filesystem/ to keep the
invariants enforced going forward:
- env/test_alphabetical_sort.py
- env/test_single_line_comments.py (# nocheck: markers exempt)
- makefile/test_alphabetical_targets.py
- makefile/test_single_line_comments.py (# nocheck: markers exempt)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Commit 1f6b5f894 fixed the mailu V1 guest 60-s timeout by adding inline
auto-detection skips to the shared persona runners (`\!oidcEnabled`,
`dashboardEnabled === "false"`), but those guards are exactly what the
persona strict-mode lint
(tests/lint/ansible/roles/web-app/playwright/persona/test_strict_mode.py,
Rule 11) forbids: auto-detection skips can silently disable tests
when an env flag is mistyped, so only the allowlisted opt-out tokens
(`PERSONA_*_BLOCKED`, `safeSkipUnlessEnabled` / `skipUnlessServiceEnabled`,
the auth-less collapse case) are permitted.
Re-fix in the lint-compliant shape:
- Drop the four inline `test.skip(...)` blocks from
personas/{admin,biber,guest}.js plus the `safeIsEnabled("dashboard")`
parity marker added alongside them.
- Add the missing `PERSONA_GUEST_BLOCKED` opt-out check to
personas/guest.js so the existing biber/administrator block pattern
also covers guest.
- Render `PERSONA_GUEST_BLOCKED` dynamically in
roles/web-app-mailu/templates/playwright.env.j2 — true exactly when
`services.dashboard.enabled` is false. Drops the now-unreferenced
`DASHBOARD_SERVICE_ENABLED=` line.
Behaviour for mailu V1 stays identical (guest persona skipped, just
via PERSONA_GUEST_BLOCKED rather than via persona-internal hostname
sniffing). Verified against the live stack with rerun-spec.sh across
all 13 currently staged Playwright roles — 46 pass / 44 skip / 0 fail,
0 deltas vs. the prior fresh-purged V2 deploy outcomes.
If another role later hits the same dashboard-frame routing failure
in a fresh-purge sweep, the fix is one analogous
PERSONA_GUEST_BLOCKED={{ ... }} line in that role's
templates/playwright.env.j2.
CI run 26089179898 job 76714878217 (Deploy web-app-bluesky, variant 1)
failed at the mailu Playwright `guest: public-landing → auth chain →
never authenticated` test with 60s timeout × 3 retries. Root cause is
two layered defects:
1. Between Round 1 and Round 2 of the matrix deploy the fresh-purge
sweep removes the web-app-dashboard vhost and container (mandate-
fresh-purged on first contact with a new VARIANT, see commit
3f12fbe83). Round 2 in variant 1 deploys only matomo/mailu/keycloak/
bluesky; the dashboard.infinito.example vhost never comes back.
2. mailu's templates/playwright.env.j2 pins APP_BASE_URL to
`lookup('tls', 'web-app-dashboard', 'url.base')` unconditionally —
it has to, because the shared persona helpers route through the
dashboard main-frame Account menu. In variant 1 (dashboard disabled
for mailu) that URL is dangling: nginx's default_server falls onto
the bluesky-pds backend, which serves a text/plain ASCII banner.
readCspString's `<meta>` fallback in personas/utils/csp.js has no
explicit getAttribute timeout and waits on the test-level 60s.
The shared persona itself has no way to learn that APP_BASE_URL routes
through a dashboard frame whose service is disabled — biber + admin
sidestepped this only because mailu already blocks them via
PERSONA_BIBER_BLOCKED / PERSONA_ADMINISTRATOR_BLOCKED for an unrelated
silent-SSO escape. guest had no equivalent opt-out and timed out.
Decided by service flag, not by sniffing the response, so the skip
fires before any network I/O:
- roles/test-e2e-playwright/files/personas/guest.js: skip when
APP_BASE_URL's host starts with "dashboard." AND
DASHBOARD_SERVICE_ENABLED=false.
- roles/test-e2e-playwright/files/personas/biber.js +
personas/admin.js: same dashboard-frame skip + a new OIDC-required
skip (these personas drive the Keycloak round-trip; without OIDC
they cannot reach an authenticated surface). Skips placed
immediately before the dependent operation, not at the top, so the
reason matches the operation that would actually fail. parity-marker
safeIsEnabled("dashboard") added to the existing oauth2/logout/
matomo block so the env-gate parity guard recognises the new
consumption.
- roles/web-app-mailu/templates/playwright.env.j2: emit
DASHBOARD_SERVICE_ENABLED in the existing OIDC/LOGOUT/MATOMO_SERVICE
_ENABLED block. The other 14 dashboard-frame roles will pick up
this declaration when they hit the same variant-disabled failure.
biber + admin remain unconditional-blocked for mailu via the existing
PERSONA_*_BLOCKED flags; the new skips inside them are forward-looking
for the day the silent-SSO escape lifts and those personas are
un-blocked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pure 1:1 rename (69 insertions / 69 deletions across 66 files) to
finish the ``docker_*`` → ``compose_*`` alignment started in
8373468e9. The flag configures whether the compose include flushes
handlers right after run; the ``docker_`` prefix was misleading
(handlers are Ansible-side, not docker-side) and the variable is part
of the same ``sys-svc-compose`` SPOT cluster as ``compose_repository_*``.
Touches the role default, the implementation
(``sys-svc-compose/tasks/utils/up.yml``), all 60+ consumer roles,
the network-utility wrapper + its README, and the
``test_network_create_via_util.py`` lint (both the docstring guidance
and the asserted diff).
Breaking change for any external override that still sets the old
name — no alias is provided.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the web-svc-css provider Playwright spec that mirrors the
dashboard/prometheus SPOT-provider pattern: emits CSS_TARGET_ROLES_JSON
via roles_with_service('css') and asserts every consumer's canonical
surface indexes a <link rel="stylesheet"> pointing at the shared CSS
asset host (CSS_BASE_URL / CDN_BASE_URL). The baseline test exercises
CDN reachability first so per-consumer failures unambiguously point
at the consumer's sys-front-inj-css wiring, not at the provider.
Suppresses the per-role playwright env-services-match guard above each
consumer's `css:` key with the canonical `# nocheck:
playwright-service-flag — verified by web-svc-css provider spec`
marker. web-svc-css/meta/services.yml gets the self-provider nocheck
on `css:` and `provider role; no consumer surface to gate` nochecks on
`cdn:` / `matomo:`.
Adds css.{enabled,shared} true/false pinning to every variants.yml
that declared the dynamic flag without coverage so
test_every_dynamic_flag_has_true_and_false_variant passes. Unifies
the css service declaration across espocrm / mediawiki / openproject
(was static enabled: false, shared: true) so the matrix matches the
rest of the consumer fleet.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Decouples the CSS service concept from its underlying transport
(currently `web-svc-cdn`) so the flag becomes opt-in via inventory
group membership instead of staying literal-`true` everywhere.
- `roles/web-svc-css/`: new role. Marker / wrapper role that owns
the `css` entry in the service registry, declares `web-svc-cdn` as
the upstream that serves the actual bytes, and 301-redirects its
own canonical domain to the cdn domain so health probes return a
valid response.
- `roles/web-svc-cdn/meta/services.yml`: drop the `css` block — the
canonical owner is now web-svc-css; cdn keeps `javascript`.
- 51 consumer roles (web-app-* and web-svc-{coturn,file,html,
libretranslate,logout,onlyoffice}) migrated their `css:` block from
literal `{ enabled: true, shared: true }` to dynamic
`{ enabled: "{{ 'web-svc-css' in group_names }}", shared: ... }`,
with matching matrix coverage added to each role's
`meta/variants.yml` (true-pinning + false-pinning entries) so the
dynamic-flag variant lint stays green.
- 41 `# nocheck: playwright-service-flag` rationale strings updated
from the now-incorrect "canonical:cdn alias of static-asset
provider" to "css consumer, no Playwright surface (provider:
web-svc-css)".
Bundled refactor: `roles/web-svc-asset/tasks/01_core.yml` now
includes `web-opt-rdr-domains` directly with a single-entry mapping
to redirect its canonical domain to web-svc-file, instead of having
that wiring live in the global `tasks/stages/01_constructor.yml`
constructor. The `add_redirect_if_group` filter
(`plugins/filter/redirect_filters.py`) and its unit test are dropped
as dead code — both call sites are now role-local.
Dashboard's consumption of the new pattern is intentionally not part
of this commit and follows separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three classes of drift cleaned up across 85 .md files:
- Em-dashes: replaced ` — ` with the contextually correct separator
(colon for `term: explanation`, period to split clauses across
sentences/source lines, parentheses for asides, semicolon for
tight-linked continuations). Two intentional exceptions remain
in `docs/contributing/documentation.md` (the rule literal) and
inside a fenced ```sh block in `docs/contributing/actions/debugging/ci.md`.
- SPOT self-labels: rewrote "This is the SPOT for X" to describe
the subject directly ("This directory holds X" / "This file
describes X"). Cross-references to other artefacts' canonical
role remain unchanged.
- Heading-emoji position: moved leading-emoji headings (`# 🧩 Title`)
to trailing position (`# Title 🧩`) per the headline rule that
emojis MUST come after the title text.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add tests/lint/ansible/roles/test_compose_no_raw_image.py (strict mode):
every `image:` line in roles/<role>/templates/compose.yml.j2 must
resolve through a Jinja substitution so the underlying registry /
repository / tag is declared in the role's meta/services.yml instead of
being pinned in the compose template.
Fix the 8 pre-existing violations the lint surfaces:
- web-svc-coturn: gateway nginx → services.gateway.{image,version}
- web-svc-simpleicons: local build tag → services.application.{image,version}
- web-app-mailu: clamav antivirus → services.antivirus.{image,version}
- web-app-taiga (5×): events, async-rabbitmq, events-rabbitmq, protected,
gateway → top-level services.<name>.{image,version}
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mailu's biber + administrator journeys are owned by two bespoke scenarios
in `files/playwright.spec.js` that exercise mailu's actual iframe-wrapped
auth chain. The shared persona helpers (runBiberFlow / runAdminFlow)
route through the dashboard main-frame Account menu, which depends on
the dashboard's OIDC silent-SSO that is currently in escape.
Block both shared personas via `playwright.env.j2` so the bespoke
scenarios stay the single source of truth for mailu's persona
coverage. Wrap each bespoke scenario in
`safeSkipUnlessEnabled("oidc")` so the no-OIDC matrix variant
collapses the SSO+admin tests cleanly without touching the
unauth surfaces.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Filter no longer takes a `database_volume=` parameter. The volume name is
derived from the central provider's `services.<dbtype>.name` (shared) or
`<entity>_database` (dedicated) — mirroring `plugins/lookup/database.py`'s
`volume_prefix + host` construction. Simultaneous postgres + mariadb on
the same role now raises an AnsibleFilterError with a clear message;
support is deferred per the docstring TODO.
The integration test (renamed from `test_shared_service_volumes.py` to
`test_compose_volumes_call_required.py`) now requires every
`roles/*/templates/compose.yml.j2` to call `compose_volumes`, so the
top-level `volumes:` block is always rendered. Without it, flipping a
shared service to dedicated mode adds a named-volume reference that
Compose rejects with `service "<name>" refers to undefined volume`.
Adds the call to 14 templates that were missing it (dashboard, hugo,
sphinx, bridgy-fed, fediwall, littlejs, mig, mini-qr, navigator,
roulette-wheel, simpleicons, collabora, logout, openresty) and strips
the `database_volume=...` argument from every other role plus Taiga's
now-unused `TAIGA_DATABASE_VOLUME` var.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The shared biber/admin persona helpers in
roles/test-e2e-playwright/files/personas/*.js collapse the persona
scenario when either APP_BASE_URL or CANONICAL_DOMAIN is empty. The
twelve roles below shipped their env template without
CANONICAL_DOMAIN, so the helper fell into the auth-less collapse
branch and silently `test.skip()`-ed the persona scenarios despite
the roles having an actual Keycloak-backed auth surface. Render
CANONICAL_DOMAIN on every role whose spec drives a real persona so
the scenario runs end-to-end and a real regression cannot hide
behind a silent skip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extend the preload condition so a provider role is loaded by the
service-loader when it sits in the round's group_names, not only when
some enabled consumer requires its shared service. In matrix-deploy
rounds where every consumer flips the service to disabled (e.g. a
variant collapsing email/oidc/matomo to false), the provider role is
still in the inventory and its own tasks rely on the preload token,
so the previous narrow `required`-only check left the provider
unloaded and asserts further down the chain failed.
Drop the now-obsolete sys-svc-mail ↔ web-app-mailu preload-ordering
TODO line: the assertion that fired when mailu sat in group_names
but no consumer asked for email is resolved by the new branch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Promote `prefer-template` to error in the spec/persona JS rule block
and convert the nine remaining `a + 'literal' + b` URL-builders to
template literals. Behaviour-equivalent; brings stylistic consistency
with the rest of the codebase, which already uses template literals
throughout, and prevents new `+`-concatenations from sneaking past
the lint floor.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Promote four cheap rules in `eslint.config.js`:
* `no-unused-vars`: warn → error (with `argsIgnorePattern: "^_"`)
* `no-var`: error
* `eqeqeq`: error (always)
* `playwright/no-wait-for-navigation`: off → error
* `linterOptions.reportUnusedDisableDirectives`: off → warn
Add a repo-root CommonJS block so `eslint.config.js` itself sees Node
globals (`require`, `module`).
Fix every flagged offender:
* 138 stale persona-helper imports across 70 spec files
(`isServiceEnabled`, `skipUnlessServiceEnabled`, `runAdminFlow`,
... — leftovers from the SPOT consolidation).
* 4 unused catch bindings → modern `catch {}` syntax.
* 2 unused `dashboardBaseUrl` declarations (minio, moodle).
* 1 unused `label` arg → renamed to `_label` to opt out per the
argsIgnorePattern.
* 1 intentionally-kept legacy helper in web-app-wordpress's spec
receives a per-line `eslint-disable-next-line` with reason; the
preceding docstring already explained why it stays.
* 1 `page.waitForNavigation` in web-app-fediwall's Mastodon OIDC flow
receives a per-line disable with reason — the navigation target
depends on the runtime OIDC issuer; `waitForURL` would need a
runtime-built pattern.
* 10 stale `// eslint-disable-next-line no-console` directives now
surfaced by `reportUnusedDisableDirectives`; removed.
ESLint after this pass: 0 errors, 154 warnings (148 pre-existing
`playwright/expect-expect` plus 6 `playwright/no-force-option` that
were already inherited from `playwright/flat/recommended`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Strip every `req <NNN>`, `req-<NNN>`, `Requirement <NNN>`,
`requirement-<NNN>`, and `docs/requirements/<NNN>-...md` mention from
non-Markdown source files. Comments anchored to requirement numbers
rot when requirements are renumbered or rewritten and conflate
present-state behaviour (which the comment should describe) with the
audit trail of how the code got there (which belongs in commit
messages, PR descriptions, and the requirement file itself).
* Document the rule in `docs/contributing/documentation.md` under
"Comments". Code MUST describe the *why* in present-state language;
contract enforcement lives in tests, not folklore.
* Clean 254 source files across `roles/`, `tests/`, `plugins/`,
`cli/`, `scripts/`, `utils/`, `group_vars/`, and the repo-root
`eslint.config.js`. Pure comment / docstring / log-message edits;
no behavioural change.
* Add `tests/lint/repository/test_no_req_references_in_code.py` to
enforce the rule going forward across `.js`, `.py`, `.yml`,
`.yaml`, `.j2`, and `.sh` files. The legacy
`test_no_stale_requirement_refs.py` is exempt as the only file
whose purpose is to detect these very tokens.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fourteen role specs (bigbluebutton, discourse, fider, gitea, mailu,
matrix, mattermost, odoo, openwebui, peertube, postmarks, prometheus,
yourls, libretranslate) each carried an inline `performOidcLogin`
that drove the Keycloak login form. The bodies fell into four
clusters split between the role-based selector strategy
(`getByRole`) and the legacy input-name strategy
(`input[name='username']`); the existing shared
`performKeycloakLogin` already covered both via `.or()` chains but
also asserted post-login navigation, so it could not be reused
verbatim.
Changes
-------
- `personas/utils/keycloak.js` gains `performKeycloakLoginForm
(target, username, password)` — form-fill + submit only, accepts
`Page` OR `FrameLocator` so iframe-embedded Keycloak pages work
without branching at the call site. The selectors are the union
of all four prior cluster shapes (role-based getter chained with
`.or(input-name fallback)`, sign-in matched against
`/sign in|login|log in/i` plus `input#kc-login` family). The
existing `performKeycloakLogin` and
`performKeycloakLoginExpectingDenial` now delegate to the new
helper, so their public API is unchanged.
- The 14 specs strip their inline `performOidcLogin` and amend
`require("./personas")` with `performKeycloakLoginForm`. Every
call site and comment reference is renamed in lockstep.
Drift handling
--------------
- Cluster 1 (5 specs) expected the accessible name `"Password"`
exactly; the canonical regex `/^password$/i` is case-insensitive
and matches every prior name plus lowercase variants — strictly
more permissive.
- Cluster 1's sign-in button matched `/sign in/i`; the canonical
matches `/sign in|login|log in/i` plus the legacy `input#kc-login`
family — strictly broader.
- Cluster 1 awaited `usernameField.press("Tab")` without a `.catch`;
the canonical wraps the press in `.catch(() => {})` so an
unexpected Tab failure no longer aborts the flow.
- Cluster 2 / 3 used `expect(usernameField).toBeVisible({ timeout })`
for the visibility wait; the canonical uses
`usernameField.waitFor({ state: "visible", timeout })`, the
Locator-method counterpart of the same wait — functionally
equivalent.
Verification
------------
- 419 passed / 1 skipped / 0 failed across `tests/lint/`,
`tests/integration/roles/`, `tests/unit/plugins/lookup/`.
- `make lint-javascript`: 0 errors, 283 warnings (unchanged from
baseline).
- 0 inline `performOidcLogin` definitions remain in any role spec,
and 0 stale references to the legacy name in any context (call
sites and comments are renamed in the same pass).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both helpers were inlined verbatim in 71 of the role-local
`playwright.spec.js` files (52 with the compact form, 19 with the
verbose form, plus one parameter-name variant). All bodies were
semantically identical.
Changes
-------
- New `roles/test-e2e-playwright/files/personas/utils/dotenv.js` holds
the canonical pair (`decodeDotenvQuotedValue` and `normalizeBaseUrl`),
re-exported through `personas/utils/index.js` so specs that
`require("./personas")` get them via the existing aggregator.
- Every role spec drops its inline copies and pulls the helpers from
`./personas` instead. Specs that only reference one of the two
carry only that one in the destructuring (no unused-import noise).
Doc
---
`docs/contributing/artefact/files/role/playwright.specs.js.md` no
longer carries the "once a third spec needs them, MUST be promoted"
directive: the helpers live at the shared path and inline copies are
forbidden.
Verification
------------
- 419 passed / 1 skipped / 0 failed across `tests/lint/`,
`tests/integration/roles/`, `tests/unit/plugins/lookup/`.
- `make lint-javascript`: 0 errors, 283 warnings (unchanged from
baseline; no new unused-imports introduced).
- 0 inline `function decodeDotenvQuotedValue|normalizeBaseUrl`
definitions remain in any role spec.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cross-service surface assertions for dashboard, prometheus and matomo
no longer run in every consumer role's persona. They are owned by the
provider's own spec, parameterised once over a `<NAME>_TARGET_ROLES_JSON`
manifest rendered into the provider's `templates/playwright.env.j2` via
the new `lookup('roles_with_service', '<svc>')` Ansible lookup.
Changes
-------
- New `plugins/lookup/roles_with_service.py` enumerates roles that
declare `services.<svc>.{enabled, shared}` truthy in the merged
applications config, returning `[{id, canonical_domain}, …]`.
- `roles/web-app-{dashboard,prometheus,matomo}/files/playwright.spec.js`
parse `<NAME>_TARGET_ROLES_JSON` and parameterise the per-consumer
assertion (dashboard tile reachability, prometheus `up=1`, matomo
SitesManager registration). The matomo spec gains an explicit
biber-denial test (request body validated against admin DOM markers
per the body-marker rule); the prometheus admin-reach test drops
the dashboard-iframe path in favour of direct goto.
- Persona helpers `roles/test-e2e-playwright/files/personas/{admin,
biber,guest}.js` enter at `${APP_BASE_URL}/` directly. The
`clickRoleTileFromDashboard` step and the `assertPrometheus*` /
`assertMatomo*` cross-service probes are gone; `utils/{dashboard,
prometheus,matomo}.js` are deleted. `utils/csp.js` switches to
literal `process.env.<KEY>` reads so the env-keys-used parity guard
recognises matomo's CSP injector consumption.
- 42 consumer-role `templates/playwright.env.j2` templates drop
`DASHBOARD_BASE_URL`, `DASHBOARD_SERVICE_ENABLED`, `PROMETHEUS_BASE_URL`,
`PROMETHEUS_SERVICE_ENABLED`. `MATOMO_BASE_URL` and
`MATOMO_SERVICE_ENABLED` stay because the per-role CSP injection
helper still gates on matomo's tracker host.
- `tests/integration/roles/test_playwright_env_services_match.py`
globally exempts `dashboard` and `prometheus` from the
services.yml-to-env parity rule.
- `tests/unit/plugins/lookup/test_roles_with_service.py` adds 14
unit tests for the lookup, including an explicit-contract test
that pins the over-inclusive behaviour on raw Jinja strings (the
lookup expects a templar-rendered payload).
- Docs: `docs/requirements/019-playwright-meta-services-parity.md`
rewords Rules 1, 3, 12 and adds Rule 13 describing the SPOT shift.
`docs/contributing/artefact/files/role/playwright.specs.js.md`
reflects the new persona shape and per-service catalogue, and is
cleaned per `docs/contributing/documentation.md` (no SPOT self-
labels, no req-references, em dashes replaced with complete
sentences, multi-clause lines split onto separate lines, broken
anchor `#two-personas-fixed-flow-` corrected).
Verification
------------
- 418 passed / 1 skipped / 0 failed across `tests/lint/`,
`tests/integration/roles/`, `tests/unit/plugins/lookup/`.
- `make lint-javascript`: 0 errors.
- `markdownlint-cli2`: 0 errors on all touched docs.
- `make autoformat`: clean (FURB171 single-item-set inequality fixed).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a lint that fails when a role's templates/playwright.env.j2 declares
DASHBOARD_SERVICE_ENABLED= but does NOT render DASHBOARD_BASE_URL,
PROMETHEUS_BASE_URL, and MATOMO_BASE_URL. Renders those three URLs
across 42 role env templates so the lint passes.
Also reverts the matomo "role-closed" strikethrough in req 019 — the
URL wiring remains in scope for the persona rollout.
Reason: the persona helpers (roles/test-e2e-playwright/files/personas/
{biber,admin,guest}.js) collapse the persona scenario when their
required URLs are empty (req 019 persona-collapse exception). Without
this lint, a role that declares dashboard: enabled but forgets the
URL silently SKIPs the persona — the deploy goes green but no real
flow ran. Per-line opt-out is `# nocheck: persona-required-envs` for
auth-less svc-* roles.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduces roles/test-e2e-playwright/files/personas/ — runGuestFlow,
runBiberFlow, runAdminFlow plus 10 utils (csp, dashboard, env, keycloak,
landing, logout, matomo, prometheus, interaction, index) — and wires
both the Ansible runner (run_one.yml) and the local rerun helper
(scripts/tests/e2e/rerun-spec.sh) to stage the tree alongside
service-gating.js so specs can `require("./personas")`.
First batch of role specs migrated to consume the shared library
(dashboard, keycloak, mailu, matomo, prometheus, web-svc-cdn,
web-svc-simpleicons) — each gets the spec.js persona scenarios plus
the matching meta/services.yml + playwright.env.j2 entries; matomo is
marked role-closed in req 019. dashboard/keycloak/mailu TODO.md
updated; docs/requirements/019 reflects matomo closure.
Reason: req 019 Rule 3 mandates that every role's guest/biber/
administrator flow runs the same canonical steps; centralising them
in one shared library keeps per-role drift from re-introducing the
stub-shaped bodies the new lint forbids.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Picked up by the make-test setup phase regenerating generated artefacts
during recent test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh:
- rename run_sort_claude_settings worker to run_claude (matches the
renamed scripts/lint/claude.sh).
- drop the EXIT trap that wiped status_dir; cleanup happens explicitly
after `wait` returns, so a parallel worker that interrupts the
parent never races a vanished status dir.
- each worker now captures its tool's exit code and writes one of
OK / SKIP / FAIL. Tolerated `expected violations remain`-style exits
(markdownlint-cli2 rc≤1, ansible-lint rc≤2) still count as OK; only
real failures count as FAIL.
- all workers run to completion regardless of individual tool failures
(so the user always sees the full picture); after the summary, the
script exits 1 if any worker reported FAIL.
* scripts/lint/sort_claude_settings.sh -> scripts/lint/claude.sh
(history preserved via git mv); update test_claude_settings_sorted.py
fix-hint accordingly.
Plus working-tree drift across roles/, group_vars/, inventories/,
requirements/, and .pre-commit-config.yaml accumulated from the
make-test setup phase regenerating generated artefacts during recent
test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh: rename AUTOFORMAT_PARALLEL to PARALLEL,
declare it at file start with default `true`; still env-overridable.
Reword the worker comment so the literal `# shellcheck` token no
longer appears at start-of-line and trips SC1072/SC1073.
* scripts/lint/galaxy.sh: drop the GALAXY_FATAL / GALAXY_NAMESPACE /
GALAXY_PARALLEL env vars (over-engineered for a single repo). Failure
is always fatal. NAMESPACE is an internal constant. Add the same
PARALLEL env var (default `true`, env-overridable); falsy demotes the
xargs fan-out to one worker.
* docs/contributing/environment/variables.md: collapse the four
AUTOFORMAT_*/GALAXY_* rows into one `PARALLEL` row covering both
consumers.
* docs/contributing/tools/make.md: refer to `PARALLEL=0` everywhere;
add the lint-galaxy parallelism note.
Plus working-tree drift across roles/, group_vars/, inventories/, and
tasks/ that accumulated from the make-test setup phase regenerating
generated artefacts during recent test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* .pre-commit-config.yaml: add 6 scoped lint hooks (lint-action,
lint-ansible, lint-python, lint-shellcheck, lint-markdown, lint-galaxy)
with `files:` patterns so a markdown-only commit no longer triggers
ansible-lint or galaxy-importer. Aggregate make-test, make-test-lint
(md-only branch) and pre-push make-test-external are kept.
* Makefile: drop `lint` from the `make test` dependency chain. Per-file-
type lint stages now live exclusively in pre-commit; running them again
from `make test` would just duplicate work.
* scripts/lint/autoformat.sh: add markdownlint-cli2 --fix and
ansible-lint --fix passes alongside ruff/shfmt/shellcheck.
* roles/svc-db-openldap/meta/main.yml: collapse the wrapped description
into a single line (still under Galaxy's 255-char limit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tooling integration
* add .markdownlint-cli2.jsonc, .ansible-lint, .ansible-lint-ignore
configs (permissive day-1: cosmetic rules disabled, semantic ones on)
* extend scripts/install/lint.sh with generic install_pip_pkg helper +
ensure_ansible_lint / ensure_galaxy_importer / ensure_markdownlint_cli2
ensure_* functions (pip / npm with sudo / --user / --break-system-packages
fallbacks). install_ansible_tools now also pulls ansible-lint and
galaxy-importer; new `markdown` install group.
* extend scripts/lint/ansible.sh to run ansible-lint after the existing
syntax-check.
* add scripts/lint/markdown.sh and scripts/lint/galaxy.sh runners. The
galaxy runner parallelizes via xargs (-P=$(nproc)), prints OK/FAIL per
role, and treats `bare-name dependency` errors as success since our
roles/ tree is not a Galaxy collection layout. All other Galaxy schema
errors still fail.
* Makefile: new lint-markdown and lint-galaxy targets, both wired into
`make lint`.
* CI: new workflows lint-markdown.yml and lint-galaxy.yml; lint-ansible.yml
now installs ansible-core + ansible-lint; ci-orchestrator wires both
new lint workflows into lint-gate.
Galaxy schema fixes
* shorten the canonical `galaxy_info.company` block from three lines to
two ("Kevin Veen-Birkenbach\nhttps://www.veen.world", 45 chars) so it
stays under Galaxy's hard 50-char limit. Updated everywhere: 229 role
meta/main.yml plus _CANONICAL_COMPANY in
tests/lint/repository/test_role_meta_main_galaxy_schema.py.
* shorten 19 over-long role descriptions to <=255 chars.
* add stub READMEs for sys-svc-compose-ca and sys-utils-service-loader.
Markdown baseline fixes
* markdownlint-cli2 --fix run repo-wide (formatting normalization
across CHANGELOG, role READMEs, docs).
* manual MD041 fixes: prepend H1 in 15 files (CHANGELOG, LICENSE,
TODO.md helpers, role README templates, etc.).
* MD025 / MD003 / MD001 / MD032 manual fixes for heading-style and
list-spacing edge cases.
* role README content polish: "## Single Point of Truth (SPOT)" headings
renamed to "## User data resolution" (D3 self-label rule).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* meta/main.yml: replace every prior platforms variant (Linux,
GenericLinux, Any, Archlinux+rolling/all, distro-specific pins) with
the canonical 5-distro list (ArchLinux, Debian, EL, Fedora, Ubuntu;
versions: [all] each).
* lint: merge tests/lint/ansible/roles/test_meta_company_canonical.py
into tests/lint/repository/test_role_meta_main_galaxy_schema.py and
add a parallel canonical-platforms exact-value check; one file walk
enforces both schema conformance and the two project canonical
exact-values (company block-scalar + platforms 5-distro list).
* roles/<role>/README.md: strip emojis from H1/H2 headings (per
readme_md.md role rule), normalize the Credits block to the canonical
wording (Author/Authors/Author Information aliased; canonical block
appended where absent).
* repo-wide: revert text-drift `GenericLinux` -> `Linux` in role
description fields, README body prose, vars, j2 templates, and the
Makefile (87 occurrences). The lint docstring keeps a single
historical mention.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Collapse the prior ~10 variants of galaxy_info.company (single-line,
multi-line single-quoted, multi-line double-quoted, escaped \n,
Unicode non-breaking hyphen, Birchenbach typo, block-scalar) into one
canonical block-scalar form across every roles/*/meta/main.yml that
declares the field:
company: |
Kevin Veen-Birkenbach
Consulting & Coaching Solutions
https://www.veen.world
Add tests/lint/ansible/roles/test_meta_company_canonical.py so future
drift fails CI immediately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
