`process.env.X || <value>` is a second source for something the deployment
already renders, and the two drift without a sound. playwright.config.js read
`process.env.PLAYWRIGHT_ACTION_TIMEOUT || 30_000` while nothing ever set that
variable, so every run used the unscaled 30 s and no failure said so.
Normalising an absent variable stays allowed, because it substitutes nothing:
`|| ""`, `|| '[]'`, `|| false`, `|| 0` and falling through to another
`process.env` entry only let the caller test absence itself.
The six sites the rule found:
n8n and flowise built a Postgres port from `|| '5432'` while env.j2 renders it
from `lookup('database', application_id, 'port')`. Node throws
ERR_SOCKET_BAD_PORT on the resulting NaN, so an unfed port now names itself
instead of connecting somewhere plausible.
The bluesky broker did the same with BROKER_PORT, which compose.yml.j2 renders
unconditionally. Its INSECURE_TLS default was `"false"`, which is what `""`
already evaluates to.
matomo defaulted MATOMO_TRACKING_SCOPE to "sub"; the only comparison is against
"root", so `""` takes the identical branch.
moodle and simpleicons defaulted a name their playwright.env.j2 always renders.
simpleicons already asserts the value is truthy; moodle now fails on the scope
it cannot find rather than on a name nobody configured.
Four values the code itself owns keep a default under `// nocheck: env-default`:
PDS_INTERNAL_URL (which falls through to requireEnv, not to a literal),
HANDOFF_COOKIE_NAME, HANDOFF_COOKIE_MAX_AGE and PASSWORD_SALT_HASH_ROUNDS. No
role renders any of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Playwright declares the timeout option on locator.isVisible() and isHidden() as deprecated and ignored: both return immediately and never wait. Passing resolveTimeout(10_000) to one therefore changes nothing at runtime and only tells the reader a wait happened that never did. That reading is what let 7538b8a0b conclude two genuine call sites were fine, and what hid the false negative b97cf8888 had to fix.
Sixty call sites across fifty-eight role-local specs lose the argument and nothing else. Every added line in this commit is a bare isVisible(); a grep for added lines that are not one returns zero, so a behaviour change cannot have entered here. Files whose diff also alters waiting semantics are deliberately not part of this commit.
A repository lint added in b97cf8888 keeps the construct out. It matches isVisible and isHidden only, because isChecked, isDisabled, isEditable and isEnabled take a real timeout.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve conflicts, resolved by intent rather than by side.
Six shared one shape: the incoming side adds the new rescue_index.sh call but
drops this branch's `# nocheck: shell-or-true` markers. The markers carry a
lint contract here, so the incoming structure was taken and the markers put
back — dedicated.sh, guide/one.sh, guide/host_deploy.sh, swarm/routine/00_one.sh
and trigger_units.sh, where the incoming version additionally writes the full
journal to a file instead of only echoing a tail.
web-app-matrix keeps HEAD's _synapse_extra_args accumulator (the superset: CA
mount plus the onion case) but adopts the incoming improvement inside it — the
inlined regex_replace gives way to lookup('subnet_gateway').
handlers/__init__.py drops rescue_diagnostics_dir from the old import group
(the incoming side moved it under .infinito.rescue, keeping both would import
it twice) and keeps tor_socks_port, a HEAD-only handler ORDERED_HANDLERS still
needs.
svc-bkp-volume-2-local takes the incoming side throughout: both branches landed
the same feature, but swarm-nfs evolved it — the knob moved from the global
STORAGE_BACKUP_SNAPSHOT_MODE to services.volume-2-local.snapshot_mode, plus the
subvolume helper, 04_snapshot_subject.yml, meta/packages.yml and baudolo 3.2.2.
Three files that left behind were removed: files/baudolo_snapshot.py and the two
worktree tests at their pre-refactor paths, all byte-identical to their moved
counterparts.
Six follow-on failures the merge produced, fixed at the root:
- STORAGE_BACKUP_SNAPSHOT_MODE lost its last reader with the config-lookup
design and is gone from group_vars.
- utils/github/variant_bundle_limits.py sat next to variant_bundles.py, which
the prefix lint forbids; the pair now lives in utils/github/variant/ as
bundles.py and bundle_limits.py, with the -m entry point, the ruff
per-file-ignore, the design-doc link and the mirroring test path updated.
- The incoming WordPress addon spec navigated with a raw page.goto and carried
three bare timeout literals; both route through gotoOnion and resolveTimeout
now, which is what makes navigation survive onion transport.
- Six unmarked `|| true` got a concrete reason each: docker.socket is absent on
distros that ship docker without it, the loop device is already detached when
the probe failed before attaching, and modprobe zfs may fail because the
/dev/zfs check decides.
make quality green across all four targets. The cold-domains budget test that
had to be waived here passes on its own now — 19d56f07e raises it to 100s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three of the six failures triaged today were explainable only after downloading
a rescue artifact, and two not even then. The job log carried "not converged:
docker_docker", a bare "rc=1", and "failed after 3 attempt(s)".
The convergence gate now prints the task rows of a service that will not
converge - but only those whose error field is filled. A stack coming up
normally stays silent, while the one that killed svc-db-postgres would have
shown `error=Head "https://ghcr.io/...": request canceled` in the job log
instead of hiding it in an artifact. awk rather than grep: grep exits 1 when
nothing matches, which under set -e would take the script with it.
The remote pull decodes its exit status. os.system returns rc << 8, so the
message said 5888 where rsync said 23, and every attempt now names its code.
The sleep after the final attempt is gone - five wasted minutes per run.
The matomo poller hands its last 40 lines to stderr on deadline, redacted at
the source, and the failing task interpolates them the way the compose flavor
already does. No no_log: MASK_CREDENTIALS_IN_LOGS is force-set anyway and would
have censored exactly the payload this exists to deliver.
The `-p warning` filter added to trigger_units.sh a few commits ago is removed.
systemd journals a unit's stdout and stderr at info, so that filter would have
swallowed the per-provider attribution the same commit introduced.
stack_ready.sh gets its first tests: replica comparison, the completed-oneshot
branch, and both halves of the new reporting - the error reaches the caller, the
error-free rows do not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The spec navigates through gotoOnion and carries no timeout literals of its
own, so the import had no consumer and failed eslint's no-unused-vars, which
took the whole CI lint job down.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Swarm can replace the matomo task mid-patch; a container id resolved earlier
then points at a dead container and the config patch fails with
"container ... is not running" (observed in the swarm CI matrix deploy).
Patch config.ini.php directly on the matomo_data volume (state NFS export is
no_root_squash on both kernel and ganesha flavors), so the DB-credential (05)
and trusted-hosts (06) writes no longer depend on container liveness. Resolve
the host bind path once via a shared utils/resolve_config_host.yml.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Conflicts: keep both new passthrough env vars; adopt the swarm-nfs
ollama manager-ops split while porting our model-pull retry hardening
into it. Strip three narration comments the stricter comment lint
rejects.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The node-local matomo_tmp volume populated as root on any node that had not run the first-install entrypoint: the matomo image ships /var/www/html empty (code lives in /usr/src), and the entrypoint's chown -R www-data runs only when matomo.php is absent, which on a rescheduled task is never the case because matomo_data is shared NFS. The rescheduled task then died on 'tmp/cache/tracker could not be created', pausing every forced service update. Build a custom image whose /var/www/html/tmp is www-data-owned: docker volume populate replicates the image path's ownership into every fresh volume on every node (proven via moby populateVolume/CopyDir and a local reproduction), so reschedules get a writable tmp deterministically.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Matomo's swarm bootstrap failed rc=1: the high-churn tmp/templates_c Twig cache lived on the soft NFS state share, so a single mid-write RPC timeout surfaced as EIO and stalled the installer 300s; the DB was already seeded, so retries took the eraseAllTables path that wedged Matomo on the CoreUpdater screen the bootstrap image cannot click through. Mount a node-local matomo_tmp volume (nfs: false) for /var/www/html/tmp so caches never traverse NFS, and drop+recreate the app's own database in the pre-retry cleanup (via a container-run reset_database.php over the app credentials, gated exactly like the config-aside task on attempt>0 + no installed config) so every retry takes the clean-install path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Over Tor the redirect chain behind a login/logout/OIDC click routinely exceeds Playwright's fixed 30s action timeout, so the click throws before the flow's scaled waits run. Sweep all role Playwright suites and the shared persona helpers: give every navigation/auth-transition click an onion-scaled timeout (.click({ timeout: resolveTimeout(30_000) })) and route the remaining raw millisecond literals (timeout options, waitForTimeout, sleep-promise setTimeout, positional wait args) through resolveTimeout. Local UI clicks (menus, toggles, modal dismissals, DOM clicks inside evaluate) stay untouched.
Two new lints keep it that way: test_no_raw_timeout_literals forbids raw numeric timeout literals in Playwright specs (opt-out: nocheck: raw-timeout), test_persona_clicks_scaled requires every click in the shared persona auth flows to carry a resolveTimeout timeout (opt-out: nocheck: unscaled-click).
Verified: node --check on all 133 changed JS files, all 13 playwright lints green, make autoformat clean, make test green (2745+196+163+4 tests OK).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A flaked installer that created the superuser and site but never reached
the finished step leaves Matomo redirecting every request to CoreUpdater,
so the sidecar probe never passes. The old until-retry only force-restarted
the sidecar between attempts and never reset that partial state, so all
three attempts failed identically and the job died after ~29 minutes
(everything depending on matomo hung with it).
Drive the attempts through a loop over swarm_attempt.yml that, before each
retry, moves the partial config aside (config.ini.php.wedged-<ts>, never
deleted) and clears the cache so the installer runs fresh; the eraseAllTables
step handles leftover tables. The move is gated to the config-absent-at-entry
path and skips the whole attempt once a token is obtained, so a healthy
install is never touched. bootstrap_swarm_poll.sh anchors its token scrape
with --since so a token still buffered from the previous task is not returned
before the restarted installer emits its own, and surfaces a failed move
instead of masking it. The swarm abort message reports the swarm attempt
count.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New lint test_no_raw_goto.py forbids raw `page.goto(...)` in role playwright specs + persona flows (the gotoOnion wrapper in personas/utils is exempt) so navigation retries transient Tor-transport errors over onion. Migrates every raw call across ~75 roles (343 sites) to `gotoOnion(page, ...)` and teaches gotoOnion to detect relative onion URLs (`isOnionCanonical`). gotoOnion is a behaviour-identical drop-in on clearnet (single attempt, same return). Verified: no-raw-goto lint green, eslint 0 errors, make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the global TIMEOUT_FACTOR and a shared timeouts.js helper: resolveTimeout(base) scales by TIMEOUT_FACTOR and, over a .onion target, an extra multiplier, so Tor's per-request latency no longer trips hard-coded timeouts. Migrate every raw timeout in the specs (timeout:/timeout=/setTimeout/waitForTimeout) to resolveTimeout, stage the helper in all three staging paths, and add test_no_raw_timeouts to forbid raw numeric timeouts. Clearnet behaviour is unchanged (factor 1: resolveTimeout(N) === N).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In swarm the bootstrap service now deploys at replicas: 0 (created but unscheduled); the poll script scales it 0->1 with --force --replicas 1 so exactly one fresh installer run prints the token while the poll is already scraping its logs, removing the deploy-time-vs-poll-time race. Compose is unaffected (bootstrap stays profile-gated, run via run --rm).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a per-deployment Matomo tracking scope, defaulting to per-subdomain:
- services.matomo.tracking.scope 'sub' (default) registers one Matomo site per
consumer subdomain with an isolated cookie; 'root' keeps the prior behaviour
of one shared site per registrable base domain with cross-subdomain linking.
- sys-front-inj-matomo derives matomo_site_domain (full host vs registrable
base) and the shared matomo_site_url from it; getSitesId / addSite and the
tracking snippet (setCookieDomain / setDomains / enableCrossDomainLinking) all
follow the scope.
- The matomo Playwright env emits MATOMO_TRACKING_SCOPE; the SitesManager test
matches the consumer's full host in 'sub' and its base domain in 'root'.
- variants.yml variant 0 sets scope=root so the matrix exercises both flavours
(every other deploy runs the sub default).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Break the single 457-line playwright.spec.js into topic files discovered by the
harness testMatch "**/*.@(spec|test).js", mirroring the pihole / gitea layout:
- _shared.js env vars, the setup hook, loginAsAdmin, attachDiagnostics,
hostOf / baseDomainOf and the MATOMO_TARGET_ROLES_JSON list
- playwright.spec.js CSP + canonical-domain smoke (filename kept; harness asserts it)
- auth.spec.js admin login/logout and biber denied at the admin surface
- tracking.spec.js SitesManager site registration and per-consumer _paq injection
- persona-flows.spec.js guest / biber / admin universal-logout flows
Test logic is unchanged: same titles, selectors, regexes, timeouts and
assertions (verified by title parity and an adversarial diff against HEAD). The
only behavioural delta is that the extracted loginAsAdmin waits for the username
field to be visible before filling, which the SitesManager login previously
skipped (hardening only). The duplicated admin-login form-fill and the old
beforeEach are now the shared loginAsAdmin and setupMatomoPage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The matomo injection registers one shared SitesManager site per registrable base
domain (addSite urls=proto://base_domain), so a subdomained consumer such as
logout.infinito.example is tracked under the infinito.example site, not its own.
The SitesManager test asserted the full canonical host as a substring of a site
URL, which never matched the base-domain site and failed in swarm.
Reduce the consumer's canonical_domain to its base domain (mirroring the regex in
sys-front-inj-matomo/vars/main.yml) and match it against each site's hostname
exactly. The exact match also drops the substring false positive, where a site at
evil-infinito.example would have satisfied infinito.example. Per-consumer tracker
presence stays covered by the separate per-role injection test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Matomo 5.x rejects the SitesManager.getAllSites API call with 401 when only the browser session cookie is sent; it requires token_auth. Surface the admin API token into the Playwright env (MATOMO_API_TOKEN) and pass it in the request POST body so it stays out of the URL/logs. Mode-agnostic test/env change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
enable_framed_pages=1 disables Matomo's JS frame-buster so the dashboard can iframe-embed it (CSP frame-ancestors still gates who may frame). Shared web-svc consumers (logout, cdn, file) render body_filter.lua in the earlier web-svc service-loader bucket, before web-app-matomo writes its API token, so the tracker is gated off and never re-rendered; after the token is persisted, re-run sys-front-inj-all for each web-svc matomo consumer so the tracker lands. The loop is restricted to web-svc-* because re-running it for a web-app consumer outside its own play would dereference unloaded role vars and crash; web-svc roles declare no javascript injection.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Swarm bootstrap fix: the single 'container service update --force' plus one-shot log-poll aborted on a flaky installer (restart_policy:no never re-runs it). Merge force-update and token-poll into one retried block (each attempt re-forces a fresh installer task, 2-3 attempts); the poll loop lives in files/bootstrap_swarm_poll.sh (script module, vars passed as env).
Restructure: split the bootstrap into mode flavors (05_bootstrap/flavors/{swarm,compose}.yml) and per-mode debug (05_bootstrap/debug/{swarm,compose}.yml), both selected by one DEPLOYMENT_MODE-driven include; renumber 07-10 to 06-09. Swarm-only behavior changes; compose paths unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Splits the matomo install path into compose-mode and swarm-mode
manager-side flows so the role works under both deployment models:
- 03_manager_ops.yml: dual `when: DEPLOYMENT_MODE == 'swarm'` /
`\!= 'swarm'` blocks discover the matomo container differently
(`container service ps` for swarm to wait for task convergence +
resolve the hosting node; `container inspect` for compose). Both
blocks set MATOMO_HOST_NODE so the downstream steps `delegate_to:`
the correct node. Container-ID resolution then takes the right
branch via an inline Jinja toggle.
- 05_bootstrap.yml: wraps the compose and swarm bootstrap-launch
tasks in a single `block:` with a paired `rescue:` that loads
06_bootstrap_debug.yml (only under MODE_DEBUG) and then hard-fails
with the captured rc/stdout/stderr from the failed run. Drops the
redundant `| default('')` on `matomo_bootstrap_run.stdout` inside
the swarm `until:` (the lint catches it; the attribute is always
set on a registered task).
- 04_install_path.yml: removes the stale `when: matomo_token_value …`
gate on the api-config include that referenced the deleted
silent-skip rescue. 09_config.yml now runs unconditionally; the
legitimate token-shape assertions live in 05_bootstrap.yml.
- files/patch_trusted_hosts.sh: adds the missing `#\!/bin/sh` shebang
so lint-shellcheck stops flagging SC2148.
Plus a lint bugfix:
- tests/lint/ansible/structure/test_no_oversized_ansible_block.py
was counting `rescue:` / `always:` siblings as direct block
children, double-counting them under the >3 limit. Walker now
stops at `rescue:`/`always:` keys aligned with the `block:`
keyword (each section has its own child list), and computes the
terminator column from the `block:` keyword itself instead of the
regex-captured indent (which is 0 for the `- block:` form because
the dash absorbs into the prefix group).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Renumber the tasks/ tree to a flat NN_ scheme (drop the 02b / 03a / 03b
suffixes) and update all include_tasks references; main.yml -> 01_core
stays unchanged. Lift the hardcoded matomo bootstrap timezone label
into a global HOST_TIMEZONE_LABEL constant in group_vars/all so other
apps can share it; default to "United Kingdom" because GB has a single
IANA timezone and matomo's getTimezoneName collapses to the country
name in that case. Drop unconditional double quotes from env.j2 and
bootstrap.env.j2 since docker stack deploy preserves env-file quotes
literally and was feeding matomo "matomo" instead of matomo as the
DB name. Add MATOMO_ADMIN_EMAIL via the users lookup so the bootstrap
template can reference a typed variable instead of constructing the
address inline. Switch 08_patch_trusted_hosts to ansible.builtin.shell
with the script body piped in via stdin from lookup('file', ...) and
shrink the script file to just the inner sh body. Strip section-banner
and restate-the-code comments from vars/main.yml, env.j2 and the
extracted script; keep the two genuine trip-wire comments (matrix
deploy round-2 DB creds, include_role-chain role_vars pollution) in a
shorter Exception form.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements docs/requirements/021-sso-flavor-migration.md atomically:
* Schema: every role's meta/services.yml replaces the parallel
`oauth2:` / `oidc:` top-level blocks with a single `sso:` block
carrying `enabled`, `shared`, and a `flavor` discriminator
(`oidc` | `oauth2` | `saml`; default `oidc`). Flavor-specific
sub-keys live under `sso.<flavor>.*` (`sso.oauth2.{origin,acl,
allowed_groups}`, `sso.oidc.plugin`).
* Provider role: roles/web-app-oauth2-proxy/ is deleted in its
entirety; its 5 sidecar templates move under
roles/web-app-keycloak/templates/sso_proxy/, and the per-consumer
rendering task lives at roles/web-app-keycloak/tasks/sso_proxy.yml.
web-app-keycloak now declares `provides: sso`.
* Runtime: new resolver utils/roles/applications/services/sso.py
centralises the SSO predicates (`is_enabled`, `is_proxy_gated`,
`is_oidc_native`) and the oauth2 sub-values
(`oauth2_origin_{host,port}`, `oauth2_acl`, `oauth2_allowed_groups`).
Three lookup plugins consume the resolver: `sso` (single-app
predicate access), `sso_proxy_consumers` (enumerates oauth2-flavored
consumers), `sso_oidc_plugin` (renamed from `oidc_flavor`, Nextcloud
plugin selector). sys-stk-backend, sys-svc-compose, sys-svc-proxy
and plugins/filter/compose_volumes.py all gate on
`lookup('sso', ..., 'is_proxy_gated')` instead of manually combining
`enabled` and `flavor`.
* Identifier renames repo-wide: `services.<entity>.ports.local.oauth2`
→ `ports.local.sso`; `PORT_BANDS.local.oauth2` → `PORT_BANDS.local.sso`;
`OAUTH2_PROXY_*` env/Jinja vars → `SSO_PROXY_*`;
`oauth2_proxy_cookie_secret` → `sso_proxy_cookie_secret`;
`OAUTH2_SERVICE_ENABLED` + `OIDC_SERVICE_ENABLED` collapse into
`SSO_SERVICE_ENABLED`; Playwright gating helpers
(isServiceEnabled/skipUnlessServiceEnabled/requireService/safeIsEnabled/
safeSkipUnlessEnabled) move from `oidc`/`oauth2` to `sso`.
* Dual-block resolution: per req's Background-Table, web-app-bookwyrm
is `flavor: oauth2`; web-app-gitea and web-app-friendica keep
`flavor: oauth2` with their existing `oauth2.acl.blacklist`
(/user/login) and `oauth2.acl.whitelist` (federation + discovery
endpoints) — preserving the operative oauth2-proxy gating.
* Tests: the two mutual-exclusion guards
(tests/integration/iam/oauth2_oidc/test_mutual_exclusive.py and
test_acl_mutual_exclusive.py) are deleted per Decision #10;
test_oauth2_contract.py is renamed and rewritten to
test_sso_contract.py, asserting that `flavor: oauth2` roles carry
`oauth2.origin.{host,port}` plus `ports.local.sso`; test_sso.py
(web-app integration) now requires the unified `sso:` block with an
explicit flavor or a `# nocheck: sso` opt-out; new
tests/lint/repository/no_legacy_sso_paths/ guards the source tree
against re-introducing the legacy strings.
* Docs: req 21 itself has every AC checked off, and reqs 006, 008,
009, 013, 015, 016, 019, 020 plus 10+ role READMEs and
docs/contributing/actions/testing/suppression.md are updated to the
unified SSO naming.
Empirically validated end-to-end against the validation app set
(web-app-keycloak, web-app-nextcloud, web-app-bookwyrm, web-app-gitea,
web-app-friendica, web-app-dashboard) in matrix variants 0 and 1; v2
also passes the 4 target roles' Playwright after a documented
pre-existing KC_HOSTNAME-drift workaround (separate concern, not
caused by this refactor).
BREAKING CHANGE: every consumer of `services.oauth2.*`,
`services.oidc.*`, `OAUTH2_PROXY_*`, `OAUTH2_SERVICE_ENABLED`,
`OIDC_SERVICE_ENABLED`, `oauth2_proxy_cookie_secret`,
`oauth2_proxy_application_id`, `ports.local.oauth2`, the
`web-app-oauth2-proxy` role, or the legacy gating helper service names
(`oidc`/`oauth2`) must migrate to the unified `sso` shape documented
in docs/requirements/021-sso-flavor-migration.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Removes prose block comments, JSDoc headers, and multi-line nocheck
rationale paragraphs added in the previous two commits — keeps only
the bare nocheck markers (which carry actual exception semantics
the lint test enforces) and the unchanged function/test bodies. No
behaviour change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the shared `assertInjectedAssetLoadsWithoutCspBlock(page, opts)`
helper in personas/utils/csp.js. It installs the CSP violation
observer, navigates the consumer URL with `waitUntil: networkidle`,
and asserts both:
1. the browser observed a successful 2xx/3xx response for the
requested resourceType(s) from one of the injector hosts, AND
2. no `securitypolicyviolation` event named one of those hosts as
its `blockedURI`.
Closes the loophole where the markup-only checks would pass even
when the consumer's CSP `style-src` / `script-src` did not whitelist
the injector host — the `<link>` / `<script>` was in the HTML but
the browser silently dropped the load.
Wires the helper into the three consumer-injection provider specs:
- web-svc-css: replaces the HTML-regex link check with the helper
(`stylesheet` resource type, css.* + cdn.* hosts).
- web-svc-logout: extends the existing `logout.js`-in-HTML check with
the helper (`script` resource type, cdn.* host). Adds CDN_BASE_URL
to the env template since logout.js is served from the CDN.
- web-app-matomo: extends the per-consumer tracker test with the
helper (`script` resource type, matomo canonical host) before the
existing `_paq` / `matomo.php` markup tokens, so the matomo.js
async fetch after DOMContentLoaded is verified end-to-end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- plugins/lookup/roles_with_service.py: resolve url.base via the tls
sublookup and return canonical_url alongside id/canonical_domain.
- web-app-matomo/files/playwright/playwright.spec.js: parameterised
matomo-tracker injection assertion per consumer over
MATOMO_TARGET_ROLES_JSON, using target.canonical_url.
- web-svc-logout: new files/playwright/playwright.spec.js + templates/
playwright.env.j2; checks universal-logout injection on every consumer
declared in LOGOUT_TARGET_ROLES_JSON. meta/services.yml gains nocheck
markers explaining why this provider has no user-facing surface to
assert on.
- tests/unit/plugins/lookup/test_roles_with_service.py: cover the new
canonical_url field via a stubbed tls sublookup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The `matomo: biber is denied access at the admin surface` test
expects matomo's `/` to redirect through Keycloak — that requires
oauth2-proxy in front of matomo. Today
`roles/web-app-matomo/meta/services.yml` carries `oauth2.enabled:
false` (TODO at line 12 says "Enable oauth2 for matomo"), so the
URL returns matomo's own UI directly and the redirect-toContain
poll times out.
Path 2 fix: gate the test on `isServiceEnabled("oauth2")` so it
reports `skipped` until matomo's deploy actually wires the proxy.
The test stays in the spec as the canonical contract for "biber
cannot reach Matomo"; once oauth2-proxy lands in matomo's
services.yml the gate flips and the test starts running.
The 019 matrix note for matomo cross-references the parking +
points at the services.yml TODO for the future iteration.
Promote four cheap rules in `eslint.config.js`:
* `no-unused-vars`: warn → error (with `argsIgnorePattern: "^_"`)
* `no-var`: error
* `eqeqeq`: error (always)
* `playwright/no-wait-for-navigation`: off → error
* `linterOptions.reportUnusedDisableDirectives`: off → warn
Add a repo-root CommonJS block so `eslint.config.js` itself sees Node
globals (`require`, `module`).
Fix every flagged offender:
* 138 stale persona-helper imports across 70 spec files
(`isServiceEnabled`, `skipUnlessServiceEnabled`, `runAdminFlow`,
... — leftovers from the SPOT consolidation).
* 4 unused catch bindings → modern `catch {}` syntax.
* 2 unused `dashboardBaseUrl` declarations (minio, moodle).
* 1 unused `label` arg → renamed to `_label` to opt out per the
argsIgnorePattern.
* 1 intentionally-kept legacy helper in web-app-wordpress's spec
receives a per-line `eslint-disable-next-line` with reason; the
preceding docstring already explained why it stays.
* 1 `page.waitForNavigation` in web-app-fediwall's Mastodon OIDC flow
receives a per-line disable with reason — the navigation target
depends on the runtime OIDC issuer; `waitForURL` would need a
runtime-built pattern.
* 10 stale `// eslint-disable-next-line no-console` directives now
surfaced by `reportUnusedDisableDirectives`; removed.
ESLint after this pass: 0 errors, 154 warnings (148 pre-existing
`playwright/expect-expect` plus 6 `playwright/no-force-option` that
were already inherited from `playwright/flat/recommended`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Strip every `req <NNN>`, `req-<NNN>`, `Requirement <NNN>`,
`requirement-<NNN>`, and `docs/requirements/<NNN>-...md` mention from
non-Markdown source files. Comments anchored to requirement numbers
rot when requirements are renumbered or rewritten and conflate
present-state behaviour (which the comment should describe) with the
audit trail of how the code got there (which belongs in commit
messages, PR descriptions, and the requirement file itself).
* Document the rule in `docs/contributing/documentation.md` under
"Comments". Code MUST describe the *why* in present-state language;
contract enforcement lives in tests, not folklore.
* Clean 254 source files across `roles/`, `tests/`, `plugins/`,
`cli/`, `scripts/`, `utils/`, `group_vars/`, and the repo-root
`eslint.config.js`. Pure comment / docstring / log-message edits;
no behavioural change.
* Add `tests/lint/repository/test_no_req_references_in_code.py` to
enforce the rule going forward across `.js`, `.py`, `.yml`,
`.yaml`, `.j2`, and `.sh` files. The legacy
`test_no_stale_requirement_refs.py` is exempt as the only file
whose purpose is to detect these very tokens.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Nine role specs (bigbluebutton, dashboard, discourse, keycloak,
matomo, matrix, moodle, openwebui, wordpress) each carried inline
copies of the same six CSP helpers and the `EXPECTED_CSP_DIRECTIVES`
constant. The bodies were semantically identical; only message
wording, brace style, and one captured-field-set differed.
Changes
-------
- Extends `roles/test-e2e-playwright/files/personas/utils/csp.js`
(already home to `assertCspInjections`) with
`installCspViolationObserver`, `readCspViolations`,
`parseCspHeader`, `assertCspResponseHeader`,
`assertCspMetaParity`, `expectNoCspViolations`, and the
`EXPECTED_CSP_DIRECTIVES` constant. The helpers are re-exported
through `personas/utils/index.js` so specs that
`require("./personas")` get them via the existing aggregator.
- Strips the inline definitions from the nine role specs and amends
each spec's `require("./personas")` destructuring with whichever
helpers the body still references.
Drift handling
--------------
- moodle's reduced violation-record shape (only `violatedDirective`
+ `blockedURI`) is replaced by the canonical 5-field record.
Moodle's downstream consumer asserts `.toEqual([])` on the array
only, so the extra fields are diagnostic-only — no behaviour
change.
- dashboard's 2-arg `expectNoCspViolations(page, label)` call site
is updated to the canonical 3-arg form
`expectNoCspViolations(page, null, label)`. The shared helper
guards `if (diagnostics && Array.isArray(diagnostics.cspRelated))`
before checking the diagnostics arm, so passing `null` keeps the
exact pre-migration behaviour for that call site.
- bigbluebutton and matrix had subtly different failure-message text
in `assertCspResponseHeader` and `expectNoCspViolations`. The
canonical wording is more descriptive; passing tests are
unaffected, failing tests get clearer diagnostics.
Doc
---
`docs/contributing/artefact/files/role/playwright.specs.js.md` is
extended in two places: the invariants section names the full CSP
test surface and pins the "import from `./personas`, no inline copy"
rule, and the per-service catalogue's CSP / injectors row lists the
helpers by name.
Verification
------------
- 419 passed / 1 skipped / 0 failed across `tests/lint/`,
`tests/integration/roles/`, `tests/unit/plugins/lookup/`.
- `make lint-javascript`: 0 errors, 283 warnings (unchanged from
baseline).
- `markdownlint-cli2`: 0 errors on the touched doc.
- 0 inline definitions of any extracted helper / constant remain in
any role spec.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both helpers were inlined verbatim in 71 of the role-local
`playwright.spec.js` files (52 with the compact form, 19 with the
verbose form, plus one parameter-name variant). All bodies were
semantically identical.
Changes
-------
- New `roles/test-e2e-playwright/files/personas/utils/dotenv.js` holds
the canonical pair (`decodeDotenvQuotedValue` and `normalizeBaseUrl`),
re-exported through `personas/utils/index.js` so specs that
`require("./personas")` get them via the existing aggregator.
- Every role spec drops its inline copies and pulls the helpers from
`./personas` instead. Specs that only reference one of the two
carry only that one in the destructuring (no unused-import noise).
Doc
---
`docs/contributing/artefact/files/role/playwright.specs.js.md` no
longer carries the "once a third spec needs them, MUST be promoted"
directive: the helpers live at the shared path and inline copies are
forbidden.
Verification
------------
- 419 passed / 1 skipped / 0 failed across `tests/lint/`,
`tests/integration/roles/`, `tests/unit/plugins/lookup/`.
- `make lint-javascript`: 0 errors, 283 warnings (unchanged from
baseline; no new unused-imports introduced).
- 0 inline `function decodeDotenvQuotedValue|normalizeBaseUrl`
definitions remain in any role spec.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cross-service surface assertions for dashboard, prometheus and matomo
no longer run in every consumer role's persona. They are owned by the
provider's own spec, parameterised once over a `<NAME>_TARGET_ROLES_JSON`
manifest rendered into the provider's `templates/playwright.env.j2` via
the new `lookup('roles_with_service', '<svc>')` Ansible lookup.
Changes
-------
- New `plugins/lookup/roles_with_service.py` enumerates roles that
declare `services.<svc>.{enabled, shared}` truthy in the merged
applications config, returning `[{id, canonical_domain}, …]`.
- `roles/web-app-{dashboard,prometheus,matomo}/files/playwright.spec.js`
parse `<NAME>_TARGET_ROLES_JSON` and parameterise the per-consumer
assertion (dashboard tile reachability, prometheus `up=1`, matomo
SitesManager registration). The matomo spec gains an explicit
biber-denial test (request body validated against admin DOM markers
per the body-marker rule); the prometheus admin-reach test drops
the dashboard-iframe path in favour of direct goto.
- Persona helpers `roles/test-e2e-playwright/files/personas/{admin,
biber,guest}.js` enter at `${APP_BASE_URL}/` directly. The
`clickRoleTileFromDashboard` step and the `assertPrometheus*` /
`assertMatomo*` cross-service probes are gone; `utils/{dashboard,
prometheus,matomo}.js` are deleted. `utils/csp.js` switches to
literal `process.env.<KEY>` reads so the env-keys-used parity guard
recognises matomo's CSP injector consumption.
- 42 consumer-role `templates/playwright.env.j2` templates drop
`DASHBOARD_BASE_URL`, `DASHBOARD_SERVICE_ENABLED`, `PROMETHEUS_BASE_URL`,
`PROMETHEUS_SERVICE_ENABLED`. `MATOMO_BASE_URL` and
`MATOMO_SERVICE_ENABLED` stay because the per-role CSP injection
helper still gates on matomo's tracker host.
- `tests/integration/roles/test_playwright_env_services_match.py`
globally exempts `dashboard` and `prometheus` from the
services.yml-to-env parity rule.
- `tests/unit/plugins/lookup/test_roles_with_service.py` adds 14
unit tests for the lookup, including an explicit-contract test
that pins the over-inclusive behaviour on raw Jinja strings (the
lookup expects a templar-rendered payload).
- Docs: `docs/requirements/019-playwright-meta-services-parity.md`
rewords Rules 1, 3, 12 and adds Rule 13 describing the SPOT shift.
`docs/contributing/artefact/files/role/playwright.specs.js.md`
reflects the new persona shape and per-service catalogue, and is
cleaned per `docs/contributing/documentation.md` (no SPOT self-
labels, no req-references, em dashes replaced with complete
sentences, multi-clause lines split onto separate lines, broken
anchor `#two-personas-fixed-flow-` corrected).
Verification
------------
- 418 passed / 1 skipped / 0 failed across `tests/lint/`,
`tests/integration/roles/`, `tests/unit/plugins/lookup/`.
- `make lint-javascript`: 0 errors.
- `markdownlint-cli2`: 0 errors on all touched docs.
- `make autoformat`: clean (FURB171 single-item-set inequality fixed).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduces roles/test-e2e-playwright/files/personas/ — runGuestFlow,
runBiberFlow, runAdminFlow plus 10 utils (csp, dashboard, env, keycloak,
landing, logout, matomo, prometheus, interaction, index) — and wires
both the Ansible runner (run_one.yml) and the local rerun helper
(scripts/tests/e2e/rerun-spec.sh) to stage the tree alongside
service-gating.js so specs can `require("./personas")`.
First batch of role specs migrated to consume the shared library
(dashboard, keycloak, mailu, matomo, prometheus, web-svc-cdn,
web-svc-simpleicons) — each gets the spec.js persona scenarios plus
the matching meta/services.yml + playwright.env.j2 entries; matomo is
marked role-closed in req 019. dashboard/keycloak/mailu TODO.md
updated; docs/requirements/019 reflects matomo closure.
Reason: req 019 Rule 3 mandates that every role's guest/biber/
administrator flow runs the same canonical steps; centralising them
in one shared library keeps per-role drift from re-introducing the
stub-shaped bodies the new lint forbids.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes Verification checkboxes in requirement 003. Each spec exercises
the `lookup('applications')` + `lookup('users')` pathway end-to-end:
login via lookup('users'), OIDC round-trip from dashboard (where the
app supports it), canonical-domain DOM assertion, and a full CSP
assertion covering every directive emitted by csp_filters.build_csp_header
(15 directives), meta-parity, enforced policy (not Report-Only), and
zero securitypolicyviolation events.
The dashboard spec is extended in place with the CSP + canonical-domain
assertions while keeping its existing login/logout coverage.
rerun-spec.sh is pointed at the Jinja template + the staged copy of
package.json (the repo has templates/package.json.j2, not files/package.json),
so the inner spec-only iteration loop works without re-deploying.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
