Commit Graph
28 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 5.5 fe486a21e5 fix(web-app-peertube): back-fill the application defaults only when missing
With the psql command tags gone the OIDC plugin task stays `ok`, but PeerTube
was still force-recreated on every deploy: "Back-fill column defaults so
bootstrap INSERT keeps NOT NULL" runs an unconditional `ALTER TABLE …
SET DEFAULT`, database_query reports every non-SELECT as changed, and the
task notifies compose-up, which runs `compose up -d --force-recreate`.

A SELECT on information_schema.columns now lists the bootstrap columns
(`nodeVersion`, `nodeABIVersion`) that still have no default, and the ALTER
with its compose-up notification runs only when that list is not empty. A
fresh database still gets the defaults before PeerTube's bootstrap INSERT
retries; every later deploy skips the task. The unused register of the
ALTER task is gone, and both read-only checks now live under
files/sql/check-application/ as the sibling-prefix rule requires.

Refs #1164

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-22 20:36:32 +02:00
kevinveenbirkenbachandClaude Opus 5 575a5a87ac refactor(layout): sort shipped scripts by language and split the unit suite
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).

This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.

Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:

  * 7 `src:` values built from a role variable holding a bare filename
    (svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
    The variable doubles as the destination name inside the container, so the
    language segment belongs at the call site, not in the variable.
  * `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
    through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
    18 roles ship exactly that and have no template fallback: the play would
    have died on each.
  * `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
    interface now reads `files/javascript/javascript.js`.
  * `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
    path through `path_absolute(... 'files', ...)`.
  * 4 `ansible.builtin.script:` invocations naming a bare command
    (pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
  * `dev-python/files/shell/install.sh` pointed at a sibling role's script.

Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 01:31:09 +02:00
kevinveenbirkenbach e04c448cdd Merge branch 'feature/swarm-nfs' into feature/svc-net-tor
Three conflicts, none resolved by picking a side wholesale:

- Makefile: took the incoming ACT_INPUTS, which routes through SWARM_DISTROS
  instead of inlining the same expression a second time.
- test-e2e-playwright/02_run_one.yml: took the incoming rc handling. An absent
  rc now counts as failure; the previous default(0) let an aborted Playwright
  run pass silently, contradicting the debug message right above it.
- web-app-peertube spec: combined both. Kept gotoOnion and resolveTimeout,
  which this branch's lint enforces, and adopted the incoming retry-inside-poll
  for the OIDC button plus its networkidle wait.
2026-07-30 15:38:15 +02:00
kevinveenbirkenbachandClaude Opus 5 cfd6f08795 fix(swarm): recreate tasks so in-container changes reach the process
docker stack deploy leaves tasks running when the service spec is unchanged, so a change made inside a container never reaches the process. PeerTube's CLI installs plugins with register:false, so the OIDC plugin sat installed and enabled in the database while the running server had loaded nothing: /login rendered only the local form and both OIDC specs timed out through all three attempts. Compose never showed it because its handler force-recreates.

Add a container recreate notify. In swarm it forces a task recreation per stack service and waits for convergence; in compose it maps onto the existing --force-recreate, so a role states the intent once and both modes honour it.

In the peertube role the install probe now tests node_modules/<npm-name>, where pnpm puts the plugin code; the old path was the data directory that registerPlugin creates at boot, so it never existed and the plugin was reinstalled on every run. The settings upsert derives changed from the rowcount via RETURNING, because database_query reports changed from the kind of statement rather than from what it touched. A registration probe notifies when the running process has not loaded the plugin, so an already-broken instance heals itself instead of walking past both earlier probes. The plugin:enable task is gone: npm has no such script, the call always failed and failed_when swallowed it.

The spec now polls the button lookup instead of counting once twenty milliseconds after goto, while still asserting on the URL so a failure keeps printing the address the browser actually sat on.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 15:08:50 +02:00
kevinveenbirkenbachandClaude Fable 5 d88f272f9f Merge branch 'feature/swarm-nfs' into feature/svc-net-tor
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 15:31:08 +02:00
kevinveenbirkenbachandClaude Fable 5 c505ce7466 test(lint): forbid unmarked '|| true' in shell scripts; grandfather the corpus
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 20:39:16 +02:00
kevinveenbirkenbachandClaude Opus 4.8 6f9142027d fix(e2e): skip seaweedfs filer check on onion targets
When an app rides onion, Playwright routes through the node Tor SOCKS
proxy, which cannot resolve the clearnet-internal SeaweedFS filer UI
domain (net::ERR_SOCKS_CONNECTION_FAILED). Skip the filer storage check
on onion targets in all consumer specs — same pattern already proven in
the nextcloud spec. Clearnet runs are unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-14 19:34:59 +02:00
kevinveenbirkenbachandClaude Opus 4.8 07c6ad8e7c test(e2e): scale every nav click + timeout literal for onion targets
Over Tor the redirect chain behind a login/logout/OIDC click routinely exceeds Playwright's fixed 30s action timeout, so the click throws before the flow's scaled waits run. Sweep all role Playwright suites and the shared persona helpers: give every navigation/auth-transition click an onion-scaled timeout (.click({ timeout: resolveTimeout(30_000) })) and route the remaining raw millisecond literals (timeout options, waitForTimeout, sleep-promise setTimeout, positional wait args) through resolveTimeout. Local UI clicks (menus, toggles, modal dismissals, DOM clicks inside evaluate) stay untouched.

Two new lints keep it that way: test_no_raw_timeout_literals forbids raw numeric timeout literals in Playwright specs (opt-out: nocheck: raw-timeout), test_persona_clicks_scaled requires every click in the shared persona auth flows to carry a resolveTimeout timeout (opt-out: nocheck: unscaled-click).

Verified: node --check on all 133 changed JS files, all 13 playwright lints green, make autoformat clean, make test green (2745+196+163+4 tests OK).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 11:35:53 +02:00
kevinveenbirkenbachandClaude Opus 4.8 094c02a69d feat(diagnostics): single SPOT rescue script with CI-uploaded snapshots
Every Ansible rescue: block used to carry ad-hoc per-role failure dumps that
drifted apart and captured different subsets. Replace them all with one
diagnostics script (scripts/system/diagnostics/rescue.sh) that snapshots
containers (full logs plus inspect per container), swarm services (ps plus
logs per service), journal, host resources, runtime info and stats into a
per-run folder and always exits 1 so the deploy still fails. A new lint
enforces the shape: every rescue ends with the script, at most one
nocheck-justified extra task before it (postgres pg_hba dump, xwiki
extension-repo probe), and a nocheck rescue-recovery exemption for the one
block that must genuinely recover: keycloak object updates, where a
cross-node reschedule needs a swarm-wide node re-resolve plus a delegate_to
change that only a rescue can express; its recovery is kept and now also
captures diagnostics before the terminal failure.

The snapshot folder path is a group_vars SPOT: DIR_RESCUE_DIAGNOSTICS and
DIR_VAR_LIB live in group_vars/all/05_paths.yml as plain strings; new env
handlers derive INFINITO_RESCUE_DIAGNOSTICS_DIR and INFINITO_DIR_VAR_LIB
from it for the generated .env, replacing the default.env entries, and the
play-level environment forwards the value to every task host so rescue.sh
needs no second default. CI uploads the folder like the inventory and
playwright artifacts: compose copies it out of the deploy container in
dedicated.sh, swarm captures per node (manager and both workers) into
node-named subfolders.

Deleting the ad-hoc dumps orphaned helper files (postgres inspect_container,
terminator docker_diagnostics, peertube diagnostics, matomo bootstrap debug,
xwiki and openresty rescue includes) and 14 role vars; all removed. Generic
pieces of those dumps (runtime version and info, stats, dmesg oom evidence,
swapon) moved into the SPOT script instead.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-11 03:04:06 +02:00
kevinveenbirkenbachandClaude Opus 4.8 4c91df5de7 test(e2e): forbid raw page.goto, migrate all specs to Tor-resilient gotoOnion
New lint test_no_raw_goto.py forbids raw `page.goto(...)` in role playwright specs + persona flows (the gotoOnion wrapper in personas/utils is exempt) so navigation retries transient Tor-transport errors over onion. Migrates every raw call across ~75 roles (343 sites) to `gotoOnion(page, ...)` and teaches gotoOnion to detect relative onion URLs (`isOnionCanonical`). gotoOnion is a behaviour-identical drop-in on clearnet (single attempt, same return). Verified: no-raw-goto lint green, eslint 0 errors, make test green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-08 20:59:38 +02:00
kevinveenbirkenbachandClaude Opus 4.8 35dee215c8 test(playwright): route all timeouts through resolveTimeout for onion-compat
Add the global TIMEOUT_FACTOR and a shared timeouts.js helper: resolveTimeout(base) scales by TIMEOUT_FACTOR and, over a .onion target, an extra multiplier, so Tor's per-request latency no longer trips hard-coded timeouts. Migrate every raw timeout in the specs (timeout:/timeout=/setTimeout/waitForTimeout) to resolveTimeout, stage the helper in all three staging paths, and add test_no_raw_timeouts to forbid raw numeric timeouts. Clearnet behaviour is unchanged (factor 1: resolveTimeout(N) === N).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-08 10:38:06 +02:00
kevinveenbirkenbachandClaude Opus 4.8 fb1ad790a5 fix(web-app-peertube): dismiss the first-login modal and harden the publish-button wait (toBeEnabled) so the SeaweedFS upload test no longer hangs (DiD-verified green)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-15 01:14:29 +02:00
kevinveenbirkenbachandClaude Opus 4.8 70ad33bc13 fix(web-app-peertube): repair the SeaweedFS object-store e2e test for PeerTube v8.2.1
The test timed out: the publish-button locator no longer matched the v8.2.1
manage-video wizard, and the inline 78-byte MP4 stub was structurally invalid,
so PeerTube rejected the upload as "Video file unreadable" and left the Save
button disabled.

- test-seaweedfs.js: target the v8.2.1 save control (.save-button > button),
  upload PeerTube's real video_short.mp4 fixture, and assert a version-tolerant
  saved marker.
- add fixture roles/web-app-peertube/files/playwright/fixtures/video_short.mp4.
- test-e2e-playwright: stage role-local files/playwright/fixtures/ next to the
  tests (the *.js glob ships neither binaries nor subdirectories), so specs load
  a fixture via path.join(__dirname, "fixtures", <name>).
- docs: document the fixtures/ folder and the per-scenario test-*.js split in the
  playwright spec contract and the runner README.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-13 14:07:03 +02:00
kevinveenbirkenbachandClaude Opus 4.8 20bdd12219 test(web-app): seaweedfs object-store consumer e2e tests for 7 apps + gitea/opentalk exemptions
Add files/playwright/test-seaweedfs.js (+ SEAWEEDFS_* playwright.env vars, spec require wiring, and removal of the playwright-service-flag nocheck above the seaweedfs service) for baserow, bookwyrm, mattermost, openwebui, peertube, penpot, pixelfed. Each test reuses the role's real login flow and triggers an upload that writes to the app's S3 bucket (file-field, avatar, chat attachment, composer photo, video), verified via the shared runSeaweedfsStorageCheck (bucket grew through the Filer UI).

openwebui: also export dismissAllOpenModals from _shared.js (used by the new test) and match the uploaded file by basename (not the full filename) to avoid the extension-split false-negative.

Exempt gitea and opentalk via meta/services.yml nocheck markers (seaweedfs-playwright + playwright-service-flag): gitea has no automatable UI login in this stack (oauth2-proxy + OIDC account-link / native login both fail; personas blocked); opentalk only writes S3 for recordings/whiteboard exports, which need the recorder service + a live multi-party meeting.

Builds on 9b6c6415f (shared runSeaweedfsStorageCheck helper + consumer lint).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 20:17:35 +02:00
kevinveenbirkenbachandClaude Opus 4.7 24d4ab8e4a refactor(sso)\!: collapse oauth2 + oidc into unified services.sso block (req 21)
Implements docs/requirements/021-sso-flavor-migration.md atomically:

* Schema: every role's meta/services.yml replaces the parallel
  `oauth2:` / `oidc:` top-level blocks with a single `sso:` block
  carrying `enabled`, `shared`, and a `flavor` discriminator
  (`oidc` | `oauth2` | `saml`; default `oidc`). Flavor-specific
  sub-keys live under `sso.<flavor>.*` (`sso.oauth2.{origin,acl,
  allowed_groups}`, `sso.oidc.plugin`).
* Provider role: roles/web-app-oauth2-proxy/ is deleted in its
  entirety; its 5 sidecar templates move under
  roles/web-app-keycloak/templates/sso_proxy/, and the per-consumer
  rendering task lives at roles/web-app-keycloak/tasks/sso_proxy.yml.
  web-app-keycloak now declares `provides: sso`.
* Runtime: new resolver utils/roles/applications/services/sso.py
  centralises the SSO predicates (`is_enabled`, `is_proxy_gated`,
  `is_oidc_native`) and the oauth2 sub-values
  (`oauth2_origin_{host,port}`, `oauth2_acl`, `oauth2_allowed_groups`).
  Three lookup plugins consume the resolver: `sso` (single-app
  predicate access), `sso_proxy_consumers` (enumerates oauth2-flavored
  consumers), `sso_oidc_plugin` (renamed from `oidc_flavor`, Nextcloud
  plugin selector). sys-stk-backend, sys-svc-compose, sys-svc-proxy
  and plugins/filter/compose_volumes.py all gate on
  `lookup('sso', ..., 'is_proxy_gated')` instead of manually combining
  `enabled` and `flavor`.
* Identifier renames repo-wide: `services.<entity>.ports.local.oauth2`
  → `ports.local.sso`; `PORT_BANDS.local.oauth2` → `PORT_BANDS.local.sso`;
  `OAUTH2_PROXY_*` env/Jinja vars → `SSO_PROXY_*`;
  `oauth2_proxy_cookie_secret` → `sso_proxy_cookie_secret`;
  `OAUTH2_SERVICE_ENABLED` + `OIDC_SERVICE_ENABLED` collapse into
  `SSO_SERVICE_ENABLED`; Playwright gating helpers
  (isServiceEnabled/skipUnlessServiceEnabled/requireService/safeIsEnabled/
  safeSkipUnlessEnabled) move from `oidc`/`oauth2` to `sso`.
* Dual-block resolution: per req's Background-Table, web-app-bookwyrm
  is `flavor: oauth2`; web-app-gitea and web-app-friendica keep
  `flavor: oauth2` with their existing `oauth2.acl.blacklist`
  (/user/login) and `oauth2.acl.whitelist` (federation + discovery
  endpoints) — preserving the operative oauth2-proxy gating.
* Tests: the two mutual-exclusion guards
  (tests/integration/iam/oauth2_oidc/test_mutual_exclusive.py and
  test_acl_mutual_exclusive.py) are deleted per Decision #10;
  test_oauth2_contract.py is renamed and rewritten to
  test_sso_contract.py, asserting that `flavor: oauth2` roles carry
  `oauth2.origin.{host,port}` plus `ports.local.sso`; test_sso.py
  (web-app integration) now requires the unified `sso:` block with an
  explicit flavor or a `# nocheck: sso` opt-out; new
  tests/lint/repository/no_legacy_sso_paths/ guards the source tree
  against re-introducing the legacy strings.
* Docs: req 21 itself has every AC checked off, and reqs 006, 008,
  009, 013, 015, 016, 019, 020 plus 10+ role READMEs and
  docs/contributing/actions/testing/suppression.md are updated to the
  unified SSO naming.

Empirically validated end-to-end against the validation app set
(web-app-keycloak, web-app-nextcloud, web-app-bookwyrm, web-app-gitea,
web-app-friendica, web-app-dashboard) in matrix variants 0 and 1; v2
also passes the 4 target roles' Playwright after a documented
pre-existing KC_HOSTNAME-drift workaround (separate concern, not
caused by this refactor).

BREAKING CHANGE: every consumer of `services.oauth2.*`,
`services.oidc.*`, `OAUTH2_PROXY_*`, `OAUTH2_SERVICE_ENABLED`,
`OIDC_SERVICE_ENABLED`, `oauth2_proxy_cookie_secret`,
`oauth2_proxy_application_id`, `ports.local.oauth2`, the
`web-app-oauth2-proxy` role, or the legacy gating helper service names
(`oidc`/`oauth2`) must migrate to the unified `sso` shape documented
in docs/requirements/021-sso-flavor-migration.md.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-23 12:59:59 +02:00
kevinveenbirkenbachandClaude Opus 4.7 1f9559a335 fix(playwright): gate bespoke SSO/LDAP tests on service flags (odoo, fider, mattermost, peertube, kix)
Bespoke `test(...)` blocks that drive a real Keycloak SSO round-trip
(odoo, fider, mattermost, peertube) had no `test.skip(\!oidcEnabled, ...)`
guard. In matrix variants that render `OIDC_SERVICE_ENABLED=false`, the
Playwright locator for the SSO button never resolves; each test waits
300s, retries 2x, and 2-3 such tests in series blow past the 1200s
Ansible task-wrapper timeout — Bundle α odoo variant 1 reproduced this.

Pattern parity with `roles/web-app-moodle/files/playwright/playwright.spec.js`:
import `service-gating`, derive `<flag>Enabled` once, prefix each
affected test with `test.skip(\!<flag>Enabled, "...")`.

- odoo (verified empirically against running variant-1 stack via
  `make run`: 4 skipped, 1 passed, 2.2s — was 1200s timeout)
- fider, mattermost, peertube: same pattern, 2 bespoke tests each
- kix: gates on `oauth2Enabled` + `ldapEnabled` instead — the role's
  variant matrix has no `oidc` flag, the flow is
  KIX → OAuth2-proxy → Keycloak → KIX-LDAP, both can be independently
  disabled per variant.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 15:30:06 +02:00
kevinveenbirkenbachandClaude Opus 4.7 c2c532c8e6 refactor(playwright): centralize dashboard-tile mechanics in web-app-dashboard SPOT; move specs into files/playwright/ subdir
Architecture
============
- New SPOT helper `roles/web-app-dashboard/files/playwright/dashboard-card-flow.js` (`runDashboardCardScenario(page, context, target)`): per `roles_with_service('dashboard')` target, asserts tile visible → click loads in `#main iframe` (outer URL gets `?iframe=` + iframe src matches canonical_domain) → header "Tab" button pops the embed into a fresh browser tab.
- `web-app-dashboard/files/playwright/playwright.spec.js` parametrised loop calls the helper for every `DASHBOARD_TARGET_ROLES_JSON` entry — the single owner of dashboard-tile mechanics across the whole codebase.

Spec relocation
===============
- 71 `roles/<role>/files/playwright.spec.js` → `roles/<role>/files/playwright/playwright.spec.js` (`git mv`, history preserved). odoo + pixelfed show as D+A in `git status` because of substantial content rewrites; coverage retained.
- New SPOT constant `ROLE_FILE_PLAYWRIGHT_SPEC` in `utils/roles/mapping.py`, registered as optional on `ROLE_TYPE_APPLICATION` (disallowed elsewhere, mirrors `ROLE_FILE_META_VARIANTS` pattern).
- `roles/test-e2e-playwright/tasks/run_one.yml`: staging task replaced with `with_fileglob` over `files/playwright/*.js` — roles can ship companion helpers (like `dashboard-card-flow.js`) next to the spec without further wiring.

Spec migrations (no more dashboard-iframe path in app specs)
============================================================
13 specs rewritten so the persona/SSO journey starts at the app URL directly, never through the dashboard tile or iframe:
* fider, mailu, mattermost, nextcloud, odoo, taiga: iframe-wrap removed (no more `frameLocator("#main iframe")`)
* bigbluebutton, discourse, kix, matrix, openwebui, peertube: dashboard goto preamble removed
* pixelfed: ~270 LOC of iframe-aware helpers/login/logout collapsed to direct-URL flow

Title cleanup
=============
- 64 specs: `"<persona>: dashboard → app → universal logout"` / `"administrator: dashboard → prometheus → app → universal logout"` → `"<persona>: app → universal logout"`.
- 13 specs: `"<persona>: dashboard to <app> ..."` → `"<persona>: <app> ..."`.

Dead-code cleanup
=================
- `dashboardBaseUrl` consts + beforeEach assertions removed from 11 specs (no functional uses; personas helper handles env validation).
- `DASHBOARD_BASE_URL=` env-template lines removed from fider, mattermost, odoo, taiga, pixelfed, kix.
- `DASHBOARD_SERVICE_ENABLED=` env-template lines removed (pixelfed retained for the still-gated spec).
- Stale "loads inside the iframe" / "FrameLocator parameter" comments cleaned in fider, odoo, prometheus, nextcloud.

Tooling
=======
- 5 lint Python files + 1 integration test now use `from utils.roles.mapping import ROLE_FILE_PLAYWRIGHT_SPEC`.
- `eslint.config.js` glob pattern + `roles/test-e2e-playwright/meta/main.yml` description + `roles/test-e2e-playwright/README.md` updated to new path.
- 25 markdown/docstring references in `docs/`, `roles/**/README.md`, `roles/**/TODO.md`, persona-helper docstrings updated to new path.
- `docs/contributing/artefact/files/role/playwright.specs.js.md`: file-placement section now documents the `files/playwright/` convention + `ROLE_FILE_PLAYWRIGHT_SPEC` SPOT constant; dashboard SPOT delegation explicit at 3 places.

Net diff: 124 files, +1142/-1564 (−422 LOC).
Integration suite: 183/183 green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-16 23:24:55 +02:00
kevinveenbirkenbachandClaude Opus 4.7 017f67cd53 feat(plugins): introduce database_query module + migrate all SQL query tasks
Replaces controller-side `community.postgresql.postgresql_query` /
`community.mysql.mysql_query` calls with a project-local `database_query`
Ansible module that wraps `container exec -i <container> <psql|mysql>`
and pipes SQL via stdin. The module routes via the DB container's own
network so the same task works identically in shared mode (central
`svc-db-*` container) and non-shared mode (per-role `<entity>-database`
container) — the `container` field of `lookup('database', …)` is
shared-aware and the module just reads it. SQL payloads come from a
file (`query_file:`, preferred) or as a raw string (`query:`, escape
hatch). `named_args:` substitutes `%(name)s` placeholders with engine-
appropriate escaping (postgres single-quote-doubling, mariadb backslash
escapes), so role tasks stop string-interpolating Jinja into SQL.

New backends extend the module's `_ENGINES` dispatch table; everything
else is engine-agnostic. The module ships at `plugins/modules/` (Ansible
collection-style layout) — the shebang lint test learned to recognise
that path as a valid module location alongside the existing per-role
`library/` layout.

Two lint tests enforce the convention:

* `tests/lint/ansible/modules/test_no_raw_db_query_module_usage.py`
  forbids `community.postgresql.postgresql_query` /
  `community.mysql.mysql_query` in role tasks. Opt out per-line with
  `# nocheck: database-query` (used by `svc-db-postgres/tasks/02_init.yml`
  where the role manages its own DB and the `database` lookup would be
  self-referential).
* `tests/lint/ansible/modules/test_no_raw_database_query_string.py`
  forbids inline `query:` on `database_query:` tasks. Opt out with
  `# nocheck: database-query-raw` (no opt-outs in the current tree —
  every SQL payload reaches the module through a `.sql` file under
  `files/sql/`).

Role migrations:

* `web-app-peertube` — 3 tasks migrated, 3 new `.sql` files,
  `templates/upsert-oidc-plugin.sql.j2` deleted in favour of
  `files/sql/upsert-oidc-plugin.sql` + `named_args:`.
* `web-app-openproject` — 3 tasks migrated, 3 new `.sql` files for
  the LDAP-auth-source check/update/insert chain.
* `web-app-odoo` — 33 tasks migrated, 9 new `.sql` files for the
  LDAP / admin / init-database flows; 17 pre-existing OIDC `.sql`
  files renamed snake_case → kebab-case and relocated from
  `files/oidc/` to `files/sql/oidc/` (via `git mv` so rename history
  stays intact).

Unit tests cover every dispatch-table entry, the named_args escaper
(strings, ints, floats, bools, None, single-quote and backslash
injection vectors), and the password-redactor that hides
PGPASSWORD/MYSQL_PWD assignments from the module's returned `cmd`
field.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 20:12:08 +02:00
kevinveenbirkenbachandClaude Opus 4.7 0e651e4a22 chore(lint): tighten ESLint rules and clean up flagged offenders
Promote four cheap rules in `eslint.config.js`:
* `no-unused-vars`: warn → error (with `argsIgnorePattern: "^_"`)
* `no-var`: error
* `eqeqeq`: error (always)
* `playwright/no-wait-for-navigation`: off → error
* `linterOptions.reportUnusedDisableDirectives`: off → warn

Add a repo-root CommonJS block so `eslint.config.js` itself sees Node
globals (`require`, `module`).

Fix every flagged offender:
* 138 stale persona-helper imports across 70 spec files
  (`isServiceEnabled`, `skipUnlessServiceEnabled`, `runAdminFlow`,
  ... — leftovers from the SPOT consolidation).
* 4 unused catch bindings → modern `catch {}` syntax.
* 2 unused `dashboardBaseUrl` declarations (minio, moodle).
* 1 unused `label` arg → renamed to `_label` to opt out per the
  argsIgnorePattern.
* 1 intentionally-kept legacy helper in web-app-wordpress's spec
  receives a per-line `eslint-disable-next-line` with reason; the
  preceding docstring already explained why it stays.
* 1 `page.waitForNavigation` in web-app-fediwall's Mastodon OIDC flow
  receives a per-line disable with reason — the navigation target
  depends on the runtime OIDC issuer; `waitForURL` would need a
  runtime-built pattern.
* 10 stale `// eslint-disable-next-line no-console` directives now
  surfaced by `reportUnusedDisableDirectives`; removed.

ESLint after this pass: 0 errors, 154 warnings (148 pre-existing
`playwright/expect-expect` plus 6 `playwright/no-force-option` that
were already inherited from `playwright/flat/recommended`).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 16:25:23 +02:00
kevinveenbirkenbachandClaude Opus 4.7 c55ad69c41 refactor: drop requirement-file references from code, add doc rule + lint
Strip every `req <NNN>`, `req-<NNN>`, `Requirement <NNN>`,
`requirement-<NNN>`, and `docs/requirements/<NNN>-...md` mention from
non-Markdown source files. Comments anchored to requirement numbers
rot when requirements are renumbered or rewritten and conflate
present-state behaviour (which the comment should describe) with the
audit trail of how the code got there (which belongs in commit
messages, PR descriptions, and the requirement file itself).

* Document the rule in `docs/contributing/documentation.md` under
  "Comments". Code MUST describe the *why* in present-state language;
  contract enforcement lives in tests, not folklore.
* Clean 254 source files across `roles/`, `tests/`, `plugins/`,
  `cli/`, `scripts/`, `utils/`, `group_vars/`, and the repo-root
  `eslint.config.js`. Pure comment / docstring / log-message edits;
  no behavioural change.
* Add `tests/lint/repository/test_no_req_references_in_code.py` to
  enforce the rule going forward across `.js`, `.py`, `.yml`,
  `.yaml`, `.j2`, and `.sh` files. The legacy
  `test_no_stale_requirement_refs.py` is exempt as the only file
  whose purpose is to detect these very tokens.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 16:25:22 +02:00
kevinveenbirkenbachandClaude Opus 4.7 6a92143d21 fix(playwright): add missing persona scenarios for Rule 3 compliance
Seven web-app-* specs failed the Rule 3 contract from
docs/contributing/artefact/files/role/playwright.specs.js.md:

  - bigbluebutton, decidim, discourse, matrix, openwebui, peertube
    each shipped `administrator:` and `biber:` test titles but no
    `guest:` test.
  - keycloak shipped four detailed admin / biber tests under non-
    canonical names (`master-realm super administrator…`, `normal-
    realm administrator logs in through account interface…`, etc.)
    and no `<persona>: <flow>`-prefixed scenarios at all.

None of the seven declared a `PERSONA_<persona>_BLOCKED=true`
opt-out in their `templates/playwright.env.j2`, and none qualify
for the auth-less collapse exception (every role has an OIDC auth
surface; keycloak IS the auth surface).

Changes
-------
- Each of the six guest-only-missing specs gains a single appended
  scenario:
    test("guest: public-landing → auth chain → never authenticated", …)
      → runGuestFlow(page);
- Keycloak gains all three persona scenarios (guest, biber,
  administrator) using the shared `runGuestFlow` / `runBiberFlow` /
  `runAdminFlow` runners. Existing detailed tests stay untouched
  alongside the persona contract.
- The `./personas` import in each spec is amended with whichever
  runners are now needed.

Verification
------------
- 420 passed / 1 skipped / 0 failed across `tests/lint/`,
  `tests/integration/roles/`, `tests/unit/plugins/lookup/`.
- `make lint-javascript`: 0 errors.
- 0 Rule-3 violators remain across all `web-app-*/files/playwright.spec.js`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 16:25:22 +02:00
kevinveenbirkenbachandClaude Opus 4.7 370f381d1c refactor(playwright): share Keycloak login-form helper
Fourteen role specs (bigbluebutton, discourse, fider, gitea, mailu,
matrix, mattermost, odoo, openwebui, peertube, postmarks, prometheus,
yourls, libretranslate) each carried an inline `performOidcLogin`
that drove the Keycloak login form. The bodies fell into four
clusters split between the role-based selector strategy
(`getByRole`) and the legacy input-name strategy
(`input[name='username']`); the existing shared
`performKeycloakLogin` already covered both via `.or()` chains but
also asserted post-login navigation, so it could not be reused
verbatim.

Changes
-------
- `personas/utils/keycloak.js` gains `performKeycloakLoginForm
  (target, username, password)` — form-fill + submit only, accepts
  `Page` OR `FrameLocator` so iframe-embedded Keycloak pages work
  without branching at the call site. The selectors are the union
  of all four prior cluster shapes (role-based getter chained with
  `.or(input-name fallback)`, sign-in matched against
  `/sign in|login|log in/i` plus `input#kc-login` family). The
  existing `performKeycloakLogin` and
  `performKeycloakLoginExpectingDenial` now delegate to the new
  helper, so their public API is unchanged.
- The 14 specs strip their inline `performOidcLogin` and amend
  `require("./personas")` with `performKeycloakLoginForm`. Every
  call site and comment reference is renamed in lockstep.

Drift handling
--------------
- Cluster 1 (5 specs) expected the accessible name `"Password"`
  exactly; the canonical regex `/^password$/i` is case-insensitive
  and matches every prior name plus lowercase variants — strictly
  more permissive.
- Cluster 1's sign-in button matched `/sign in/i`; the canonical
  matches `/sign in|login|log in/i` plus the legacy `input#kc-login`
  family — strictly broader.
- Cluster 1 awaited `usernameField.press("Tab")` without a `.catch`;
  the canonical wraps the press in `.catch(() => {})` so an
  unexpected Tab failure no longer aborts the flow.
- Cluster 2 / 3 used `expect(usernameField).toBeVisible({ timeout })`
  for the visibility wait; the canonical uses
  `usernameField.waitFor({ state: "visible", timeout })`, the
  Locator-method counterpart of the same wait — functionally
  equivalent.

Verification
------------
- 419 passed / 1 skipped / 0 failed across `tests/lint/`,
  `tests/integration/roles/`, `tests/unit/plugins/lookup/`.
- `make lint-javascript`: 0 errors, 283 warnings (unchanged from
  baseline).
- 0 inline `performOidcLogin` definitions remain in any role spec,
  and 0 stale references to the legacy name in any context (call
  sites and comments are renamed in the same pass).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 14:55:05 +02:00
kevinveenbirkenbachandClaude Opus 4.7 122c82910b refactor(playwright): share decodeDotenvQuotedValue + normalizeBaseUrl
Both helpers were inlined verbatim in 71 of the role-local
`playwright.spec.js` files (52 with the compact form, 19 with the
verbose form, plus one parameter-name variant). All bodies were
semantically identical.

Changes
-------
- New `roles/test-e2e-playwright/files/personas/utils/dotenv.js` holds
  the canonical pair (`decodeDotenvQuotedValue` and `normalizeBaseUrl`),
  re-exported through `personas/utils/index.js` so specs that
  `require("./personas")` get them via the existing aggregator.
- Every role spec drops its inline copies and pulls the helpers from
  `./personas` instead. Specs that only reference one of the two
  carry only that one in the destructuring (no unused-import noise).

Doc
---
`docs/contributing/artefact/files/role/playwright.specs.js.md` no
longer carries the "once a third spec needs them, MUST be promoted"
directive: the helpers live at the shared path and inline copies are
forbidden.

Verification
------------
- 419 passed / 1 skipped / 0 failed across `tests/lint/`,
  `tests/integration/roles/`, `tests/unit/plugins/lookup/`.
- `make lint-javascript`: 0 errors, 283 warnings (unchanged from
  baseline; no new unused-imports introduced).
- 0 inline `function decodeDotenvQuotedValue|normalizeBaseUrl`
  definitions remain in any role spec.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 14:55:05 +02:00
kevinveenbirkenbachandClaude Opus 4.7 44d51744a7 feat(playwright): migrate second batch of roles to shared persona library
Wires runGuestFlow / runBiberFlow / runAdminFlow from
roles/test-e2e-playwright/files/personas/ into another ~28 roles
(akaunting, baserow, bluesky, bookwyrm, bridgy-fed, decidim, discourse,
fider, friendica, fusiondirectory, gitea, hugo, jenkins, joomla, kix,
matrix, mattermost, mig, moodle, odoo, opencloud, peertube, pixelfed,
postmarks, sphinx, taiga, yourls, web-svc-libretranslate, web-svc-xmpp).
Each role gets the spec.js persona scenarios plus matching
meta/services.yml gate entries and playwright.env.j2 env wiring;
admin-flow callers pass a role-specific adminInteraction that drives
an actual management surface so the body is not a stub.

Reason: continuation of d3e5ebe44 — req 019 Rule 3 demands every
role exercise the canonical guest/biber/administrator flows; this
batch closes the long-tail role list against the central library so
per-role drift cannot reintroduce the stub-shaped bodies the new lint
forbids.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 16:09:46 +02:00
kevinveenbirkenbachandClaude Opus 4.7 d3afe0bc81 test(lint): forbid a file from referencing its own path as a string
Hard-coded self-path references rot the moment the file moves: a header
comment ``# utils/foo/bar.py`` at the top of a module, a docstring
mentioning ``utils/foo/bar.py``, or a docs entry linking
``utils/foo/bar.py`` from inside ``utils/foo/bar.py`` itself all silently
desync from reality after the next ``git mv``.

The new lint walks every project file, computes its POSIX-style relative
path from PROJECT_ROOT, and reports any line whose text contains that
path as a substring. Files at the repo root (no ``/`` in their relative
path) and setuptools-generated ``*.egg-info/`` artefacts are exempt.
Per-line opt-out via ``# nocheck: self-path-reference`` on the offending
line, registered in the suppression catalog.

Existing offenders cleaned up:
- 72 redundant ``# <self-path>`` header lines removed across roles,
  cli, plugins, scripts, tests, and utils.
- 4 prometheus templates that told consumers to "update
  <self-template-path>" rewritten to drop the redundant self-reference.
- 8 docstring/comment self-references in cli, plugins, tests, utils,
  and the dashboard oidc.js.j2 header rewritten to describe the layout
  without naming the file's own path.
- 17 legitimate usage examples (shell scripts, test data lists, one
  template self-include doc-block) marked with ``nocheck:
  self-path-reference``.
- ``container.sh`` heredoc usage strings switched to ``${0}`` so the
  marker stays out of the user-visible help output.
- ``healthcheck/http.yml.j2`` doc-block rewritten to prose because
  ``{# ... #}`` Jinja comments cannot host an inner ``{# nocheck #}``
  (the inner ``#}`` closes the outer comment).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 03:53:44 +02:00
kevinveenbirkenbach 2e280a0434 test(web-app-peertube): add Playwright spec + env for landing and OIDC flows
- playwright.spec.js: cover landing domain from applications lookup,
  administrator OIDC login+logout via dashboard, and biber OIDC
  login+logout via dashboard (biber persona required by the MUSTs).
- playwright.env.j2: expose APP_BASE_URL, OIDC_ISSUER_URL,
  PEERTUBE_BASE_URL, and administrator/biber credentials rendered
  from the users lookup.

Validated in the Pass 1 fresh-purged deploy: peertube 3/3 passed
(landing 1.0s, administrator OIDC 2.5s, biber OIDC 2.4s).
2026-04-23 00:41:27 +02:00
kevinveenbirkenbachandClaude Opus 4.7 97ee7e65ea Bump peertube cgroup mem_limit during plugin install
Plugin install via `container exec` shares the cgroup with the long-running
peertube node process. Combined RSS was tripping the 8g mem_limit and
SIGKILLing peertube, which surfaced as "Container is restarting" on
subsequent exec attempts.

- Temp raise mem_limit + unlimited swap for the install window, restore
  the compose-defined value on the happy path only
- Derive the bumped ceiling from PEERTUBE_MEM_LIMIT + install heap cap
  via a new role-local `install_mem_limit` filter (overhead=4 covers
  native RSS and spawned npm/node-gyp workers)
- Extract the SQL upsert into templates/upsert-oidc-plugin.sql.j2
- Move enable/disable OIDC tasks under tasks/oidc/ and rename the
  diagnostics script
- Document the extract-code convention in docs/contributing/tools/ansible.md

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-20 22:55:08 +02:00
kevinveenbirkenbachandClaude Opus 4.7 6b5e4356a6 Add peertube plugin-install diagnostics on failure
Wraps the auth-openid-connect plugin install in block/rescue so that when
the container exec trips (e.g. "Container is restarting", cgroup OOM kill),
a dedicated script dumps container state, health, resource limits, stats,
last 250 log lines, host memory+swap, and dmesg OOM-killer evidence before
the task fails explicitly. Makes the actual cause visible in CI without
another round-trip.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-04-20 22:05:45 +02:00