CI run 25973408947's `test-development` matrix failed on dashboard
variant 1 (all shared services off): three tests ran and failed
because they either had no service gate or their gate pointed at a
flag that stayed literal `true`. Switching dashboard onto the
web-svc-css opt-in pattern plus adding the missing cdn/matomo gates
makes variant 1 skip cleanly while variant 0 keeps running the full
assertion path.
- `meta/services.yml`: migrate the `css` block from hardcoded
`{ enabled: true, shared: true }` to dynamic
`{ "{{ 'web-svc-css' in group_names }}" }` so variant pinning takes
effect on the gate.
- `meta/variants.yml`: add a `css: { enabled: true/false, shared: ... }`
entry to both variant blocks to give the dynamic flag the
two-polarity matrix coverage the lint requires.
- `files/playwright/playwright.spec.js`: add the missing
`skipUnlessServiceEnabled` calls to test 546 (`cdn`) and test 570
(`cdn` + `matomo`). Both tests were ungated; variant 1 ran them
and failed because the resources they depend on were not served.
Verified that variant 0 keeps running every assertion (the gates only
fire when the upstream service is pinned `false`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Decouples the CSS service concept from its underlying transport
(currently `web-svc-cdn`) so the flag becomes opt-in via inventory
group membership instead of staying literal-`true` everywhere.
- `roles/web-svc-css/`: new role. Marker / wrapper role that owns
the `css` entry in the service registry, declares `web-svc-cdn` as
the upstream that serves the actual bytes, and 301-redirects its
own canonical domain to the cdn domain so health probes return a
valid response.
- `roles/web-svc-cdn/meta/services.yml`: drop the `css` block — the
canonical owner is now web-svc-css; cdn keeps `javascript`.
- 51 consumer roles (web-app-* and web-svc-{coturn,file,html,
libretranslate,logout,onlyoffice}) migrated their `css:` block from
literal `{ enabled: true, shared: true }` to dynamic
`{ enabled: "{{ 'web-svc-css' in group_names }}", shared: ... }`,
with matching matrix coverage added to each role's
`meta/variants.yml` (true-pinning + false-pinning entries) so the
dynamic-flag variant lint stays green.
- 41 `# nocheck: playwright-service-flag` rationale strings updated
from the now-incorrect "canonical:cdn alias of static-asset
provider" to "css consumer, no Playwright surface (provider:
web-svc-css)".
Bundled refactor: `roles/web-svc-asset/tasks/01_core.yml` now
includes `web-opt-rdr-domains` directly with a single-entry mapping
to redirect its canonical domain to web-svc-file, instead of having
that wiring live in the global `tasks/stages/01_constructor.yml`
constructor. The `add_redirect_if_group` filter
(`plugins/filter/redirect_filters.py`) and its unit test are dropped
as dead code — both call sites are now role-local.
Dashboard's consumption of the new pattern is intentionally not part
of this commit and follows separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The administrator password literally starts with `'` (verified via cat
on the rendered .env/env: `FUSIONDIRECTORY_ADMIN_PASS='$tr0ng€xampl3PW\!`),
which Docker Compose's .env parser interprets as an opening single
quote and fails with `unterminated quoted value`. Wrapping both
credential fields with the existing `| dotenv_quote` filter
(plugins/filter/dotenv.py — double-quote + escape `$`/`\\`/`"`) makes
the value compose-safe. Matches the pattern used in
web-app-{mastodon,listmonk}/templates/env.j2.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- `library/database_query.py`: drop `no_log: True` from `config` /
`named_args` argument_spec; add `mask_values` bool param (default
True). When True, the module copies their string values into
`module.no_log_values` at runtime — same effective redaction as
before, but now toggleable per task.
- Roll out `mask_values: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}"` to
every `database_query` call (38 sites across web-app-{odoo,
openproject, peertube}) so the playbook constant in
group_vars/all/00_general.yml drives the redaction end-to-end. Set
it to false on demand for debugging unredacted rc/stderr.
Bundled: roles/web-app-openproject/tasks/02_ldap.yml also drops the
`async:`/`poll:` lines from Update/Create LDAP auth source. The
combination of `async` + `database_query` local-action + `no_log`
masked the async wrapper's result fetch with "Action failed: Unknown
error" while the SQL had actually succeeded (verified: 1 row in
`ldap_auth_sources` after the failing run). Sync execution matches the
pattern used by every other `database_query` caller in the repo.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
40-fusiondirectory init script (sanity_var checks at lines 354-358 of
/etc/cont-init.d/40-fusiondirectory in the image) requires the numbered
LDAP1_* names because the script supports multiple LDAP servers via a
`for (( i = 1; i <= NUM; i++ ))` loop. The previous LDAP_* names caused
`sanity_var LDAP1_ADMIN_PASS "Admin Password"` to abort init, leaving
the container in an unhealthy state.
Also moves the `environment:` block out of compose.yml.j2 into the
canonical `templates/env.j2`; the env_file is auto-wired via
sys-svc-container/templates/base.yml.j2 once `has_env()` is true.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The patch task in 02_ldap.yml ran `compose exec friendica` before the
container was up (it sits before the meta: flush_handlers in 01_core.yml),
and even after a forced flush it raced the entrypoint rsync that copies
/usr/src/friendica/ into the /var/www/html volume. The script then saw
`addon/ldapauth/ldapauth.php` missing and exited 1 with the misleading
"NO-MATCH ldap_createaccount return-line shape changed upstream" — for
30 retries × 2s, then fatal. CI run 25967376208 (job 76334304707) hit
this on the first push that introduced the patch task (9763cd8a4).
Fix: move the patch into 04_addons.yml, after both readiness probes
(`bin/console list` for vendor/addon rsync, `addon list all` for DB)
and before the addon sync that enables ldapauth. Gated by
FRIENDICA_LDAP_ENABLED. Drops the retries/until — once those probes
pass, the file is guaranteed present, so a non-zero exit now means real
upstream drift and should fail loud.
02_ldap.yml is back to the pre-patch shape: template ldapauth.config.php
and notify compose up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same brittle substring guard as the dropped blackbox test, and stale
after the matomo flag in dashboard's playwright.env.j2 moved from
non-canonical MATOMO_ENABLED to canonical MATOMO_SERVICE_ENABLED
(gated by skipUnlessServiceEnabled('matomo')). The env-spec gate
parity is already enforced by
tests/integration/roles/playwright/test_spec_env_gates.py.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The test asserted on substring presence in the raw Jinja template
instead of rendered behavior, so it broke on any equivalent rewrite
(quote style, helper var, whitespace). Its only purpose was a
regression guard against hardcoded insecure_skip_verify: false; that
guard is not worth the brittleness.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- plugins/lookup/status.py: `list(str)` on `APPLICATIONS_WHITELIST` (CSV from `-e`) split it into chars, breaking every consumer of `.whitelist` / `.running` (web-app-roles when-clause skipped all apps, keycloak per-app LDAP mappers never created).
- test-e2e-playwright/discover_playwright_roles: tolerate Python list-repr strings from non-native-Jinja2 stringification of `"{{ lookup('status').groups }}"` — otherwise `TEST_E2E_PLAYWRIGHT_APPS` resolves to `[]` and no spec runs.
- Unit tests cover both regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- web-app-opentalk: move CONTROLLER_HOST, INSECURE, BASE_URL, OIDC_*,
SHARED_FOLDER_OPENCLOUD_URL from the application service's compose
environment block into templates/env.j2 (already mounted as env_file
via sys-svc-container/base.yml.j2 on application + controller). OIDC
keys are guarded by OPENTALK_OIDC_ENABLED; SHARED_FOLDER_* by
OPENTALK_OPENCLOUD_ENABLED. INSECURE derives from
lookup('tls', application_id, 'enabled').
- web-app-prometheus, web-app-decidim: replace bare TLS_ENABLED Jinja
references with lookup('tls', application_id, 'enabled') so they
honor per-app TLS posture like the rest of the codebase.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Removes direct process.env.<X>_SERVICE_ENABLED reads (and bespoke
per-role <ROLE>_OIDC_ENABLED / SERVICE_* / OIDC_ENABLED env flags) in
favor of skipUnlessServiceEnabled/isServiceEnabled from
roles/test-e2e-playwright/files/service-gating.js, per
docs/contributing/artefact/files/role/playwright.specs.js.md service-
gating contract.
Per-role changes:
- discourse, matrix, dashboard: replace process.env.<X>_SERVICE_ENABLED
with the helper; drop now-redundant nocheck markers on the canonical
env flags.
- taiga: drop TAIGA_OAUTH2_ENABLED / TAIGA_OIDC_ENABLED env lines;
consume OAUTH2_SERVICE_ENABLED / OIDC_SERVICE_ENABLED via the helper.
- opentalk, opencloud, nextcloud: drop <ROLE>_OIDC_ENABLED (and
NEXTCLOUD_LDAP_ENABLED) env lines; spec switches to
isServiceEnabled('oidc'|'ldap').
- moodle: drop SERVICE_OIDC / SERVICE_LDAP / SERVICE_LAM in favor of
the canonical OIDC_SERVICE_ENABLED / LDAP_SERVICE_ENABLED plus a new
LAM_SERVICE_ENABLED rendered from 'web-app-lam' group membership.
- decidim: drop OIDC_ENABLED env line; SSO-button-visible scenario
uses skipUnlessServiceEnabled('oidc').
Dashboard asset test split:
The previously bundled "dashboard loads core css, javascript,
simpleicons, and logo assets" test triple-gated (css/cdn/logout)
skipped the entire surface when any single shared service was off.
Split into six focused tests so disabling one service skips only its
specific assertion:
- shared CSS injection (gate: css)
- CDN-served role-local stylesheet (gate: cdn)
- logout.js injection (gate: logout)
- simpleicon-backed cards (gate: simpleicons; nocheck dropped)
- role-core JS modules + logos (ungated, role-core)
- iframe-sync postMessage handler (ungated, role-core)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- plugins/lookup/roles_with_service.py: resolve url.base via the tls
sublookup and return canonical_url alongside id/canonical_domain.
- web-app-matomo/files/playwright/playwright.spec.js: parameterised
matomo-tracker injection assertion per consumer over
MATOMO_TARGET_ROLES_JSON, using target.canonical_url.
- web-svc-logout: new files/playwright/playwright.spec.js + templates/
playwright.env.j2; checks universal-logout injection on every consumer
declared in LOGOUT_TARGET_ROLES_JSON. meta/services.yml gains nocheck
markers explaining why this provider has no user-facing surface to
assert on.
- tests/unit/plugins/lookup/test_roles_with_service.py: cover the new
canonical_url field via a stubbed tls sublookup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- meta/main.yml: cut the wordy auto-discovery description to a one-liner ("Runs end-to-end Playwright tests against deployed applications.").
- defaults/main.yml: `TEST_E2E_PLAYWRIGHT_ONLY_ROLES` now reads `lookup('status').groups` (the full host-deployed set) instead of `lookup('status').running` (the CLI `--id`-narrowed subset). Playwright now exercises every dashboard-discoverable app on the host regardless of the deploy's `--id` scope.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause: upstream friendica-addons/ldapauth's `ldap_createaccount`
returns the raw `User::create()` result (`['user' => UserRow, 'password' => '']`),
but the `authenticate` hook only sets `authenticated = 1` when
`$user['uid']` is non-empty — never true for the autocreate path. First-
login LDAP users (e.g. biber) were inserted into `friendica.user` but
the session was never started, so /network kept serving the login form.
Encoded as an idempotent in-container patch task in `tasks/02_ldap.yml`
+ `files/patch_ldapauth_autocreate.sh` (reports PATCHED on first run,
ALREADY on subsequent runs). Runs whenever LDAP is wired, gated by
FRIENDICA_LDAP_ENABLED.
Other changes in the same stack:
- meta/services.yml: drop keycloakpassword addon (oauth2-proxy in front
owns SSO); ldap.enabled now follows svc-db-openldap group presence so
the variant matrix can disable it without a static override.
- tasks/01_core.yml + tasks/06_user_bootstrap.yml: pre-create the
administrator only in LDAP-disabled variants — when LDAP is wired,
ldapauth materialises the row on first admin login.
- files/playwright.spec.js: variant-aware spec replacing the parked
test.skip cases. Three tests now run: guest flow (always), admin
native login (picks double-login vs direct based on
OAUTH2_SERVICE_ENABLED), biber + admin side-by-side in isolated
contexts. PERSONA_* still blocked — friendica has no header-trusted
autocreate path.
- templates/playwright.env.j2: expose BIBER_* + service-gate flags.
- inventories/development/default.yml: biber email aligned with the
oauth2-proxy `email_domains = infinito.example` filter so biber's
Keycloak login isn't 403'd at the proxy.
- TODO.md: SAML migration outlined as the long-term replacement for
the double-login (currently second hop silently bypasses Keycloak 2FA).
Drive-by: trim double-space before `# nocheck` in
roles/sys-aur/meta/services.yml.
Verified on VARIANT=0 (oauth2 + ldap + oidc all enabled): three
playwright tests green, failed=0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adding `DASHBOARD_BASE_URL` (Pattern A) and `PROMETHEUS_BASE_URL` to the
playwright.env.j2 of these roles was required by the
`persona-required-envs` lint when `DASHBOARD_SERVICE_ENABLED` is
rendered. The companion `test_env_keys_used` lint then refused them
because no .js file consumed the keys.
Declare them as const in each spec so `process.env.<KEY>` appears
literally in a tracked source. The personas that the persona-required-
envs lint cites as consumers are blocked for these roles
(`PERSONA_*_BLOCKED=true`), so a benign const declaration with an
`eslint-disable-next-line no-unused-vars` is the smallest viable bridge
between the two lints.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The matrix DM test previously hardcoded `signInViaElementOidc`, which
only works when Keycloak is wired up. In matrix variant 1 (oidc off)
Element falls back to native or LDAP-backed password auth and the
SSO-button locator times out 30s.
Add a parallel password-form path and a dispatcher:
- `oidcServiceEnabled` is read from `OIDC_SERVICE_ENABLED`.
- New `signInViaElementPassword` drives Element's m.login.password form
(`#mx_LoginForm_*` selectors) with a slim post-login state machine:
skip-verification interstitial + a credential-rejection fast-fail
(mirroring the opencloud Option-C pattern) so an invalid password
surfaces in ~5s instead of a 3-min auth-marker timeout.
- New `signInViaElement` dispatches to OIDC or password based on the
flag.
- The two explicit OIDC tests are gated with
`skipUnlessServiceEnabled("oidc")` so they cleanly skip in variant 1
instead of hunting for an SSO button that does not exist.
- The DM test now uses the dispatcher so it exercises whichever auth
path is wired up for the deployment.
Element exposes `m.login.password` whether Synapse verifies the
credential against its own DB or via `ldap_auth_provider` — LDAP is
indistinguishable from the client side, so a single password path
covers both.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The "opencloud sso login (administrator) lands on files view" test
timed out at 5min in CI 25939428268 across all three retries with two
different root causes that the original test code hid behind a generic
`locator.press: Test timeout`:
- Retries #0/#2: Keycloak unmounts the credential form as soon as
the submit fires, so `locator.press("Enter")` race-detaches its
resolved element and Playwright retries the press in a 5-minute
loop even after the form already submitted successfully.
- Retry #1: Keycloak re-rendered the form with an "Invalid username
or password" error and the test continued to wait 90s on the
post-login banner — the credential rejection was buried behind a
generic locator timeout.
Two compounding fixes:
1. Race the press with `waitForURL(<not on Keycloak host>)` and
swallow the press error: whether the press completed cleanly or
was killed by the form detach does not matter as long as the
page leaves the Keycloak host.
2. After the race, probe `#input-error / .alert-error /
.kc-feedback-text` for ~5s. When Keycloak rendered a credential
rejection, throw a dedicated error with the actual message and
final URL instead of waiting 90s on the banner. The next CI run
will surface the real cause (true credential rejection vs other
post-login state) cleanly within ~65s.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pattern B in CI 25939428268: `biber: dashboard → app → universal logout`
and `administrator: dashboard → prometheus → app → universal logout`
timed out with "Expected redirect back to <role-domain> after Keycloak
login. Received https://dashboard.infinito.example/".
The personas (`runBiberFlow` / `runAdminFlow`) navigate via
`page.goto(APP_BASE_URL)` and then assert the post-Keycloak URL
contains the role's `canonicalDomain`. Their documented contract
(biber.js / admin.js header):
> Each role's biber scenario therefore visits its OWN canonical URL
> directly (no dashboard tile click) and exercises only that role.
After the recent `e14606b6c` flipped APP_BASE_URL to the dashboard URL
for several roles, that contract broke: the persona hit the dashboard
under oauth2-proxy, Keycloak round-tripped back to the dashboard, and
the canonical-domain assertion failed.
Fix per role:
- gitea: APP_BASE_URL back to `lookup('tls', application_id, 'url.base')`.
No spec-internal `page.goto("/")` calls, so no further changes needed.
- odoo: APP_BASE_URL back to the role's own URL AND render an explicit
DASHBOARD_BASE_URL + DASHBOARD_SERVICE_ENABLED + PROMETHEUS_BASE_URL.
The spec-internal "dashboard to odoo" tests now build their dashboard
URL from DASHBOARD_BASE_URL instead of relying on baseURL, and gate
with `skipUnlessServiceEnabled("dashboard")` so they skip cleanly in
matrix-deploy variants that disable the dashboard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Inner-loop iteration on a deployed stack often needs a one-off sidecar
container against the same docker daemon the stack lives on — typical
cases are rerunning a Playwright project with a patched .env, retrying
an arbitrary CLI image against an app container, or smoke-checking a
release image without going through Ansible. Without a wrapper the
caller has to spell out `docker exec <host-container> docker run --rm
<flags...> <image> <cmd>` by hand, which is verbose enough that agents
tend to fall back to redeploying instead of validating in-place.
Add `make run IMAGE=<ref> [CMD=...] [RUN_FLAGS=...]` plus the underlying
`scripts/tests/deploy/local/exec/run.sh`. The script reuses
`scripts/meta/env/all.sh` to resolve the host container name (same SPOT
as `make exec`), forwards RUN_FLAGS verbatim, and only wraps CMD in
`sh -lc` so default-entrypoint images and shell one-liners both work.
Wire both iteration loops to mention the wrapper next to `make exec`:
the Role Loop's Inspect block gets a bullet pointing at it, and the
Playwright Spec Loop gains a short "Debugging in the nested docker
daemon" section that also reminds the agent to mirror any successful
hand-patch back into the role's playwright.env.j2.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pattern A in CI 25939428268: the "dashboard to X" tests navigate to
`page.goto("/")` and expect an `Explore X` link rendered by the
dashboard. When the matrix-deploy variant excludes web-app-dashboard
from the round's include set (variant 1 pins `dashboard.shared: false`
+ `dashboard.enabled: false`), the dashboard container is not deployed
but the specs still run blind against an undeployed URL and time out.
Render `DASHBOARD_SERVICE_ENABLED` into each role's playwright.env.j2
(plus the persona-required base URLs `DASHBOARD_BASE_URL` and the
conditional `PROMETHEUS_BASE_URL` for closure) and gate every
dashboard-flow test body with `skipUnlessServiceEnabled("dashboard")`
so the test cleanly skips in variants without a dashboard instead of
producing a 5-minute timeout. Consistent with the existing service-
gating pattern already used in baserow/akaunting/gitea specs.
friendica is unchanged here: its spec was already refactored away from
the dashboard flow toward direct oauth2-proxy + friendica /login round
trips (no `page.goto("/")` against the dashboard) in a separate WIP.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The nextcloud LDAP failure in CI 25939428268 traced to a variant-merge
propagation gap: when meta/services.yml declares a flag as a literal
and meta/variants.yml overrides that same flag, the merged value does
not reliably reach downstream `lookup('config', ...)` consumers.
Plugin tasks then run against a provider that the round's inventory
does not contain (e.g. `occ ldap:search` -> "Lost connection to LDAP
server"). The workaround is to anchor the base value to a
`'<role>' in group_names` expression so the inventory itself decides
whether the consumer activates.
Same anti-pattern applied to nine other shared-LDAP consumers:
funkwhale, keycloak, kix, openproject, opentalk, snipe-it, suitecrm,
taiga, web-svc-xmpp. Each had `services.ldap.enabled: true # nocheck:
dynamic-flag` paired with a dynamic `shared`. Replaced with
`"{{ 'svc-db-openldap' in group_names }}"` for both flags.
Add a regression guard at
tests/integration/roles/meta/services/test_variants_no_static_override.py
that fails whenever meta/variants.yml overrides a first-level
`services.<key>.enabled` or `services.<key>.shared` flag while the
base in meta/services.yml is a static literal. Rule
`variants-static-override` is catalogued in the suppression doc.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
In CI matrix-deploy, nextcloud's variant-1 round drops svc-db-openldap
from the include set (variant pins `ldap.shared: false`), but the
inter-round purge had already wiped openldap_data + stack-dir. With
`services.ldap.enabled` hardcoded to `true`, the user_ldap plugin gate
in `plugins.user_ldap.enabled` resolved to true regardless, and the
LDAP setup tasks ran against a non-existent openldap container —
`occ ldap:search` failed with "Lost connection to LDAP server".
Anchor `services.ldap.enabled` and `services.mariadb.enabled` to actual
inventory-group presence so the plugin-level gate (which lookups
`services.ldap.enabled`) correctly disables the plugin in rounds that
don't deploy the central LDAP/DB. Also drop the redundant defaults
from the dependent plugin lookups now that the source flags are always
defined via group_names.
Behavior change for non-CI: deploying nextcloud without svc-db-openldap
in the inventory now disables the user_ldap plugin entirely rather
than running it against a missing LDAP server.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Newer Ansible refuses to bool-coerce list values in conditional contexts
(`Conditional result (False) was derived from value of type 'list'`).
The `until:` block of the wait-for-application-table task used
`(_peertube_app_table_check.rows | default([]))` directly as a sub-clause,
which is a bare list. Wrap with `| length > 0` for an explicit bool result.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`container inspect mariadb` is polymorphic and matched the same-named
external Docker network (created by sys-svc-compose for shared DBs)
instead of the container, producing "map has no entry for key 'State'"
template errors on every retry until the wait task timed out.
- plugins/lookup/database.py: add `network` field (dbtype when shared,
consumer entity otherwise) so roles get the DB network for both modes.
- roles/web-app-mediawiki/tasks/main.yml: switch to typesafe
`container container inspect` against the new MEDIAWIKI_DB_CONTAINER.
- roles/web-app-mediawiki/vars/main.yml: derive MEDIAWIKI_NETWORK_DATABASE
and MEDIAWIKI_DB_CONTAINER from lookup('database', ...).
- tests: cover the new `network` field for empty/local/shared cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both specs drive the user journey from the dashboard tile (`page.goto("/")`
+ `getByRole('link', { name: 'Explore Taiga' })` / 'Explore Mattermost'),
so APP_BASE_URL must resolve to the dashboard's base URL — not the role's
own canonical URL. With the role-self URL, `page.goto("/")` lands on the
app and `Explore <App>` never matches; aligns with peertube/pixelfed
which use the dashboard URL here.
Render the value un-quoted so docker `--env-file` does not pass literal
double quotes through to `process.env.APP_BASE_URL` (Playwright sets
`use.baseURL = process.env.APP_BASE_URL`; literal quotes break
`page.goto("/")` with "Cannot navigate to invalid URL").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
BookWyrm v0.8.5 dropped requirements.txt and now declares its runtime
dependencies in pyproject.toml under [dependency-groups]. The previous
build step `pip wheel --wheel-dir /wheels -r requirements.txt` failed
the first reuse-kept deploy with `Could not open requirements file`.
Re-shape the image around the upstream pattern:
* Builder installs pip>=25.1 (PEP 735 support) then `pip install --group main`
into an isolated `/venv`. The runtime stage copies that venv via
`COPY --from=builder /venv /venv` and puts it on PATH; no offline
wheelhouse layer in between.
* Add `libsass-dev` + `SYSTEM_SASS=True` to the build stage so the
Python libsass package links against the distro library instead of
spending ~25 min compiling its own copy on arm64. Match with
`libsass1` on the runtime stage.
* Add `gettext` + `tidy` to both stages (i18n + HTML sanitisation that
BookWyrm runs at request time).
* `chown` the venv to the unprivileged `bookwyrm` user so .pyc
compile-on-import works after the USER switch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The first reuse-kept deploy after introducing `database_query` failed
with `couldn't resolve module/action 'database_query'`: ansible only
searches for custom modules along the `library` config path, and the
module had landed under `plugins/modules/` (collection-style layout)
without that path being registered in ansible.cfg.
Realign with the repo's existing convention — `library/cert_check_exists.py`
and `library/docker_bridge_gateway.py` are already there — by relocating
the module to `library/database_query.py`. Make the search path explicit
in ansible.cfg (`library = ./library`) instead of relying on ansible's
implicit playbook-adjacent discovery, so the resolver stays predictable
across CLI / playbook / role-include entry points.
Move the unit suite into `tests/unit/library/` alongside the moved
module, fix the doc-string reference in the lint guard that points
callers at the canonical module location, and drop the now-empty
`tests/unit/plugins/modules/` package.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PKGMGR_REPO_URL / PKGMGR_REF were defined with `lookup('config', role_name, …)`.
The vars are referenced via `vars:` in pkgmgr's own task that does
`include_role: name: sys-util-git-pull`, and Ansible templates that `vars:`
block inside the included-role context — so `role_name` was already
`sys-util-git-pull` at evaluation time, not `pkgmgr`.
That produced a config lookup against `applications['sys-util-git-pull'].services.sys-util-git-pull.repository`,
which fails because sys-util-git-pull's services.yml declares the entity as
`util-git-pull` (no `sys-` prefix, per the established sys-* convention) and
has no `repository` key.
Hardcoding the application_id keeps the lookup immune to role_name drift no
matter where the var is referenced from.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The `/username|email/i` regex literals in the bookwyrm row tripped markdownlint's
MD056 (column count) — markdown reads bare `|` inside table cells as cell
separators. Escape both occurrences to `/username\|email/i`. Also extend the
existing pre-push hook config so `make lint` runs alongside `make test-external`
before every push, so the next markdown drift never makes it past the push gate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
aur.archlinux.org/yay.git is git-clonable but HTTP GET returns 404,
since AUR has no plain-HTTP browsing layer. Mark it with `# nocheck: url`.
Also flip the GitHub example URL in the integration-test docstring to
example.com/foo/bar.git so the reachability scan filters it via the
reserved-host list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces controller-side `community.postgresql.postgresql_query` /
`community.mysql.mysql_query` calls with a project-local `database_query`
Ansible module that wraps `container exec -i <container> <psql|mysql>`
and pipes SQL via stdin. The module routes via the DB container's own
network so the same task works identically in shared mode (central
`svc-db-*` container) and non-shared mode (per-role `<entity>-database`
container) — the `container` field of `lookup('database', …)` is
shared-aware and the module just reads it. SQL payloads come from a
file (`query_file:`, preferred) or as a raw string (`query:`, escape
hatch). `named_args:` substitutes `%(name)s` placeholders with engine-
appropriate escaping (postgres single-quote-doubling, mariadb backslash
escapes), so role tasks stop string-interpolating Jinja into SQL.
New backends extend the module's `_ENGINES` dispatch table; everything
else is engine-agnostic. The module ships at `plugins/modules/` (Ansible
collection-style layout) — the shebang lint test learned to recognise
that path as a valid module location alongside the existing per-role
`library/` layout.
Two lint tests enforce the convention:
* `tests/lint/ansible/modules/test_no_raw_db_query_module_usage.py`
forbids `community.postgresql.postgresql_query` /
`community.mysql.mysql_query` in role tasks. Opt out per-line with
`# nocheck: database-query` (used by `svc-db-postgres/tasks/02_init.yml`
where the role manages its own DB and the `database` lookup would be
self-referential).
* `tests/lint/ansible/modules/test_no_raw_database_query_string.py`
forbids inline `query:` on `database_query:` tasks. Opt out with
`# nocheck: database-query-raw` (no opt-outs in the current tree —
every SQL payload reaches the module through a `.sql` file under
`files/sql/`).
Role migrations:
* `web-app-peertube` — 3 tasks migrated, 3 new `.sql` files,
`templates/upsert-oidc-plugin.sql.j2` deleted in favour of
`files/sql/upsert-oidc-plugin.sql` + `named_args:`.
* `web-app-openproject` — 3 tasks migrated, 3 new `.sql` files for
the LDAP-auth-source check/update/insert chain.
* `web-app-odoo` — 33 tasks migrated, 9 new `.sql` files for the
LDAP / admin / init-database flows; 17 pre-existing OIDC `.sql`
files renamed snake_case → kebab-case and relocated from
`files/oidc/` to `files/sql/oidc/` (via `git mv` so rename history
stays intact).
Unit tests cover every dispatch-table entry, the named_args escaper
(strings, ints, floats, bools, None, single-quote and backslash
injection vectors), and the password-redactor that hides
PGPASSWORD/MYSQL_PWD assignments from the module's returned `cmd`
field.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `test_semver_pinning.py` under `tests/lint/ansible/services/`.
Recursively walks every `roles/*/meta/services.yml`, surfaces a
warning for every `image:`+`version:` pair and every `repository:`+`ref:`
pair whose value is not semver-compatible. Output via the shared
annotations helper (GitHub Actions `::warning::` plus pytest's
`warnings.warn`), with a single trailing suppression hint when any
finding fires. Never fails the test session.
Suppression reuses the existing markers:
- `# nocheck: docker-version` for `version:` pins
- `# nocheck: repository-version` for `ref:` pins
Both accept same-line or line-above placement. The existing external
update checks (`utils/update/docker.suppressed_services` and
`utils/update/repository.suppressed_ref_lines`) are loosened from
strict `line-above` to the default `same-or-above` for consistent
marker semantics across all three consumers.
Updates `docs/contributing/actions/testing/suppression.md` so the
catalog lists both consumers per marker and adds the previously
missing `repository-version` row.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three classes of drift cleaned up across 85 .md files:
- Em-dashes: replaced ` — ` with the contextually correct separator
(colon for `term: explanation`, period to split clauses across
sentences/source lines, parentheses for asides, semicolon for
tight-linked continuations). Two intentional exceptions remain
in `docs/contributing/documentation.md` (the rule literal) and
inside a fenced ```sh block in `docs/contributing/actions/debugging/ci.md`.
- SPOT self-labels: rewrote "This is the SPOT for X" to describe
the subject directly ("This directory holds X" / "This file
describes X"). Cross-references to other artefacts' canonical
role remain unchanged.
- Heading-emoji position: moved leading-emoji headings (`# 🧩 Title`)
to trailing position (`# Title 🧩`) per the headline rule that
emojis MUST come after the title text.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Folds the parallel Docker-image and git-ref version-update plumbing
into a single subsystem so the auto-update workflow, the external
freshness tests, and the unit tests all share one module layout.
utils/
update/
base.py # semver primitives shared by both backends
docker.py # OCI registry lookups + version: line rewriter
repository.py # git ls-remote walker + ref: line rewriter
cli/contributing/update/
docker/ # python -m cli.contributing.update.docker
repository/ # python -m cli.contributing.update.repository
scripts/github/update/
image_versions.sh
repository_refs.sh
Tests mirror the same shape under tests/{unit,external}/update/{docker,repository}/.
Also widens the Docker bump scope to all roles (previously gated on
`web-*`) so non-web roles like sys-svc-container get bumped too;
pins Bookwyrm's python image at 3.11-bookworm via
`# nocheck: docker-version` with a TODO until upstream supports
Python >= 3.12.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Flagged by the new external `test_repository_versions.py` — upstream
`bookwyrm-social/bookwyrm` had moved on from v0.7.5 (latest semver
tag v0.8.5).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `tests/external/repository/test_repository_versions.py` — the
git-source counterpart to `tests/external/docker/test_image_versions.py`.
Walks every role's `meta/services.yml` recursively, collects every
`(repository, ref)` pair (top-level entity, sub-entity, plugin map),
filters to semver-compatible refs, queries upstream via `git ls-remote
--tags`, and emits GitHub Actions `::warning::` annotations when a
newer semver tag exists. Always passes — outdated refs are warnings,
not hard failures, so the suite stays green when remotes are slow or
unreachable. Suppress per-entry with `# nocheck: repository-version`
on the line directly above the `ref:` key.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`postOnFriendicaViaUi` previously used
`getByRole("button", { name: "Sign in", exact: true })`. Friendica's
own form labels the submit button "Sign in", but when oauth2-proxy is
in front of Friendica every `/login` request first redirects to
Keycloak whose login form labels the submit "Sign In" (capital I).
Exact-match misses that variant, so the click never fires, the
Keycloak round-trip never completes, and the test times out at 300s
with the form still visible.
Swap the locator to `getByRole("button", { name: /sign\s*in/i }).first()`
so either label resolves to the same element.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The role's source-build pattern (declared `repository:` + `ref:` in
meta/services.yml) replaced the in-repo notes; the file no longer
tracked an open action item.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The per-role default network is created externally by `community.docker.docker_network`
in this task before `compose up` runs. Compose then refuses to take
over the existing network with:
network <name> was found but has incorrect label
com.docker.compose.network set to "" (expected: "default")
Reproduced in CI run 25890032686 for web-app-matrix, web-app-gitea,
and — as a downstream of the same compose-up failure — web-app-mediawiki
(whose mariadb dependency never started).
Apply the two labels compose writes itself when it owns a network's
lifecycle: `com.docker.compose.network=default` (the network short name
under the top-level `networks:` block in our templates is `default`,
overridden to the per-entity docker name via the `name:` field) and
`com.docker.compose.project=<entity>` (matches both the explicit
`--project` flag on `compose pull` and the `basename(chdir)` default
used by `compose up` in this repo).
Different mechanism than commit 762586a03 (which set `external: true`
on the compose template and was reverted by 602c05ed3 because it broke
the first-deploy path for roles that did not go through this task) —
this fix keeps the network compose-managed at the template level and
only adds the labels the daemon would have emitted on its own.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-locates every git-clone URL with its owning entity in the role's
`meta/services.yml` so the unified `image`/`version` ↔ `repository`/`ref`
naming convention covers secondary repos (plugins, build helpers,
bootstrap sources) just as cleanly as the primary entity source.
Migrations:
- sys-aur, pkgmgr (non-invokable bootstrap roles): `meta/services.yml`
gains `repository:`/`ref:` on the role's own entity; `vars/main.yml`
renders them via `lookup('config', role_name, …)` because these
roles never set `application_id`.
- web-app-{akaunting,bigbluebutton,bluesky,discourse,roulette-wheel,
taiga}: declare the secondary git URL on the most fitting entity
(`akaunting` itself for the docker-helper repo; `view` for bluesky's
social-app web client; `plugins.discourse-ldap-auth` for that
Discourse plugin; `oidc` for taiga's OIDC plugin). Each role gains
named `<ROLE>_REPOSITORY` / `<ROLE>_REF` constants in `vars/main.yml`
so tasks and templates stay readable instead of inlining a
`lookup('config', application_id, …)` string at every call site.
- svc-db-postgres `templates/Dockerfile.j2`: keep the `pgvector`
extension clone but mark the line with `{# nocheck: repository-declared #}`
since pgvector is an extension source, not the role's primary repo.
New integration test
`tests/integration/roles/meta/services/test_repository_declared_in_services.py`
walks every role's text files and fails if a literal `https://…/<…>.git`
URL is not declared as a `repository:` value somewhere in that role's
`meta/services.yml` (recursive — top-level entity, sub-entity, plugin
map). Legitimate exceptions opt out with `# nocheck: repository-declared`
on the same line or the preceding line; Jinja templates may use the
`{# … #}` form so a `\`-continued Dockerfile shell command stays valid.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Unify the source-repo half of the entity naming convention introduced
by `0cc37d0fd`. Every entity that declares `repository:` MUST now
also declare `ref:` (git ref — tag, branch, or commit SHA). The
overloaded `version:` (when used as git ref) and the alternative
`branch:` key are gone.
Roles migrated:
* web-app-bigbluebutton: `version: main` → `ref: main`
(consumed in tasks/01_core.yml as `docker_git_repository_version`).
* web-app-discourse: `version: main` → `ref: main`;
`DISCOURSE_VERSION` renamed to `DISCOURSE_REF`.
* web-app-bridgy-fed: `branch: main` → `ref: main`;
`BRIDGY_REPO_BRANCH` renamed to `BRIDGY_REF` across vars, the
Dockerfile ARG, and the compose build-args.
* web-app-postmarks: identical migration to bridgy-fed.
New guard `tests/integration/roles/meta/services/test_repository_requires_ref.py`
walks every `meta/services.yml` and fails when an entity declares
`repository` without `ref` (or vice versa), so the next role that
forks the source-build pattern cannot regress to either half-
declaration without `make test` catching it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Disentangle the docker-image vs source-repository dimensions in
`meta/services.yml`:
* `image` + `version` → OCI image used as `FROM image:version` (and
the final stage). Matches the existing convention used by every
pre-built role.
* `repository` + `ref` → git source clone. `ref` is the umbrella term
for tag / branch / commit SHA (matches `git ls-remote`,
GitHub Actions `with: { ref }`, GitLab CI `ref:` etc.), avoiding
the misleading "tag" when the value is a branch.
Propagation:
* `vars/main.yml`: `BOOKWYRM_IMAGE` + `BOOKWYRM_VERSION` +
`BOOKWYRM_REPOSITORY` + `BOOKWYRM_REF`. The custom image is now
tagged with `BOOKWYRM_REF` so the in-registry tag reflects the
upstream source revision (more useful than the base-image tag).
* `compose.yml.j2`: passes all four through as build args.
* `files/Dockerfile`: `ARG BOOKWYRM_IMAGE/VERSION/REPOSITORY/REF`,
both `FROM` stages use `${BOOKWYRM_IMAGE}:${BOOKWYRM_VERSION}`,
`git clone --branch "${BOOKWYRM_REF}" "${BOOKWYRM_REPOSITORY}"`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two changes that together stop one badly-behaved role from stalling
the whole deploy:
1. `roles/test-e2e-playwright/tasks/run_one.yml` — pass `timeout:
{{ TEST_E2E_PLAYWRIGHT_RUN_TIMEOUT }}` to the `command` task. New
default `1200` (20 min) lives in `defaults/main.yml`. Without an
outer Ansible-side cap, a single hung test could burn 5 min ×
3 retries × N specs before either `--max-failures` or the
deploy supervisor noticed; friendica's run today sat at 27 min
before the operator killed it manually.
2. `roles/test-e2e-playwright/defaults/main.yml` — append
`--max-failures=5` to the default Playwright command. Once five
tests have failed, the sidecar stops instead of chewing through
the rest of the retry budget for tests that are obviously broken.
3. `roles/web-app-friendica/templates/playwright.env.j2` — declare
`PERSONA_BIBER_BLOCKED=true` and `PERSONA_ADMINISTRATOR_BLOCKED=true`.
Friendica authenticates against its own local DB / LDAP via the
`ldapauth` addon, not OIDC. The generic `runBiberFlow` /
`runAdminFlow` helpers drive an OAuth2-proxy + Keycloak round-trip
that Friendica cannot reconcile, so the generic persona tests
wedge on the Keycloak sign-in form. The bespoke admin/biber tests
above own the actual login flow.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- web-app-hugo/meta/services.yml: nginx 1.31.0-alpine → 1.31-alpine
(drop the patch component to stay on the rolling 1.31.x tag)
- web-app-taiga/meta/services.yml: rabbitmq 3.8-management-alpine →
4.3-management-alpine for both async-rabbitmq and events-rabbitmq
Note on RabbitMQ: 3.8 → 4.x is a major version jump that skips
3.9–3.13. A fresh CI deploy on an empty volume is fine — RabbitMQ
initialises Mnesia with 4.x defaults. An in-place upgrade of a
populated cluster would require landing 3.13 first to enable feature
flags before 4.x will accept the database. Classic mirrored queues
are gone in 4.x; if any taiga RabbitMQ config still uses them, switch
to quorum queues.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
