The two-line block
# variant pinning every dynamic services.yml flag to false so the
# matrix also exercises the out-of-group_names branch.
sat inside the variant-0 (true-pinning) block of 46 variants.yml files —
inverted relative to its content. Removed across the board along with
the now-orphaned blank line that used to separate it from the
preceding service entry.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Replace 13 raw `container exec mysql/psql` invocations across
web-app-pixelfed, web-app-fider, and web-app-mattermost with the
canonical `database_query` module + `query_file:` + `named_args:`
pattern. The module already speaks both shared (central `svc-db-*`)
and non-shared (per-role `<entity>-database`) modes via the
`database` lookup, so the same task shape works regardless of DB
topology — the per-role hardcoded container-exec argv was the only
blocker.
- Extract every inline SQL body into per-role `files/sql/` trees.
Pixelfed splits further into `files/sql/oidc-mapping/` because the
OIDC-mapping cluster is a distinct concern from the bootstrap /
administrator queries. Each SQL file is single-purpose and named by
verb-noun (count-users, select-admin-id, upsert, etc.) to match the
precedent set by web-app-odoo.
- Fider's existing `files/{create_admin_user,configure_oidc_provider}.sql`
move to `files/sql/` and switch from psql-specific `:'var'`
variable substitution (which forced `stdin:` piping with `-v`
flags) to engine-neutral `%(name)s` placeholders the module
escapes per backend.
- Pixelfed loses `PIXELFED_MARIADB_ARGS` + 3 derived DB vars
(`*_CONTAINER`, `*_USERNAME`, `*_PASSWORD`) from `vars/main.yml` —
they were the manual argv bundle for the old container-exec
pattern and become orphan once `lookup('database', application_id)`
takes over.
- Every refactored task carries `mask_values:
MASK_CREDENTIALS_IN_LOGS` (module-level credential redaction);
tasks that previously used `no_log: MASK_CREDENTIALS_IN_LOGS`
keep both directives so the task-level masking stays alongside
the module-level one.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add the missing `---` document-start marker to two `meta/variants.yml`
files and re-indent the inline comment in web-app-mattermost so it
aligns with its enclosing block, matching the convention applied across
the other variants files in 5cfb24277.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Bespoke `test(...)` blocks that drive a real Keycloak SSO round-trip
(odoo, fider, mattermost, peertube) had no `test.skip(\!oidcEnabled, ...)`
guard. In matrix variants that render `OIDC_SERVICE_ENABLED=false`, the
Playwright locator for the SSO button never resolves; each test waits
300s, retries 2x, and 2-3 such tests in series blow past the 1200s
Ansible task-wrapper timeout — Bundle α odoo variant 1 reproduced this.
Pattern parity with `roles/web-app-moodle/files/playwright/playwright.spec.js`:
import `service-gating`, derive `<flag>Enabled` once, prefix each
affected test with `test.skip(\!<flag>Enabled, "...")`.
- odoo (verified empirically against running variant-1 stack via
`make run`: 4 skipped, 1 passed, 2.2s — was 1200s timeout)
- fider, mattermost, peertube: same pattern, 2 bespoke tests each
- kix: gates on `oauth2Enabled` + `ldapEnabled` instead — the role's
variant matrix has no `oidc` flag, the flow is
KIX → OAuth2-proxy → Keycloak → KIX-LDAP, both can be independently
disabled per variant.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Matcher in web-app-prometheus's parametrized scrape test now also
reads `labels.app`. Native scrape jobs (set `app="web-app-<id>"` via
each role's `templates/prometheus.yml.j2` fragment) now match alongside
blackbox-healthz targets. A single iteration asserts up=1 for both
surfaces per consumer, so per-app native tests become redundant.
- Scenario I (`/metricz`) iterates `prometheusTargetRoles` and asserts
each consumer's `app="<role_id>"` label is present in the
lua-resty-prometheus output. Subsumes the per-app metricz-label spot
checks.
- Remove four redundant per-app prometheus tests from web-app-gitea and
web-app-mattermost playwright specs (metricz-label + native-scrape
up=1), drop the orphaned `PROMETHEUS_BASE_URL=` declarations from
their env.j2 files, and clean up `OIDC_ISSUER_URL=` in gitea's env.j2
(only used by the removed scenarios).
- Drop the dead `const prometheusBaseUrl` declarations plus their
eslint-disable shut-up comments from web-app-fider, web-app-odoo, and
web-app-taiga playwright specs, along with the matching
`PROMETHEUS_BASE_URL=` blocks in those roles' env.j2.
- Correct the persona-required-envs lint: only `MATOMO_BASE_URL` is
actually consumed (by `personas/utils/csp.js`). `DASHBOARD_BASE_URL`
and `PROMETHEUS_BASE_URL` were aspirationally required by a lint
whose docstring described persona-collapse behaviour that no persona
implements. Update `_REQUIRED_URLS` and the docstring, and drop the
unused `\!dashboardBaseUrl` token from `_ALLOWED_SKIP_GUARDS` in
test_strict_mode.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- New bidirectional contract test ensures `services.prometheus.native_metrics.enabled`
and `roles/<role>/templates/prometheus.yml.j2` stay consistent across the roles tree,
replacing per-role spot checks for gitea/mattermost/matrix. Document the `native-metrics`
suppression rule and apply it to web-app-mattermost (prepared-but-default-off fragment).
- Move every test in `tests/integration/meta/config/` into the per-domain layout under
`tests/integration/roles/meta/{services,server}`. Group prometheus tests under
`services/prometheus/`, repositories tests under `services/repositories/`, the
`server.csp` shape test under `server/`. Move `test_non_empty.py` one level up
(scans all meta/*.yml, not just services.yml).
- Delete `tests/integration/meta/spots/` — both files were only `assertIn` over file
contents (brittle, no behavioural coverage, knowledge belongs in code comments or
proper integration tests).
- Harden `test_project_root_import.py` to catch chained `Path(__file__)…parent.parent.…`
walks (the attribute-access twin of `parents[N]` that previously slipped through).
Refactor 6 surfaced sites to import canonical `PROJECT_ROOT` / `ROLES_DIR`, plus
delete the now-redundant `_find_roles_dir` / `_find_project_root` helpers in
`native_metrics_apps` lookup and `invokable_paths` filter plugin.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Decouples the CSS service concept from its underlying transport
(currently `web-svc-cdn`) so the flag becomes opt-in via inventory
group membership instead of staying literal-`true` everywhere.
- `roles/web-svc-css/`: new role. Marker / wrapper role that owns
the `css` entry in the service registry, declares `web-svc-cdn` as
the upstream that serves the actual bytes, and 301-redirects its
own canonical domain to the cdn domain so health probes return a
valid response.
- `roles/web-svc-cdn/meta/services.yml`: drop the `css` block — the
canonical owner is now web-svc-css; cdn keeps `javascript`.
- 51 consumer roles (web-app-* and web-svc-{coturn,file,html,
libretranslate,logout,onlyoffice}) migrated their `css:` block from
literal `{ enabled: true, shared: true }` to dynamic
`{ enabled: "{{ 'web-svc-css' in group_names }}", shared: ... }`,
with matching matrix coverage added to each role's
`meta/variants.yml` (true-pinning + false-pinning entries) so the
dynamic-flag variant lint stays green.
- 41 `# nocheck: playwright-service-flag` rationale strings updated
from the now-incorrect "canonical:cdn alias of static-asset
provider" to "css consumer, no Playwright surface (provider:
web-svc-css)".
Bundled refactor: `roles/web-svc-asset/tasks/01_core.yml` now
includes `web-opt-rdr-domains` directly with a single-entry mapping
to redirect its canonical domain to web-svc-file, instead of having
that wiring live in the global `tasks/stages/01_constructor.yml`
constructor. The `add_redirect_if_group` filter
(`plugins/filter/redirect_filters.py`) and its unit test are dropped
as dead code — both call sites are now role-local.
Dashboard's consumption of the new pattern is intentionally not part
of this commit and follows separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adding `DASHBOARD_BASE_URL` (Pattern A) and `PROMETHEUS_BASE_URL` to the
playwright.env.j2 of these roles was required by the
`persona-required-envs` lint when `DASHBOARD_SERVICE_ENABLED` is
rendered. The companion `test_env_keys_used` lint then refused them
because no .js file consumed the keys.
Declare them as const in each spec so `process.env.<KEY>` appears
literally in a tracked source. The personas that the persona-required-
envs lint cites as consumers are blocked for these roles
(`PERSONA_*_BLOCKED=true`), so a benign const declaration with an
`eslint-disable-next-line no-unused-vars` is the smallest viable bridge
between the two lints.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pattern A in CI 25939428268: the "dashboard to X" tests navigate to
`page.goto("/")` and expect an `Explore X` link rendered by the
dashboard. When the matrix-deploy variant excludes web-app-dashboard
from the round's include set (variant 1 pins `dashboard.shared: false`
+ `dashboard.enabled: false`), the dashboard container is not deployed
but the specs still run blind against an undeployed URL and time out.
Render `DASHBOARD_SERVICE_ENABLED` into each role's playwright.env.j2
(plus the persona-required base URLs `DASHBOARD_BASE_URL` and the
conditional `PROMETHEUS_BASE_URL` for closure) and gate every
dashboard-flow test body with `skipUnlessServiceEnabled("dashboard")`
so the test cleanly skips in variants without a dashboard instead of
producing a 5-minute timeout. Consistent with the existing service-
gating pattern already used in baserow/akaunting/gitea specs.
friendica is unchanged here: its spec was already refactored away from
the dashboard flow toward direct oauth2-proxy + friendica /login round
trips (no `page.goto("/")` against the dashboard) in a separate WIP.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both specs drive the user journey from the dashboard tile (`page.goto("/")`
+ `getByRole('link', { name: 'Explore Taiga' })` / 'Explore Mattermost'),
so APP_BASE_URL must resolve to the dashboard's base URL — not the role's
own canonical URL. With the role-self URL, `page.goto("/")` lands on the
app and `Explore <App>` never matches; aligns with peertube/pixelfed
which use the dashboard URL here.
Render the value un-quoted so docker `--env-file` does not pass literal
double quotes through to `process.env.APP_BASE_URL` (Playwright sets
`use.baseURL = process.env.APP_BASE_URL`; literal quotes break
`page.goto("/")` with "Cannot navigate to invalid URL").
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI run 25868178119 — both jobs failed Playwright with the same
signature as web-app-kix's pre-fix state:
Error: no in-app logout control reachable on the current
authenticated surface
The generic `runBiberFlow` / `runAdminFlow` helpers cannot find an
in-app logout control on:
* Taiga's post-login UI (the bespoke `dashboard to taiga login and
logout` test at spec line 543 drives Taiga's own logout flow).
* Mattermost's `/landing#/` post-login surface (the bespoke
`dashboard to mattermost: sso login, verify channel view, logout`
test at spec line 273 drives Mattermost's own logout flow).
Declare `PERSONA_BIBER_BLOCKED=true` + `PERSONA_ADMINISTRATOR_BLOCKED=true`
per Rule 11 so the generic flows skip cleanly. Same pattern already
applied to web-app-{kix,nextcloud,postmarks,bluesky,fediwall}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The three playwright.env.j2 templates rendered
`APP_BASE_URL=lookup('tls', 'web-app-dashboard', 'url.base')` — a
stray copy-paste from the dashboard-tile flow that put the persona
helper on the dashboard login instead of the role's own surface.
After the dashboard's Keycloak round-trip the URL stayed on
dashboard.infinito.example, the canonical-domain poll
(`taiga.kanban.*`, `mattermost.*`, `social.*`) timed out at 60s and
the universal-logout test failed with
`Expected redirect back to <app>.infinito.example after Keycloak login`.
Point each `APP_BASE_URL` at the role's own canonical URL (the
project-wide convention used by baserow, bluesky, bookwyrm, akaunting,
hugo, jenkins, mig, moodle, …). The persona helper now lands on the
role's surface, the OIDC chain redirects to that role's Keycloak
client, the assertion passes, and the in-app logout step has a real
authenticated surface to operate on.
Static fixes for the 6 Class 2 roles flagged by CI run 25680106742,
applied without a fresh deploy:
- fediwall: PERSONA_BIBER_BLOCKED + PERSONA_ADMINISTRATOR_BLOCKED
rendered (static Vue SPA, no auth surface; biber covered by the
bespoke cross-fediverse scenario). Confidence 30% → 75%.
- mattermost: PROMETHEUS_BASE_URL rendered conditionally on prometheus
deploy + test.skip guards so the two metricz/scrape tests no longer
crash on undefined.
- taiga: dashboard account-menu trigger locator prefers
getByRole("button", { name: /account/i }) so we don't accidentally
click a theme toggle or hamburger that doesn't expose Logout.
- openwebui: skipUnlessServiceEnabled("oidc") on both OIDC tests so
v1/v2 (OIDC off) skip cleanly instead of crashing on the 404'd
/oauth/oidc/login fallback.
- bigbluebutton: skipUnlessServiceEnabled("oidc") + Greenlight
?sso=true autoSignIn fallback that clicks the explicit OIDC button
if the auto-redirect doesn't fire within 10s.
- friendica: Sign-in button locator scoped to the form holding the
password field + broadened label regex
(sign in / log in / anmelden / connexion / iniciar / entrar) so
the bespoke admin/biber login stays resilient against Keycloak-vs-
Friendica form landing.
Matrix updated: fediwall jumps to 75%, the other five carry per-row
notes documenting what landed and what still needs deploy-time
verification.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Filter no longer takes a `database_volume=` parameter. The volume name is
derived from the central provider's `services.<dbtype>.name` (shared) or
`<entity>_database` (dedicated) — mirroring `plugins/lookup/database.py`'s
`volume_prefix + host` construction. Simultaneous postgres + mariadb on
the same role now raises an AnsibleFilterError with a clear message;
support is deferred per the docstring TODO.
The integration test (renamed from `test_shared_service_volumes.py` to
`test_compose_volumes_call_required.py`) now requires every
`roles/*/templates/compose.yml.j2` to call `compose_volumes`, so the
top-level `volumes:` block is always rendered. Without it, flipping a
shared service to dedicated mode adds a named-volume reference that
Compose rejects with `service "<name>" refers to undefined volume`.
Adds the call to 14 templates that were missing it (dashboard, hugo,
sphinx, bridgy-fed, fediwall, littlejs, mig, mini-qr, navigator,
roulette-wheel, simpleicons, collabora, logout, openresty) and strips
the `database_volume=...` argument from every other role plus Taiga's
now-unused `TAIGA_DATABASE_VOLUME` var.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The shared biber/admin persona helpers in
roles/test-e2e-playwright/files/personas/*.js collapse the persona
scenario when either APP_BASE_URL or CANONICAL_DOMAIN is empty. The
twelve roles below shipped their env template without
CANONICAL_DOMAIN, so the helper fell into the auth-less collapse
branch and silently `test.skip()`-ed the persona scenarios despite
the roles having an actual Keycloak-backed auth surface. Render
CANONICAL_DOMAIN on every role whose spec drives a real persona so
the scenario runs end-to-end and a real regression cannot hide
behind a silent skip.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Promote four cheap rules in `eslint.config.js`:
* `no-unused-vars`: warn → error (with `argsIgnorePattern: "^_"`)
* `no-var`: error
* `eqeqeq`: error (always)
* `playwright/no-wait-for-navigation`: off → error
* `linterOptions.reportUnusedDisableDirectives`: off → warn
Add a repo-root CommonJS block so `eslint.config.js` itself sees Node
globals (`require`, `module`).
Fix every flagged offender:
* 138 stale persona-helper imports across 70 spec files
(`isServiceEnabled`, `skipUnlessServiceEnabled`, `runAdminFlow`,
... — leftovers from the SPOT consolidation).
* 4 unused catch bindings → modern `catch {}` syntax.
* 2 unused `dashboardBaseUrl` declarations (minio, moodle).
* 1 unused `label` arg → renamed to `_label` to opt out per the
argsIgnorePattern.
* 1 intentionally-kept legacy helper in web-app-wordpress's spec
receives a per-line `eslint-disable-next-line` with reason; the
preceding docstring already explained why it stays.
* 1 `page.waitForNavigation` in web-app-fediwall's Mastodon OIDC flow
receives a per-line disable with reason — the navigation target
depends on the runtime OIDC issuer; `waitForURL` would need a
runtime-built pattern.
* 10 stale `// eslint-disable-next-line no-console` directives now
surfaced by `reportUnusedDisableDirectives`; removed.
ESLint after this pass: 0 errors, 154 warnings (148 pre-existing
`playwright/expect-expect` plus 6 `playwright/no-force-option` that
were already inherited from `playwright/flat/recommended`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Strip every `req <NNN>`, `req-<NNN>`, `Requirement <NNN>`,
`requirement-<NNN>`, and `docs/requirements/<NNN>-...md` mention from
non-Markdown source files. Comments anchored to requirement numbers
rot when requirements are renumbered or rewritten and conflate
present-state behaviour (which the comment should describe) with the
audit trail of how the code got there (which belongs in commit
messages, PR descriptions, and the requirement file itself).
* Document the rule in `docs/contributing/documentation.md` under
"Comments". Code MUST describe the *why* in present-state language;
contract enforcement lives in tests, not folklore.
* Clean 254 source files across `roles/`, `tests/`, `plugins/`,
`cli/`, `scripts/`, `utils/`, `group_vars/`, and the repo-root
`eslint.config.js`. Pure comment / docstring / log-message edits;
no behavioural change.
* Add `tests/lint/repository/test_no_req_references_in_code.py` to
enforce the rule going forward across `.js`, `.py`, `.yml`,
`.yaml`, `.j2`, and `.sh` files. The legacy
`test_no_stale_requirement_refs.py` is exempt as the only file
whose purpose is to detect these very tokens.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fourteen role specs (bigbluebutton, discourse, fider, gitea, mailu,
matrix, mattermost, odoo, openwebui, peertube, postmarks, prometheus,
yourls, libretranslate) each carried an inline `performOidcLogin`
that drove the Keycloak login form. The bodies fell into four
clusters split between the role-based selector strategy
(`getByRole`) and the legacy input-name strategy
(`input[name='username']`); the existing shared
`performKeycloakLogin` already covered both via `.or()` chains but
also asserted post-login navigation, so it could not be reused
verbatim.
Changes
-------
- `personas/utils/keycloak.js` gains `performKeycloakLoginForm
(target, username, password)` — form-fill + submit only, accepts
`Page` OR `FrameLocator` so iframe-embedded Keycloak pages work
without branching at the call site. The selectors are the union
of all four prior cluster shapes (role-based getter chained with
`.or(input-name fallback)`, sign-in matched against
`/sign in|login|log in/i` plus `input#kc-login` family). The
existing `performKeycloakLogin` and
`performKeycloakLoginExpectingDenial` now delegate to the new
helper, so their public API is unchanged.
- The 14 specs strip their inline `performOidcLogin` and amend
`require("./personas")` with `performKeycloakLoginForm`. Every
call site and comment reference is renamed in lockstep.
Drift handling
--------------
- Cluster 1 (5 specs) expected the accessible name `"Password"`
exactly; the canonical regex `/^password$/i` is case-insensitive
and matches every prior name plus lowercase variants — strictly
more permissive.
- Cluster 1's sign-in button matched `/sign in/i`; the canonical
matches `/sign in|login|log in/i` plus the legacy `input#kc-login`
family — strictly broader.
- Cluster 1 awaited `usernameField.press("Tab")` without a `.catch`;
the canonical wraps the press in `.catch(() => {})` so an
unexpected Tab failure no longer aborts the flow.
- Cluster 2 / 3 used `expect(usernameField).toBeVisible({ timeout })`
for the visibility wait; the canonical uses
`usernameField.waitFor({ state: "visible", timeout })`, the
Locator-method counterpart of the same wait — functionally
equivalent.
Verification
------------
- 419 passed / 1 skipped / 0 failed across `tests/lint/`,
`tests/integration/roles/`, `tests/unit/plugins/lookup/`.
- `make lint-javascript`: 0 errors, 283 warnings (unchanged from
baseline).
- 0 inline `performOidcLogin` definitions remain in any role spec,
and 0 stale references to the legacy name in any context (call
sites and comments are renamed in the same pass).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both helpers were inlined verbatim in 71 of the role-local
`playwright.spec.js` files (52 with the compact form, 19 with the
verbose form, plus one parameter-name variant). All bodies were
semantically identical.
Changes
-------
- New `roles/test-e2e-playwright/files/personas/utils/dotenv.js` holds
the canonical pair (`decodeDotenvQuotedValue` and `normalizeBaseUrl`),
re-exported through `personas/utils/index.js` so specs that
`require("./personas")` get them via the existing aggregator.
- Every role spec drops its inline copies and pulls the helpers from
`./personas` instead. Specs that only reference one of the two
carry only that one in the destructuring (no unused-import noise).
Doc
---
`docs/contributing/artefact/files/role/playwright.specs.js.md` no
longer carries the "once a third spec needs them, MUST be promoted"
directive: the helpers live at the shared path and inline copies are
forbidden.
Verification
------------
- 419 passed / 1 skipped / 0 failed across `tests/lint/`,
`tests/integration/roles/`, `tests/unit/plugins/lookup/`.
- `make lint-javascript`: 0 errors, 283 warnings (unchanged from
baseline; no new unused-imports introduced).
- 0 inline `function decodeDotenvQuotedValue|normalizeBaseUrl`
definitions remain in any role spec.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cross-service surface assertions for dashboard, prometheus and matomo
no longer run in every consumer role's persona. They are owned by the
provider's own spec, parameterised once over a `<NAME>_TARGET_ROLES_JSON`
manifest rendered into the provider's `templates/playwright.env.j2` via
the new `lookup('roles_with_service', '<svc>')` Ansible lookup.
Changes
-------
- New `plugins/lookup/roles_with_service.py` enumerates roles that
declare `services.<svc>.{enabled, shared}` truthy in the merged
applications config, returning `[{id, canonical_domain}, …]`.
- `roles/web-app-{dashboard,prometheus,matomo}/files/playwright.spec.js`
parse `<NAME>_TARGET_ROLES_JSON` and parameterise the per-consumer
assertion (dashboard tile reachability, prometheus `up=1`, matomo
SitesManager registration). The matomo spec gains an explicit
biber-denial test (request body validated against admin DOM markers
per the body-marker rule); the prometheus admin-reach test drops
the dashboard-iframe path in favour of direct goto.
- Persona helpers `roles/test-e2e-playwright/files/personas/{admin,
biber,guest}.js` enter at `${APP_BASE_URL}/` directly. The
`clickRoleTileFromDashboard` step and the `assertPrometheus*` /
`assertMatomo*` cross-service probes are gone; `utils/{dashboard,
prometheus,matomo}.js` are deleted. `utils/csp.js` switches to
literal `process.env.<KEY>` reads so the env-keys-used parity guard
recognises matomo's CSP injector consumption.
- 42 consumer-role `templates/playwright.env.j2` templates drop
`DASHBOARD_BASE_URL`, `DASHBOARD_SERVICE_ENABLED`, `PROMETHEUS_BASE_URL`,
`PROMETHEUS_SERVICE_ENABLED`. `MATOMO_BASE_URL` and
`MATOMO_SERVICE_ENABLED` stay because the per-role CSP injection
helper still gates on matomo's tracker host.
- `tests/integration/roles/test_playwright_env_services_match.py`
globally exempts `dashboard` and `prometheus` from the
services.yml-to-env parity rule.
- `tests/unit/plugins/lookup/test_roles_with_service.py` adds 14
unit tests for the lookup, including an explicit-contract test
that pins the over-inclusive behaviour on raw Jinja strings (the
lookup expects a templar-rendered payload).
- Docs: `docs/requirements/019-playwright-meta-services-parity.md`
rewords Rules 1, 3, 12 and adds Rule 13 describing the SPOT shift.
`docs/contributing/artefact/files/role/playwright.specs.js.md`
reflects the new persona shape and per-service catalogue, and is
cleaned per `docs/contributing/documentation.md` (no SPOT self-
labels, no req-references, em dashes replaced with complete
sentences, multi-clause lines split onto separate lines, broken
anchor `#two-personas-fixed-flow-` corrected).
Verification
------------
- 418 passed / 1 skipped / 0 failed across `tests/lint/`,
`tests/integration/roles/`, `tests/unit/plugins/lookup/`.
- `make lint-javascript`: 0 errors.
- `markdownlint-cli2`: 0 errors on all touched docs.
- `make autoformat`: clean (FURB171 single-item-set inequality fixed).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a lint that fails when a role's templates/playwright.env.j2 declares
DASHBOARD_SERVICE_ENABLED= but does NOT render DASHBOARD_BASE_URL,
PROMETHEUS_BASE_URL, and MATOMO_BASE_URL. Renders those three URLs
across 42 role env templates so the lint passes.
Also reverts the matomo "role-closed" strikethrough in req 019 — the
URL wiring remains in scope for the persona rollout.
Reason: the persona helpers (roles/test-e2e-playwright/files/personas/
{biber,admin,guest}.js) collapse the persona scenario when their
required URLs are empty (req 019 persona-collapse exception). Without
this lint, a role that declares dashboard: enabled but forgets the
URL silently SKIPs the persona — the deploy goes green but no real
flow ran. Per-line opt-out is `# nocheck: persona-required-envs` for
auth-less svc-* roles.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wires runGuestFlow / runBiberFlow / runAdminFlow from
roles/test-e2e-playwright/files/personas/ into another ~28 roles
(akaunting, baserow, bluesky, bookwyrm, bridgy-fed, decidim, discourse,
fider, friendica, fusiondirectory, gitea, hugo, jenkins, joomla, kix,
matrix, mattermost, mig, moodle, odoo, opencloud, peertube, pixelfed,
postmarks, sphinx, taiga, yourls, web-svc-libretranslate, web-svc-xmpp).
Each role gets the spec.js persona scenarios plus matching
meta/services.yml gate entries and playwright.env.j2 env wiring;
admin-flow callers pass a role-specific adminInteraction that drives
an actual management surface so the body is not a stub.
Reason: continuation of d3e5ebe44 — req 019 Rule 3 demands every
role exercise the canonical guest/biber/administrator flows; this
batch closes the long-tail role list against the central library so
per-role drift cannot reintroduce the stub-shaped bodies the new lint
forbids.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh:
- rename run_sort_claude_settings worker to run_claude (matches the
renamed scripts/lint/claude.sh).
- drop the EXIT trap that wiped status_dir; cleanup happens explicitly
after `wait` returns, so a parallel worker that interrupts the
parent never races a vanished status dir.
- each worker now captures its tool's exit code and writes one of
OK / SKIP / FAIL. Tolerated `expected violations remain`-style exits
(markdownlint-cli2 rc≤1, ansible-lint rc≤2) still count as OK; only
real failures count as FAIL.
- all workers run to completion regardless of individual tool failures
(so the user always sees the full picture); after the summary, the
script exits 1 if any worker reported FAIL.
* scripts/lint/sort_claude_settings.sh -> scripts/lint/claude.sh
(history preserved via git mv); update test_claude_settings_sorted.py
fix-hint accordingly.
Plus working-tree drift across roles/, group_vars/, inventories/,
requirements/, and .pre-commit-config.yaml accumulated from the
make-test setup phase regenerating generated artefacts during recent
test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh: rename AUTOFORMAT_PARALLEL to PARALLEL,
declare it at file start with default `true`; still env-overridable.
Reword the worker comment so the literal `# shellcheck` token no
longer appears at start-of-line and trips SC1072/SC1073.
* scripts/lint/galaxy.sh: drop the GALAXY_FATAL / GALAXY_NAMESPACE /
GALAXY_PARALLEL env vars (over-engineered for a single repo). Failure
is always fatal. NAMESPACE is an internal constant. Add the same
PARALLEL env var (default `true`, env-overridable); falsy demotes the
xargs fan-out to one worker.
* docs/contributing/environment/variables.md: collapse the four
AUTOFORMAT_*/GALAXY_* rows into one `PARALLEL` row covering both
consumers.
* docs/contributing/tools/make.md: refer to `PARALLEL=0` everywhere;
add the lint-galaxy parallelism note.
Plus working-tree drift across roles/, group_vars/, inventories/, and
tasks/ that accumulated from the make-test setup phase regenerating
generated artefacts during recent test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Makefile lint integration:
* scripts/install/lint.sh: replace the prior checkmake binary install with
ensure_mbake (mbake is pip-installable, parses our Makefile correctly
whereas checkmake bailed on .ONESHELL: / .SHELLFLAGS: / ifneq).
* scripts/lint/makefile.sh: new runner — `mbake format --check Makefile`
reports formatting drift and missing .PHONY declarations.
* scripts/lint/autoformat.sh: add a two-pass `mbake format Makefile`
(mbake's formatter is not strictly idempotent on the first pass).
* Makefile: split the multi-line `.PHONY:` block into 18 single-line
`.PHONY:` declarations so mbake's parser can validate it; add the
previously-missing `install-python-dev` and `build-dependency` PHONY
targets; add `lint-makefile` aggregate-and-direct target.
* .pre-commit-config.yaml: new make-lint-makefile hook scoped to
`^Makefile$`.
* .github/workflows/lint-makefile.yml: new CI workflow (pip-install mbake
+ make lint-makefile).
* .github/workflows/ci-orchestrator.yml: wire lint-makefile into
lint-gate.needs.
Misc:
* .gitignore: add roles/importer_result.json (tooling artefact written by
galaxy-importer when it runs in roles/).
Plus working-tree drift across roles/, group_vars/, inventories/, and
tasks/ that accumulated from the make-test setup phase regenerating
generated artefacts during recent test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tooling integration
* add .markdownlint-cli2.jsonc, .ansible-lint, .ansible-lint-ignore
configs (permissive day-1: cosmetic rules disabled, semantic ones on)
* extend scripts/install/lint.sh with generic install_pip_pkg helper +
ensure_ansible_lint / ensure_galaxy_importer / ensure_markdownlint_cli2
ensure_* functions (pip / npm with sudo / --user / --break-system-packages
fallbacks). install_ansible_tools now also pulls ansible-lint and
galaxy-importer; new `markdown` install group.
* extend scripts/lint/ansible.sh to run ansible-lint after the existing
syntax-check.
* add scripts/lint/markdown.sh and scripts/lint/galaxy.sh runners. The
galaxy runner parallelizes via xargs (-P=$(nproc)), prints OK/FAIL per
role, and treats `bare-name dependency` errors as success since our
roles/ tree is not a Galaxy collection layout. All other Galaxy schema
errors still fail.
* Makefile: new lint-markdown and lint-galaxy targets, both wired into
`make lint`.
* CI: new workflows lint-markdown.yml and lint-galaxy.yml; lint-ansible.yml
now installs ansible-core + ansible-lint; ci-orchestrator wires both
new lint workflows into lint-gate.
Galaxy schema fixes
* shorten the canonical `galaxy_info.company` block from three lines to
two ("Kevin Veen-Birkenbach\nhttps://www.veen.world", 45 chars) so it
stays under Galaxy's hard 50-char limit. Updated everywhere: 229 role
meta/main.yml plus _CANONICAL_COMPANY in
tests/lint/repository/test_role_meta_main_galaxy_schema.py.
* shorten 19 over-long role descriptions to <=255 chars.
* add stub READMEs for sys-svc-compose-ca and sys-utils-service-loader.
Markdown baseline fixes
* markdownlint-cli2 --fix run repo-wide (formatting normalization
across CHANGELOG, role READMEs, docs).
* manual MD041 fixes: prepend H1 in 15 files (CHANGELOG, LICENSE,
TODO.md helpers, role README templates, etc.).
* MD025 / MD003 / MD001 / MD032 manual fixes for heading-style and
list-spacing edge cases.
* role README content polish: "## Single Point of Truth (SPOT)" headings
renamed to "## User data resolution" (D3 self-label rule).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add the four mandatory H2 sections (Description / Overview / Features /
Credits) to every role README that was missing one. Strategy:
- prose between H1 and the first H2 is wrapped with `## Description`
- synonym headings rename to canonical (Goal/Goals/Use Case →
Overview, Contents/Tasks → Features)
- everything still missing is inserted at its canonical position with
content sourced from `galaxy_info.description`
* Resolve the 18 remaining documentation.md compat findings on the
modified set:
- em-dash separators replaced by parentheses (parenthetical
definitions), commas (clause separators), or colons (label/value)
- `Single Point of Truth (SPOT)` self-label headings renamed to
`User data resolution`
- one full-URL link text shortened to the project name
* tests/lint/repository/test_role_readme.py is now green across all
227 role READMEs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* meta/main.yml: replace every prior platforms variant (Linux,
GenericLinux, Any, Archlinux+rolling/all, distro-specific pins) with
the canonical 5-distro list (ArchLinux, Debian, EL, Fedora, Ubuntu;
versions: [all] each).
* lint: merge tests/lint/ansible/roles/test_meta_company_canonical.py
into tests/lint/repository/test_role_meta_main_galaxy_schema.py and
add a parallel canonical-platforms exact-value check; one file walk
enforces both schema conformance and the two project canonical
exact-values (company block-scalar + platforms 5-distro list).
* roles/<role>/README.md: strip emojis from H1/H2 headings (per
readme_md.md role rule), normalize the Credits block to the canonical
wording (Author/Authors/Author Information aliased; canonical block
appended where absent).
* repo-wide: revert text-drift `GenericLinux` -> `Linux` in role
description fields, README body prose, vars, j2 templates, and the
Makefile (87 occurrences). The lint docstring keeps a single
historical mention.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Collapse the prior ~10 variants of galaxy_info.company (single-line,
multi-line single-quoted, multi-line double-quoted, escaped \n,
Unicode non-breaking hyphen, Birchenbach typo, block-scalar) into one
canonical block-scalar form across every roles/*/meta/main.yml that
declares the field:
company: |
Kevin Veen-Birkenbach
Consulting & Coaching Solutions
https://www.veen.world
Add tests/lint/ansible/roles/test_meta_company_canonical.py so future
drift fails CI immediately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the missing `min_ansible_version: '2.9'` and a `platforms:`
block (single Linux entry, all versions) to 63 role meta/main.yml
files so the Galaxy-publishing recommended-field check in
tests/lint/repository/test_role_meta_main_galaxy_schema.py stops
warning. Four roles only needed `platforms`; the rest needed both.
Both fields are appended at the end of the existing `galaxy_info:`
mapping; the lint only checks presence and shape, not order.
Existing scalar values (multi-line `company:`, single-line
`description:` paragraphs, etc.) stay byte-for-byte identical:
ruamel.yaml round-trip mode with `width=10000` and the project's
`indent(mapping=2, sequence=4, offset=2)` preserves the original
formatting on every untouched key.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reason: ruff parses every `# noqa: <code>` marker as a flake8 directive
and warns on every non-flake8 rule key. Every catalog rule in
`docs/contributing/actions/testing/suppression.md` is project-specific
(kebab-case), so `# noqa:` markers fired the warning on every parse.
`# nocheck:` is the project synonym ruff ignores.
Scope:
- 41 `roles/*/meta/services.yml` + 1 task + 1 .cfg.j2 template: pure
noqa→nocheck swap for catalog rules. Mixed flake8 markers
(`# noqa: E402, X`) preserved verbatim.
- 5 production .py + 2 production scripts: same swap.
- All catalog-rule lints' docstrings updated; redundant "or # nocheck:"
parentheticals collapsed to a single nocheck form.
- `utils/annotations/suppress.py`: docstring rewritten to spell out
the new convention. Parser still accepts both keywords as synonyms.
- `docs/contributing/actions/testing/suppression.md`: convention
rewritten to mandate `nocheck:`; missing `domain-spot` and
`literal-protocol-lookup` rows added; "enforced by" pointer added.
New guard:
- `tests/lint/repository/test_noqa_only_ruff_codes.py`: meta-lint
failing any `# noqa: <code>` whose codes don't match `[A-Z]+\d+`
(real ruff/flake8 shape). Allowlists the impl, its unit test, the
catalog page, the direct-yaml lint, and itself.
- `tests/lint/repository/test_no_direct_yaml_calls.py`: hardened to
reject the `noqa:` form via inlined regex (no longer routed through
`utils.annotations.suppress`).
Tangential cleanups carried in this same commit:
- `utils/cache/yaml.py`, `utils/handler/yaml.py`: 6 `direct-yaml`
markers swapped.
- `utils/cache/base.py`: `from . import PROJECT_ROOT, ROLES_DIR` moved
up to the import block (E402).
- `cli/create/role/__init__.py`: reordered so PROJECT_ROOT is defined
before `__main__` is imported (also fixed a latent circular import).
- PROJECT_ROOT migration tail: 2 new `__init__.py` modules + 10 test
files swapped from `def repo_root()` to `from . import PROJECT_ROOT`.
- ~20 ruff auto-fix removals of unused `from pathlib import Path`.
Verification:
- `ruff check .` clean.
- 23 suppression-related tests pass.
- The new lint detects all three regression shapes (project rule alone,
mixed with real ruff code, lowercase-only).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two new integration lints under tests/integration/roles/meta/:
* test_variants_coverage.py — every services.yml flag declared via
the dynamic ``"{{ '<role>' in group_names }}"`` form MUST be
exercised by meta/variants.yml: at least one variant entry pins
it to literal ``true`` AND at least one pins it to ``false``.
Pairs may share variant entries — the test only checks the union
of all variants covers both polarities for every dynamic flag.
* test_services_no_self_reference.py — a role's own primary entity
(matching ``get_entity_name(role)``) MUST NOT use the dynamic form
for ``enabled`` / ``shared``. Both MUST be literal ``true``: the
role's own services.yml is loaded only when the role itself is in
group_names, so a ``'<self>' in group_names`` Jinja is tautological.
The dynamic-flag test now skips the role's own entity to avoid a
contradiction with this new guard.
Self-reference fixes (Jinja → literal ``true: true``) on 19 roles:
svc-ai-ollama, svc-db-{mariadb, openldap, postgres}, web-app-{
dashboard, discourse, keycloak, mailu, matomo, prometheus},
web-svc-{asset, cdn, collabora, coturn, file, libretranslate, logout,
onlyoffice, simpleicons}.
Variants coverage rollout (62 new + 17 merged meta/variants.yml):
* Roles without a pre-existing variants.yml get a new file with two
entries — one pinning every dynamic flag true, one pinning every
dynamic flag false. No ``- {}`` baseline (matrix runs only the
two coverage variants).
* Roles with a pre-existing variants.yml (akaunting/baserow/bluesky/
bookwyrm/flowise/fusiondirectory/jenkins/jira/joomla/keycloak/lam/
minio/phpmyadmin/postmarks/wordpress/web-svc-xmpp) keep their
matrix variants (OIDC vs LDAP, Single-Site vs Multisite, dev creds
+ TOTP-disabled realm, autologin, etc.); the all-true coverage is
deep-merged into variant 0 and the all-false coverage into
variant 1, so each role still ships exactly two matrix rounds.
* svc-ai-ollama keeps its single CI image-shrink variant; after the
self-reference fix it has no remaining dynamic flags so no extra
coverage entry is required.
* svc-db-{mariadb, openldap, postgres} drop variants.yml entirely
for the same reason.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace literal `enabled: true / shared: true` flags across every
roles/*/meta/services.yml with a Jinja `"{{ '<role>' in group_names }}"`
form so the per-role services map becomes the single source of truth
for shared-service co-deployment, on a per-host basis. Inventory build
already runs through the templar before reaching the static `is True`
check, so runtime behaviour is unchanged; the new form only opens up
static-analysis and CLI tooling to the group-aware semantics.
Mechanism + lint:
- New integration test tests/integration/roles/meta/test_services_
dynamic_flags.py enforces the dynamic shape per top-level service
entry. Registered service keys must reference the registry-resolved
provider role; covered keys (e.g. oauth2 -> web-app-keycloak via
the new `covers:` block) reference the covering role.
- Two suppression placements documented in docs/contributing/actions/
testing/suppression.md: per-flag inline `# nocheck: dynamic-flag`
for databases (`enabled: true` stays literal, `shared` resolves
dynamically) and block-level above the key for entries whose both
flags legitimately stay literal (`css`, `javascript`, role-local
`redis`, `recaptcha`, `hcaptcha`, role-local containers with
`shared: false`).
Registry + helpers:
- utils/service_registry.py: new `is_explicit_truth(value)` helper
accepts `True` literal OR a Jinja string carrying `in group_names`.
Adopted by `resolve_service_dependency_roles_from_config`,
`ServicesResolver._is_enabled_shared`, `test_web_app_email_
integration`, `test_prometheus`, `test_universal_logout`, and the
run_after-explicit guard so static analysis keeps resolving deps.
- New `covers:` field on a primary service entity declares
transitive equivalents (web-app-keycloak.covers: [oauth2]).
Exposed via `build_role_to_covered_keys` and
`build_covered_key_to_role`, replacing the previously hard-coded
`_TRANSITIVE_SERVICE_EQUIVALENTS` / `_SPECIAL_REFERENCE_ROLE` maps
in the run_after and dynamic-flag tests.
Bulk migration:
- All 80 roles/*/meta/services.yml files migrated to the dynamic
form. DB consumer keys (mariadb, postgres, ldap) keep
`enabled: true # nocheck: dynamic-flag` literal with a dynamic
`shared`. Block-nocheck applied to css/javascript/recaptcha/
hcaptcha and role-local containers (e.g. mailu's admin/imap/smtp/
antispam/antivirus/oletools/webdav/fetchmail/webmail/resolver/
front, nextcloud's talk).
- web-app-prometheus's primary entity now declares
`provides: prometheus` + dynamic flags so the registry recognises
it as a service provider. Consequence: tasks/01_core.yml split out
with `- include_tasks: utils/once/flag.yml` as the first task and
tasks/main.yml reduced to the standard run_once-guarded loader.
- Seven minimal no-login apps (web-app-hugo, -bridgy-fed, -chess,
-littlejs, -navigator, -mini-qr, -mig) gained an explicit
`logout: { enabled: false, shared: false }` opt-out (with
`# nocheck: dynamic-flag` block-marker) to satisfy the universal-
logout schema check.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds tests/lint/repository/test_domain_primary_spot.py which fails when a
file constructs a hostname via a literal `<sub>.{{ DOMAIN_PRIMARY }}` outside
the role's `meta/server.yml` (or `meta/variants.yml`) `domains.canonical` /
`domains.aliases` blocks. Every other reference MUST flow through the
project's lookup plugins so the canonical/aliases entries stay the SPOT.
Mechanics
- Regex matches `<word>.{{ DOMAIN_PRIMARY }}` patterns; wildcard fragments
(`.{{ DOMAIN_PRIMARY }}`, `*.{{ DOMAIN_PRIMARY }}`) and bare
`{{ DOMAIN_PRIMARY }}` references are intentionally NOT flagged
- meta/server.yml and meta/variants.yml: a state-machine line scanner
exempts list items and dict entries inside `canonical:` / `aliases:`
blocks (handles compact, expanded, list, and dict YAML styles)
- README.md is exempt; docs/ and files/ subtrees inside roles are skipped
- Per-line opt-out via `# noqa: domain-spot` (or `# nocheck: domain-spot`)
- File reads route through utils.cache.files.read_text and the helpers
themselves are wrapped in lru_cache so repeated path lookups during
the traversal pay the parse cost once
Repo cleanup
- 11 role meta/server.yml files now reference Keycloak, Collabora,
OnlyOffice, Nextcloud, and their own canonical hostnames through
`lookup('domain', '<role>')` (and `lookup('config', '<role>',
'server.domains.canonical.<key>')` for the dict-canonical roles
Matrix and Bluesky) instead of duplicating the SPOT
- `roles/web-app-mattermost/tasks/main.yml` keeps a single
`# noqa: domain-spot` on the legacy domain cleanup task; the
hardcoded historical hostname must persist there to remove stale
on-disk nginx configs from the rename
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the direct /oauth/gitlab/login navigation with a real click on
the "SSO with Infinito.Nexus" button injected by javascript.js.j2, so
the test now exercises (a) login-form mount, (b) MutationObserver
injection, (c) rendered href and (d) the OIDC round-trip in one path
— a button regression now fails before the OIDC code is reached.
Wait for #input_loginId to mount before querying the button (the
observer only fires once the form anchor exists; without the wait the
button query races mount and times out on fresh contexts).
Add dismissal for the "No thanks, I'll figure it out myself" Welcome
card and re-run dismissMattermostPopups after navigating to the DM
channel — opening a DM for the first time triggers a fresh onboarding
overlay that intercepts Enter-key submission of the post composer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
req-011 — descriptive role-level metadata
* Move logo / homepage / video / display out of
meta/main.yml.galaxy_info into a new project-owned
roles/<role>/meta/info.yml (file-root convention, no `info:` wrapper).
* 78 role meta/main.yml files migrated via ruamel round-trip
(key-order / quoting / inline-comments preserved). Roles with none
of the four fields are left untouched (no empty info.yml created).
* web-app-dashboard/lookup_plugins/docker_cards.py reads logo +
display from meta/info.yml; description / galaxy_tags continue to
come from galaxy_info (Galaxy-spec fields).
* docker_cards.py now routes ALL its file reads through the
process-wide caches: load_yaml_any for vars/main.yml,
meta/main.yml, meta/info.yml; read_text from utils.cache.files for
README.md. The orphan `import yaml` is gone.
New utils/cache/files (renamed from tests/utils/fs)
* The process-wide filesystem cache (read_text, iter_project_files,
iter_project_files_with_content, _all_project_files) is not
test-specific; it now lives under utils/cache/files.py next to the
other process-wide caches. 34 consumer imports rewritten across
tests/ and docs/. The empty tests/utils/ package is removed.
* New unit suite tests/unit/utils/cache/test_files.py — 12 cases
covering read_text, iter_project_files (skip-dirs, extensions,
exclude_tests, exclude_dirs, walk-cache invariants), the combined
iterator, non-UTF-8 silent skip, and _reset.
utils/cache redundancy cleanup
* applications.py + users.py + cli/create/inventory/services_disabler.py
drop their direct yaml.safe_load calls and their private
_load_yaml_mapping/_load_yaml_variant_list copies; everything routes
through utils.cache.yaml.load_yaml / load_yaml_any. The dead
utils.cache.base._load_yaml_mapping is removed.
* users.py:_load_store_users (tokens.yml read) likewise migrated.
* _load_yaml_variant_list moves out of base.py to applications.py
(its only consumer) and is renamed _load_variants_overrides — also
routes through the cached loader now.
* utils.cache.yaml._reset_cache_for_tests is renamed _reset (kept as
alias) for parity with the per-domain reset helpers.
* utils.cache.files grows a _reset() that clears its two LRU caches.
* utils.cache.__init__._reset_cache_for_tests now orchestrates ALL
six caches (applications, users, domains, base, yaml, files) so
tests get clean state across cache layers — needed because
applications/users now share the yaml cache and would otherwise
serve stale parses across reset boundaries.
* utils.cache.files imports PROJECT_ROOT from base instead of
redefining it.
* tests/unit/utils/cache/test_data.py + test_base.py: obsolete
TestLoadYamlMapping / TestLoadYamlVariantList classes deleted
(covered by test_yaml.py + test_variants.py).
* README.md updated for the new module list (files, yaml) and the
six-module orchestrator.
Galaxy schema lint
* tests/lint/repository/test_role_meta_main_galaxy_schema.py: split
galaxy_info field requirements into hard-required {author,
description, license} and soft-recommended {min_ansible_version,
platforms}. Soft-recommended -> emit a GitHub Actions ::warning::
annotation via utils.annotations.message.warning (titled
"meta/main.yml galaxy_info" with file= pointing at the offender),
no test failure. Rationale: min_ansible_version + platforms are
only enforced by Galaxy when *publishing*, which Infinito.Nexus
does not do.
* Galaxy lint goes green (was 226 offenders before req-011 +
license_url/repository/documentation strip; was 62 before this
relaxation; now 0 hard offenders, 62 soft warnings).
* roles/svc-opt-keyboard-color/meta/main.yml: backfill `license`
(only role missing the hard-required field).
New lint
* tests/lint/repository/test_role_meta_info.py — enforces meta/info.yml
file-root convention, the four-field allowlist (logo, homepage,
video, display), and shape (logo.class string, homepage/video
non-empty strings, display bool). Also fails if any
meta/main.yml.galaxy_info still carries one of the four migrated
fields.
Docs
* docs/contributing/design/services/layout.md: new section
"Descriptive Role-Level Metadata — meta/info.yml" with allowed-
fields table, optionality rule, materialised path, and consumer
note. File-Layout table grows the meta/info.yml row;
meta/main.yml row gets the explicit "no logo/homepage/video/display"
note. req-011 added to header and Related Pages.
* utils/cache/README.md: files.py + yaml.py rows added; six-module
reset orchestrator documented.
Validation
* Unit: 1573 passed.
* Lint: 55 passed.
* Integration: 161 passed (1 skipped) via pre-commit hook.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
These three sub-keys lived under `galaxy_info:` purely as a workaround
because Galaxy silently ignores unknown keys (same pattern as the
`run_after` / `lifecycle` migration in req-010). The new lint
`tests/lint/repository/test_role_meta_main_galaxy_schema.py` rejects
them, and no consumer in `plugins/`, `utils/`, `cli/`, `filter_plugins/`
or `roles/` reads any of them, so the strip is safe.
Round-trip via ruamel.yaml so the remaining `galaxy_info:` block keeps
its key order, quoting style and inline comments.
224 files touched. Remaining lint offenders after this commit are still
present (`logo`, `homepage`, `video`, plus a handful of roles missing
required Galaxy fields like `min_ansible_version` / `platforms` /
`license`); those are a separate decision and not folded into this
change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
