Modern MariaDB images (>=11) dropped the `mysql` symlink, so the
hardcoded `mysql` binary in the mariadb engine spec started failing
with rc=127 ("executable file not found") — visible e.g. in the
Pixelfed users-table check looping through its `until` retries.
Switch `_ENGINES["mariadb"]` to an ordered `binaries=("mariadb",
"mysql")`, try them in sequence, and only fall back when the OCI
runtime reports the exact binary we tried missing — engine-side
rc=127s (SQL errors etc.) are not masked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reason: the lookup only consulted services.ldap.enabled, so even with OIDC
fully disabled it would still return "sociallogin" (or "oidc_login" when
LDAP was on). Nextcloud's services.yml then enables the matching OIDC
plugin, which redirects to Keycloak — but the Keycloak client's
redirectUris correctly excludes apps with oidc.enabled=false, so the
handoff dies with "Invalid parameter: redirect_uri". Caused all 3
nextcloud variant-1 Playwright errors (`nextcloud login`,
`nextcloud talk admin settings`, `biber logs into nextcloud via OIDC`).
The lookup now short-circuits to "" when oidc.enabled is falsy, which
makes all three plugin gates (oidc_login, sociallogin, user_oidc) compare
false and 06_setup_plugin.yml's disable branch take over. Existing
flavor-selection paths (oidc.enabled=true) are unchanged. Adds two
regression tests covering the new short-circuit and updates the
`_apps` test helper to default oidc.enabled=true so existing
flavor-selection tests stay focused on the ldap/flavor axis.
Reason: an unrendered `{{ lookup('env', 'DOMAIN') | default(...) }}` value would
otherwise propagate verbatim into CURRENT_PLAY_DOMAINS / CURRENT_PLAY_REDIRECT_DOMAINS
and end up as an nginx `server_name` — emitting `{{`/`}}` into a vhost makes nginx
fail with `directive "server_name" is not terminated by ";"`, breaking all roles
that share the openresty validation (observed on web-svc-asset/-legal/-file).
The filter now eagerly resolves domain_primary via the existing
`render_ansible_strict` env-lookup fallback, and `_add_default_domain` fails
loudly if any Jinja marker still survives. Adds two regression tests covering
the env-lookup path and the fail-fast guard.
The bespoke `pixelfed oidc login (biber)` test always timed out at the
1200s Ansible-task wrapper because pixelfed's first-time OIDC user-
creation runs Laravel's `email:strict,filter_unicode,dns,spoof`
validator. Egulias `DNSCheckValidation` hard-rejects any TLD in its
RESERVED_DNS_TOP_LEVEL_NAMES list (`test`, `example`, `invalid`, …)
BEFORE any DNS lookup, so `biber@infinito.example` always fails and
Laravel auto-redirects `back()` to the Keycloak Referer — breaking the
SSO callback chain and never returning to pixelfed.
The fix mutates biber's Keycloak profile to `biber@infinito.nexus` for
the duration of this spec (real TLD, real MX) and restores the original
afterwards via the Keycloak admin API. Verified empirically against
the live stack: `pixelfed oidc login (biber)` was 1200s timeout, is
now 20s pass; the `afterAll` restorer puts `biber@infinito.example`
+ `emailVerified=true` back cleanly.
Also splits the spec into one file per test so each scenario stays
atomar and individually inspectable:
- `playwright.spec.js` orchestrator: `test.use` + lifecycle + `require()`
- `_shared.js` env, locator candidates, login/logout flow,
keycloak admin helpers, `beforeAll`/`afterAll`
- `test-oidc-login-biber.js`
- `test-oidc-login-administrator.js`
- `test-native-login-administrator.js` new — exercises pixelfed's
stock email+password `/login` form (no infinito-specific selectors),
gated to run only when OIDC is disabled
- `test-guest-persona.js`
- `test-biber-persona.js`
- `test-administrator-persona.js`
The deploy's existing `with_fileglob: "*.js"` copies the companions
into `tests/`; Playwright's testMatch only discovers `playwright.spec.js`
so the `test-*.js` modules are pure `require()` targets, not auto-
discovered specs.
Updates two lints to scan the whole `files/playwright/` directory so
both monolithic and split layouts pass:
- `tests/lint/ansible/roles/web-app/playwright/persona/test_naming.py`
aggregates persona titles across all sibling `.js` files.
- `tests/integration/roles/playwright/test_spec_env_gates.py`
globs `*.js` for service-gate helper calls instead of reading only
the main spec.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The `image` lookup inferred `role_id` from `role_name` when only
`service_name` was passed. Inference broke lazy re-evaluation: a default
like `HEALTH_CSP_IMAGE: "{{ lookup('image', 'csp-checker', 'ref') }}"`
defined in sys-ctl-hlth-csp got re-evaluated inside sys-service's
systemctl.service.j2 template (via `system_service_tpl_exec_start`),
where `role_name='sys-service'`. The lookup then searched
`roles/sys-service/meta/services.yml` for `csp-checker` and aborted with
`no image mapping found for role 'sys-service' service 'csp-checker'`.
CI run 25976825584 surfaced this on six apps (web-app-{bridgy-fed,chess,
mig,hugo,mini-qr,postmarks}) whose deploy reached sys-svc-webserver-core
outside the service-loader preload context, which is what actually
triggers the sys-ctl-hlth-csp -> sys-service include chain.
Make `role_id` mandatory in the plugin signature, remove `_infer_role_id`,
and update the three callsites that relied on inference
(sys-ctl-hlth-csp, sys-svc-container, test-e2e-playwright) to pass the
owning role id explicitly. A migration guard catches the removed
`(service_name, want)` form and emits a clear error instead of the
confusing "missing services.yml" file-not-found message.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Matrix-deploy orchestrator now pushes `VARIANT_INDEX` (0-based round
id) alongside the existing `ASYNC_ENABLED` extra var for both PASS 1
(sync) and PASS 2 (async) runs. Test-e2e-playwright's archive step
consumes both to derive a per-(variant, pass) destination so a
subsequent round cannot overwrite the previous one's reports.
- New `roles/test-e2e-playwright/files/archive_playwright_reports.sh`
moves the playwright-junit.xml / playwright-report/ / test-results/
outputs into `variant-<N>/<sync|async>/` under the role's reports
dir, idempotently (re-running the same tuple replaces rather than
fails). `run_one.yml` invokes the script via `ansible.builtin.script`
so the YAML stays a single 7-line task.
- `VARIANT_INDEX: 0` lives in `defaults/main.yml` to keep the variable
defined for dev / non-matrix invocations. Role defaults are not
considered SPOT, so the `| default(0)` belt-and-suspenders inside
the script invocation stays valid against the spot-default lint.
- Summary script now renders an 8-column table: Time | App | Variant |
Pass | Test | Status | Duration | Message. Per-test ISO timestamps
are derived from the JUnit `<testsuite timestamp>` attribute plus the
cumulative duration of preceding cases; file mtime is the fallback
when the attribute is absent. App / Variant / Pass are extracted
from the artefact path layout produced by the archive script.
- Matrix-deploy unit tests refreshed to expect the new
`extra_ansible_vars` shape; summary unit tests gain coverage for
`_to_iso_z`, `_parse_iso_timestamp`, the four supported path
layouts, and the file-mtime fallback (55 tests total).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- New `utils/github/playwright_summary.py` parses every
`playwright-junit.xml` under a given artefact directory and prints a
markdown table to stdout with one row per test case: App, Test,
Status (🟢 passed / 🔴 failed / 🔵 skipped), Duration (sub-second
precision), Message. Sub-second formatting keeps useful per-test
signal that integer rounding would erase. Edge cases (missing dir,
no junit files, all-unparseable) emit a friendly placeholder note
instead of crashing. Covered by 26 unit tests in
`tests/unit/utils/github/test_playwright_summary.py`.
- `test-deploy-server.yml` runs the script with `if: always()` after
the deploy step and pipes its output into `$GITHUB_STEP_SUMMARY`, so
every matrix job's run page surfaces a per-app test table without
needing to download artefacts.
- Playwright config adds the `github` reporter so failing tests also
emit inline `::error file=...,line=...::`-annotations directly on the
workflow run page and in PR diffs.
- New lint `tests/lint/filesystem/test_plugin_paths_files.py` enforces
that files under the Ansible plugin source directories declared in
`ansible.cfg` (`action_plugins`, `filter_plugins`, `lookup_plugins`,
`library`, `module_utils`) are Python source files (README.md
exempt). Paths are read live from `ansible.cfg` so a future config
edit automatically extends the lint's coverage. Counterpart to the
existing `test_scripts_folder_files.py` lint that enforces
`scripts/` contains only `.sh`.
Known limitation (pre-existing pipeline behaviour, not introduced by
this change): when a role runs through multiple matrix variants, the
container path `/var/lib/infinito/logs/test-e2e-playwright/<app_id>/`
has no variant component, so each variant overwrites the previous
variant's reports. The summary therefore reflects only the LAST
variant per role. A future change to namespace the reports per
variant (or copy them after each variant pass) will make the table
cover every variant without touching the script or workflow.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add MCR_REGISTRY constant, is_mcr / mcr_repo / fetch_mcr_tags helpers
mirroring the Docker Hub and GHCR pattern. Tags are fetched via the
Docker Registry V2 API with RFC 5988 Link-header pagination (anonymous
access; MCR is a public read mirror). Wire MCR into both the external
test_image_versions warning surface and the find_outdated_updates
orchestrator in utils.update.docker.
- Bump mcr.microsoft.com/playwright in test-e2e-playwright from
v1.59.1-noble to v1.60.0-noble — the update was surfaced by the new
MCR detection path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Matcher in web-app-prometheus's parametrized scrape test now also
reads `labels.app`. Native scrape jobs (set `app="web-app-<id>"` via
each role's `templates/prometheus.yml.j2` fragment) now match alongside
blackbox-healthz targets. A single iteration asserts up=1 for both
surfaces per consumer, so per-app native tests become redundant.
- Scenario I (`/metricz`) iterates `prometheusTargetRoles` and asserts
each consumer's `app="<role_id>"` label is present in the
lua-resty-prometheus output. Subsumes the per-app metricz-label spot
checks.
- Remove four redundant per-app prometheus tests from web-app-gitea and
web-app-mattermost playwright specs (metricz-label + native-scrape
up=1), drop the orphaned `PROMETHEUS_BASE_URL=` declarations from
their env.j2 files, and clean up `OIDC_ISSUER_URL=` in gitea's env.j2
(only used by the removed scenarios).
- Drop the dead `const prometheusBaseUrl` declarations plus their
eslint-disable shut-up comments from web-app-fider, web-app-odoo, and
web-app-taiga playwright specs, along with the matching
`PROMETHEUS_BASE_URL=` blocks in those roles' env.j2.
- Correct the persona-required-envs lint: only `MATOMO_BASE_URL` is
actually consumed (by `personas/utils/csp.js`). `DASHBOARD_BASE_URL`
and `PROMETHEUS_BASE_URL` were aspirationally required by a lint
whose docstring described persona-collapse behaviour that no persona
implements. Update `_REQUIRED_URLS` and the docstring, and drop the
unused `\!dashboardBaseUrl` token from `_ALLOWED_SKIP_GUARDS` in
test_strict_mode.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- New bidirectional contract test ensures `services.prometheus.native_metrics.enabled`
and `roles/<role>/templates/prometheus.yml.j2` stay consistent across the roles tree,
replacing per-role spot checks for gitea/mattermost/matrix. Document the `native-metrics`
suppression rule and apply it to web-app-mattermost (prepared-but-default-off fragment).
- Move every test in `tests/integration/meta/config/` into the per-domain layout under
`tests/integration/roles/meta/{services,server}`. Group prometheus tests under
`services/prometheus/`, repositories tests under `services/repositories/`, the
`server.csp` shape test under `server/`. Move `test_non_empty.py` one level up
(scans all meta/*.yml, not just services.yml).
- Delete `tests/integration/meta/spots/` — both files were only `assertIn` over file
contents (brittle, no behavioural coverage, knowledge belongs in code comments or
proper integration tests).
- Harden `test_project_root_import.py` to catch chained `Path(__file__)…parent.parent.…`
walks (the attribute-access twin of `parents[N]` that previously slipped through).
Refactor 6 surfaced sites to import canonical `PROJECT_ROOT` / `ROLES_DIR`, plus
delete the now-redundant `_find_roles_dir` / `_find_project_root` helpers in
`native_metrics_apps` lookup and `invokable_paths` filter plugin.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Decouples the CSS service concept from its underlying transport
(currently `web-svc-cdn`) so the flag becomes opt-in via inventory
group membership instead of staying literal-`true` everywhere.
- `roles/web-svc-css/`: new role. Marker / wrapper role that owns
the `css` entry in the service registry, declares `web-svc-cdn` as
the upstream that serves the actual bytes, and 301-redirects its
own canonical domain to the cdn domain so health probes return a
valid response.
- `roles/web-svc-cdn/meta/services.yml`: drop the `css` block — the
canonical owner is now web-svc-css; cdn keeps `javascript`.
- 51 consumer roles (web-app-* and web-svc-{coturn,file,html,
libretranslate,logout,onlyoffice}) migrated their `css:` block from
literal `{ enabled: true, shared: true }` to dynamic
`{ enabled: "{{ 'web-svc-css' in group_names }}", shared: ... }`,
with matching matrix coverage added to each role's
`meta/variants.yml` (true-pinning + false-pinning entries) so the
dynamic-flag variant lint stays green.
- 41 `# nocheck: playwright-service-flag` rationale strings updated
from the now-incorrect "canonical:cdn alias of static-asset
provider" to "css consumer, no Playwright surface (provider:
web-svc-css)".
Bundled refactor: `roles/web-svc-asset/tasks/01_core.yml` now
includes `web-opt-rdr-domains` directly with a single-entry mapping
to redirect its canonical domain to web-svc-file, instead of having
that wiring live in the global `tasks/stages/01_constructor.yml`
constructor. The `add_redirect_if_group` filter
(`plugins/filter/redirect_filters.py`) and its unit test are dropped
as dead code — both call sites are now role-local.
Dashboard's consumption of the new pattern is intentionally not part
of this commit and follows separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Same brittle substring guard as the dropped blackbox test, and stale
after the matomo flag in dashboard's playwright.env.j2 moved from
non-canonical MATOMO_ENABLED to canonical MATOMO_SERVICE_ENABLED
(gated by skipUnlessServiceEnabled('matomo')). The env-spec gate
parity is already enforced by
tests/integration/roles/playwright/test_spec_env_gates.py.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The test asserted on substring presence in the raw Jinja template
instead of rendered behavior, so it broke on any equivalent rewrite
(quote style, helper var, whitespace). Its only purpose was a
regression guard against hardcoded insecure_skip_verify: false; that
guard is not worth the brittleness.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- plugins/lookup/status.py: `list(str)` on `APPLICATIONS_WHITELIST` (CSV from `-e`) split it into chars, breaking every consumer of `.whitelist` / `.running` (web-app-roles when-clause skipped all apps, keycloak per-app LDAP mappers never created).
- test-e2e-playwright/discover_playwright_roles: tolerate Python list-repr strings from non-native-Jinja2 stringification of `"{{ lookup('status').groups }}"` — otherwise `TEST_E2E_PLAYWRIGHT_APPS` resolves to `[]` and no spec runs.
- Unit tests cover both regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- plugins/lookup/roles_with_service.py: resolve url.base via the tls
sublookup and return canonical_url alongside id/canonical_domain.
- web-app-matomo/files/playwright/playwright.spec.js: parameterised
matomo-tracker injection assertion per consumer over
MATOMO_TARGET_ROLES_JSON, using target.canonical_url.
- web-svc-logout: new files/playwright/playwright.spec.js + templates/
playwright.env.j2; checks universal-logout injection on every consumer
declared in LOGOUT_TARGET_ROLES_JSON. meta/services.yml gains nocheck
markers explaining why this provider has no user-facing surface to
assert on.
- tests/unit/plugins/lookup/test_roles_with_service.py: cover the new
canonical_url field via a stubbed tls sublookup.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The nextcloud LDAP failure in CI 25939428268 traced to a variant-merge
propagation gap: when meta/services.yml declares a flag as a literal
and meta/variants.yml overrides that same flag, the merged value does
not reliably reach downstream `lookup('config', ...)` consumers.
Plugin tasks then run against a provider that the round's inventory
does not contain (e.g. `occ ldap:search` -> "Lost connection to LDAP
server"). The workaround is to anchor the base value to a
`'<role>' in group_names` expression so the inventory itself decides
whether the consumer activates.
Same anti-pattern applied to nine other shared-LDAP consumers:
funkwhale, keycloak, kix, openproject, opentalk, snipe-it, suitecrm,
taiga, web-svc-xmpp. Each had `services.ldap.enabled: true # nocheck:
dynamic-flag` paired with a dynamic `shared`. Replaced with
`"{{ 'svc-db-openldap' in group_names }}"` for both flags.
Add a regression guard at
tests/integration/roles/meta/services/test_variants_no_static_override.py
that fails whenever meta/variants.yml overrides a first-level
`services.<key>.enabled` or `services.<key>.shared` flag while the
base in meta/services.yml is a static literal. Rule
`variants-static-override` is catalogued in the suppression doc.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`container inspect mariadb` is polymorphic and matched the same-named
external Docker network (created by sys-svc-compose for shared DBs)
instead of the container, producing "map has no entry for key 'State'"
template errors on every retry until the wait task timed out.
- plugins/lookup/database.py: add `network` field (dbtype when shared,
consumer entity otherwise) so roles get the DB network for both modes.
- roles/web-app-mediawiki/tasks/main.yml: switch to typesafe
`container container inspect` against the new MEDIAWIKI_DB_CONTAINER.
- roles/web-app-mediawiki/vars/main.yml: derive MEDIAWIKI_NETWORK_DATABASE
and MEDIAWIKI_DB_CONTAINER from lookup('database', ...).
- tests: cover the new `network` field for empty/local/shared cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The first reuse-kept deploy after introducing `database_query` failed
with `couldn't resolve module/action 'database_query'`: ansible only
searches for custom modules along the `library` config path, and the
module had landed under `plugins/modules/` (collection-style layout)
without that path being registered in ansible.cfg.
Realign with the repo's existing convention — `library/cert_check_exists.py`
and `library/docker_bridge_gateway.py` are already there — by relocating
the module to `library/database_query.py`. Make the search path explicit
in ansible.cfg (`library = ./library`) instead of relying on ansible's
implicit playbook-adjacent discovery, so the resolver stays predictable
across CLI / playbook / role-include entry points.
Move the unit suite into `tests/unit/library/` alongside the moved
module, fix the doc-string reference in the lint guard that points
callers at the canonical module location, and drop the now-empty
`tests/unit/plugins/modules/` package.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
aur.archlinux.org/yay.git is git-clonable but HTTP GET returns 404,
since AUR has no plain-HTTP browsing layer. Mark it with `# nocheck: url`.
Also flip the GitHub example URL in the integration-test docstring to
example.com/foo/bar.git so the reachability scan filters it via the
reserved-host list.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces controller-side `community.postgresql.postgresql_query` /
`community.mysql.mysql_query` calls with a project-local `database_query`
Ansible module that wraps `container exec -i <container> <psql|mysql>`
and pipes SQL via stdin. The module routes via the DB container's own
network so the same task works identically in shared mode (central
`svc-db-*` container) and non-shared mode (per-role `<entity>-database`
container) — the `container` field of `lookup('database', …)` is
shared-aware and the module just reads it. SQL payloads come from a
file (`query_file:`, preferred) or as a raw string (`query:`, escape
hatch). `named_args:` substitutes `%(name)s` placeholders with engine-
appropriate escaping (postgres single-quote-doubling, mariadb backslash
escapes), so role tasks stop string-interpolating Jinja into SQL.
New backends extend the module's `_ENGINES` dispatch table; everything
else is engine-agnostic. The module ships at `plugins/modules/` (Ansible
collection-style layout) — the shebang lint test learned to recognise
that path as a valid module location alongside the existing per-role
`library/` layout.
Two lint tests enforce the convention:
* `tests/lint/ansible/modules/test_no_raw_db_query_module_usage.py`
forbids `community.postgresql.postgresql_query` /
`community.mysql.mysql_query` in role tasks. Opt out per-line with
`# nocheck: database-query` (used by `svc-db-postgres/tasks/02_init.yml`
where the role manages its own DB and the `database` lookup would be
self-referential).
* `tests/lint/ansible/modules/test_no_raw_database_query_string.py`
forbids inline `query:` on `database_query:` tasks. Opt out with
`# nocheck: database-query-raw` (no opt-outs in the current tree —
every SQL payload reaches the module through a `.sql` file under
`files/sql/`).
Role migrations:
* `web-app-peertube` — 3 tasks migrated, 3 new `.sql` files,
`templates/upsert-oidc-plugin.sql.j2` deleted in favour of
`files/sql/upsert-oidc-plugin.sql` + `named_args:`.
* `web-app-openproject` — 3 tasks migrated, 3 new `.sql` files for
the LDAP-auth-source check/update/insert chain.
* `web-app-odoo` — 33 tasks migrated, 9 new `.sql` files for the
LDAP / admin / init-database flows; 17 pre-existing OIDC `.sql`
files renamed snake_case → kebab-case and relocated from
`files/oidc/` to `files/sql/oidc/` (via `git mv` so rename history
stays intact).
Unit tests cover every dispatch-table entry, the named_args escaper
(strings, ints, floats, bools, None, single-quote and backslash
injection vectors), and the password-redactor that hides
PGPASSWORD/MYSQL_PWD assignments from the module's returned `cmd`
field.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `test_semver_pinning.py` under `tests/lint/ansible/services/`.
Recursively walks every `roles/*/meta/services.yml`, surfaces a
warning for every `image:`+`version:` pair and every `repository:`+`ref:`
pair whose value is not semver-compatible. Output via the shared
annotations helper (GitHub Actions `::warning::` plus pytest's
`warnings.warn`), with a single trailing suppression hint when any
finding fires. Never fails the test session.
Suppression reuses the existing markers:
- `# nocheck: docker-version` for `version:` pins
- `# nocheck: repository-version` for `ref:` pins
Both accept same-line or line-above placement. The existing external
update checks (`utils/update/docker.suppressed_services` and
`utils/update/repository.suppressed_ref_lines`) are loosened from
strict `line-above` to the default `same-or-above` for consistent
marker semantics across all three consumers.
Updates `docs/contributing/actions/testing/suppression.md` so the
catalog lists both consumers per marker and adds the previously
missing `repository-version` row.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three classes of drift cleaned up across 85 .md files:
- Em-dashes: replaced ` — ` with the contextually correct separator
(colon for `term: explanation`, period to split clauses across
sentences/source lines, parentheses for asides, semicolon for
tight-linked continuations). Two intentional exceptions remain
in `docs/contributing/documentation.md` (the rule literal) and
inside a fenced ```sh block in `docs/contributing/actions/debugging/ci.md`.
- SPOT self-labels: rewrote "This is the SPOT for X" to describe
the subject directly ("This directory holds X" / "This file
describes X"). Cross-references to other artefacts' canonical
role remain unchanged.
- Heading-emoji position: moved leading-emoji headings (`# 🧩 Title`)
to trailing position (`# Title 🧩`) per the headline rule that
emojis MUST come after the title text.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Folds the parallel Docker-image and git-ref version-update plumbing
into a single subsystem so the auto-update workflow, the external
freshness tests, and the unit tests all share one module layout.
utils/
update/
base.py # semver primitives shared by both backends
docker.py # OCI registry lookups + version: line rewriter
repository.py # git ls-remote walker + ref: line rewriter
cli/contributing/update/
docker/ # python -m cli.contributing.update.docker
repository/ # python -m cli.contributing.update.repository
scripts/github/update/
image_versions.sh
repository_refs.sh
Tests mirror the same shape under tests/{unit,external}/update/{docker,repository}/.
Also widens the Docker bump scope to all roles (previously gated on
`web-*`) so non-web roles like sys-svc-container get bumped too;
pins Bookwyrm's python image at 3.11-bookworm via
`# nocheck: docker-version` with a TODO until upstream supports
Python >= 3.12.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `tests/external/repository/test_repository_versions.py` — the
git-source counterpart to `tests/external/docker/test_image_versions.py`.
Walks every role's `meta/services.yml` recursively, collects every
`(repository, ref)` pair (top-level entity, sub-entity, plugin map),
filters to semver-compatible refs, queries upstream via `git ls-remote
--tags`, and emits GitHub Actions `::warning::` annotations when a
newer semver tag exists. Always passes — outdated refs are warnings,
not hard failures, so the suite stays green when remotes are slow or
unreachable. Suppress per-entry with `# nocheck: repository-version`
on the line directly above the `ref:` key.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-locates every git-clone URL with its owning entity in the role's
`meta/services.yml` so the unified `image`/`version` ↔ `repository`/`ref`
naming convention covers secondary repos (plugins, build helpers,
bootstrap sources) just as cleanly as the primary entity source.
Migrations:
- sys-aur, pkgmgr (non-invokable bootstrap roles): `meta/services.yml`
gains `repository:`/`ref:` on the role's own entity; `vars/main.yml`
renders them via `lookup('config', role_name, …)` because these
roles never set `application_id`.
- web-app-{akaunting,bigbluebutton,bluesky,discourse,roulette-wheel,
taiga}: declare the secondary git URL on the most fitting entity
(`akaunting` itself for the docker-helper repo; `view` for bluesky's
social-app web client; `plugins.discourse-ldap-auth` for that
Discourse plugin; `oidc` for taiga's OIDC plugin). Each role gains
named `<ROLE>_REPOSITORY` / `<ROLE>_REF` constants in `vars/main.yml`
so tasks and templates stay readable instead of inlining a
`lookup('config', application_id, …)` string at every call site.
- svc-db-postgres `templates/Dockerfile.j2`: keep the `pgvector`
extension clone but mark the line with `{# nocheck: repository-declared #}`
since pgvector is an extension source, not the role's primary repo.
New integration test
`tests/integration/roles/meta/services/test_repository_declared_in_services.py`
walks every role's text files and fails if a literal `https://…/<…>.git`
URL is not declared as a `repository:` value somewhere in that role's
`meta/services.yml` (recursive — top-level entity, sub-entity, plugin
map). Legitimate exceptions opt out with `# nocheck: repository-declared`
on the same line or the preceding line; Jinja templates may use the
`{# … #}` form so a `\`-continued Dockerfile shell command stays valid.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Unify the source-repo half of the entity naming convention introduced
by `0cc37d0fd`. Every entity that declares `repository:` MUST now
also declare `ref:` (git ref — tag, branch, or commit SHA). The
overloaded `version:` (when used as git ref) and the alternative
`branch:` key are gone.
Roles migrated:
* web-app-bigbluebutton: `version: main` → `ref: main`
(consumed in tasks/01_core.yml as `docker_git_repository_version`).
* web-app-discourse: `version: main` → `ref: main`;
`DISCOURSE_VERSION` renamed to `DISCOURSE_REF`.
* web-app-bridgy-fed: `branch: main` → `ref: main`;
`BRIDGY_REPO_BRANCH` renamed to `BRIDGY_REF` across vars, the
Dockerfile ARG, and the compose build-args.
* web-app-postmarks: identical migration to bridgy-fed.
New guard `tests/integration/roles/meta/services/test_repository_requires_ref.py`
walks every `meta/services.yml` and fails when an entity declares
`repository` without `ref` (or vice versa), so the next role that
forks the source-build pattern cannot regress to either half-
declaration without `make test` catching it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drops the bespoke ``_emit`` that hand-rolled ``::warning file=...::``
directives written to ``sys.__stderr__`` and switches to the shared
``utils.annotations.message.warning`` helper so the PR-side annotation
shape matches every other lint:
::warning title=Deprecated pkgmgr role include,file=X,line=Y::body
Before, GitHub rendered the annotation with no title column and a body
that duplicated the location prefix (``X:Y: deprecated 'pkgmgr' ...
Line: name: pkgmgr-install``). After, the title column carries the
rule name, ``file=`` / ``line=`` carry the location, and the body
contains just the message plus the offending snippet in parentheses.
The local pytest warnings-summary line keeps its ``path:line: body``
shape via ``warnings.warn`` so terminal output stays scannable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Modules imported by tests run inside the same pytest process as the
tests themselves. Raw ``path.read_text(...)`` calls in those modules
bypass the cache the tests just populated. ``roles/`` stays excluded
(deployed scripts have no ``utils/`` on PYTHONPATH); ``utils/cache/``
stays excluded (cache implementation itself).
Five conversions to ``utils.cache.files.read_text``:
- ``utils/handler/yaml.py`` (YAML loader, read-only)
- ``utils/annotations/summarize.py`` (CI log scanner, with
``UnicodeDecodeError`` fallback replacing ``errors="replace"``)
- ``cli/meta/roles/applications/resolution/combined/__main__.py``
(single-shot CLI JSON input)
- ``cli/core/help.py`` (read_folder_title + read_folder_description;
same README is read many times across CLI help iteration)
Eight ``# nocheck: cache-read`` markers, each annotated inline:
- ``utils/claude/settings_sort.py`` (read then rewrite sorted)
- ``utils/docker/version_updater.py`` (read then rewrite config)
- ``cli/administration/deploy/development/network.py`` (tests mock
``Path.read_text`` directly; caching would bypass the mock)
- ``cli/administration/deploy/development/log_utils.py`` (append flow)
- ``cli/contributing/changelog/archive/changelog_md.py``,
``__main__.py``, ``package_mirror.py`` (file mutated in same flow)
Two callers refactored to variable extraction so ruff's 88-char wrap
keeps the ``# nocheck`` marker on the ``read_text(...)`` AST span line
(``cli/.../network.py`` and ``package_mirror.py:105``).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Widens the cache-read lint guard from ``tests/lint/`` to every ``.py``
under ``tests/``. Integration tests show the same compounding cost
pattern (one role's compose template / nginx conf / scrape fragment
read across many shape tests) that motivated the original lint for
lint/.
Converts 91 violations across 28 test files:
- **Cache (utils.cache.files.read_text)** for project-tracked reads
and single-read tempdir fixtures, e.g. test_prometheus.py (32x),
test_keycloak_import_flow.py (11x), test_dashboard_oidc.py (3x),
test_changelog_md.py (8x), test_package_mirror.py (8x), plus
test_logging.py / test_summarize.py / test_version_updater.py /
test_get_url_retry.py / test_pull_specific_host.py.
- **Cache + UnicodeDecodeError handler** for project-walk scanners
that previously used ``errors="replace"`` / ``errors="ignore"`` to
defend against non-UTF-8 files (compose volumes, password quoting,
template image, mailu dependency, URL reachability). Wraps each
read in ``try/except (OSError, UnicodeDecodeError): continue``,
which has the same observable effect (skip undecodable files).
- **Refactor away the binary heuristic** in test_no_raw_docker.py:
``read_bytes() + b"\x00" not in raw`` becomes
``read_text() / except UnicodeDecodeError``. Same files skipped,
with cache benefit.
- **Variable extraction** in test_changelog_md.py / test_package_mirror.py
for cases where ``self.assertEqual(path.read_text(...), expected)``
exceeded ruff's 88-char limit. Ruff was wrapping the marker onto a
line outside the AST span of the read_text call.
- **Compute-from-setup** for two test methods that previously read
``self.changelog`` twice (before/after) to verify a no-op:
``before`` is now derived from the known input, dropping the first
read.
Five ``# nocheck: cache-read`` markers remain, each annotated inline
with the reason: idempotency tests where the second read must hit
disk to catch non-idempotent regressions, and tests with intermediate
``write_text`` whose subsequent read would return stale cache data.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Counterpart to `test_env_keys_used.py` — that one catches dead env
declarations (env declared, spec never reads), this one catches the
inverse failure mode (spec reads, env never declares).
The kix `DASHBOARD_BASE_URL` regression from CI run 25868178119 made
every persona test crash on the preamble assertion
`expect(dashboardBaseUrl, "DASHBOARD_BASE_URL must be set").toBeTruthy()`.
The existing env→spec lint didn't catch it because the failure was
in the opposite direction. This test scans `roles/<role>/files/playwright.spec.js`,
extracts every `process.env.X` / `process.env["X"]` / `readEnv("X")`
reference, and fails loudly if X is not declared in the sibling
`roles/<role>/templates/playwright.env.j2`.
Scope is narrowed to ONLY `playwright.spec.js` — other JS files
under `<role>/files/` (e.g. `login-broker/server.js` in web-app-bluesky)
are runtime app code whose `process.env` reads come from compose env,
not the test contract.
Excludes Node.js / Playwright / test-runner runtime names
(`NODE_ENV`, `NODE_TLS_REJECT_UNAUTHORIZED`, `PLAYWRIGHT_KEEP_ALL`,
`PWDEBUG`, etc.) via `_RUNTIME_ENV_NAMES`.
Verified: passes against the current 44 specs that ship an env
template, fails with a precise diagnostic when `DASHBOARD_BASE_URL`
is removed from `roles/web-app-kix/templates/playwright.env.j2`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`3c1d5d821` solved the cross-round backbone-abriss problem by making
`include` equal the union of every variant's closure for every round.
Side-effect: every matrix round then ALSO deployed the full union, so
`08_deploy_performance.sh` saw `dashboard.infinito.example` still
answering HTTP 200 after a matomo-only `deploy-fresh-purged-apps` —
the deploy step was no longer matomo-only at all.
Split the two concerns instead:
- `include` (PlanEntry[3]) goes back to the per-round variant-closure.
Both `init`/`builder` (`--include` for the inventory bake) and
`deploy` (`--id` for ansible) consume it. Each round therefore
deploys ONLY its own variant-closure again — the pre-`3c1d5d821`
semantics.
- `purge_set` (PlanEntry[4], new) carries the union across all rounds.
`deploy.py` uses it for the inter-round wipe instead of the previous
round's include. Result: between rounds the WHOLE matrix footprint
is purged, not just the previous round's variant slice — a variant
pinning `services.<X>.shared: false` cannot leave its bundled
provider on the host for a sibling round that wires the same service
shared. The backbone-abriss `3c1d5d821` was hunting stays fixed:
whichever round needs the backbone re-deploys it immediately after
the wipe (purge runs at the start of each round, not the end).
`previous_round_include` tracking is gone. The first iteration skips
the purge (nothing to wipe yet), which also covers `total_rounds == 1`
and `--variant N` pinned single-folder modes cleanly.
Unrelated, but staged together because it is part of the same
investigation:
- `roles/dev-nix/templates/nix.conf.j2`: enable `keep-derivations` /
`keep-outputs`, so `nix-collect-garbage -d` (run by the aggressive
prune step) does not drop the build closure of pkgmgr's flake; this
was making every deploy re-fetch the build inputs from
cache.nixos.org and overshoot pkgmgr's update timeout.
- `roles/pkgmgr/tasks/01_core.yml`: raise the pkgmgr update timeout
from 600s to 1800s and lower the retry count from 6 to 3, so a slow
fetch on a single try no longer pushes total wallclock past three
hours.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the no-op `docker restart` recovery with `POST
/service/rest/v1/repositories/{name}/invalidate-cache` against each
apt-format proxy repo (Debian, Ubuntu, …) discovered dynamically
from Nexus' REST API. Container restart never re-validated cached
content; the new path makes Nexus actually re-fetch from upstream
on the next request and keeps repo definitions, blobstore, and
admin credentials intact.
Co-located changes:
- fresh-purged-app.sh: source scripts/meta/env/all.sh so the cache
override env (incl. INFINITO_PACKAGE_CACHE_FRONTEND_IP) is set
for the deploy CLI — without this, apt traffic bypasses Nexus.
- utils/nexus/list_apt_proxy_repos.py: stdin JSON → stdout name list,
exit 1 on malformed input / no matches; used by the bash helper
for discovery (parser lives outside the heredoc for testability).
- tests/unit/utils/nexus/test_list_apt_proxy_repos.py: 10 cases
covering filter rules, empty/invalid input, non-dict entries,
missing/empty names, and a realistic live payload.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `tests/lint/repository/test_pkgmgr_deprecated.py` and the matching
`pkgmgr-deprecated` rule in the suppression catalog. The check never
fails the suite; each unmarked `name: pkgmgr` / `name: pkgmgr-install`
line is surfaced as a `DeprecationWarning` locally and as a GitHub
Actions `::warning file=…,line=…::…` annotation in CI. The warning
message points at the available pip migrations (`kpmx`, `docoli`,
`git+infinito-nexus-core`). Per-line opt-out via
`# nocheck: pkgmgr-deprecated`.
Also compacts the suppression catalog page: drops the examples
section, replaces prose-style bullet lists with tables, shortens
catalog `Effect` cells, and aligns with `documentation.md` (no em
dashes, no soft-wraps inside sentences, link texts use filenames
instead of full paths).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tightens tests/lint/ansible/roles/test_compose_resource_limits in two
steps. First, the iteration scope is reduced from "every role with a
meta/services.yml" to "every role whose folder name starts with an
invokable-category prefix" (resolved via plugins.filter.invokable_paths.
get_invokable_paths). Non-invokable categories like sys-* never ship a
top-level compose service of their own, so leaving them out removes a
class of structural noise. Second, the warn-only sentinel
(`assertGreater(len(findings), 0)`) is replaced with a hard failure
that lists every missing key — a missing limit is now a build break.
Declares min_storage / cpus / mem_reservation / mem_limit / pids_limit
on the 48 invokable primary entities that previously lacked one or more
of the keys. Container roles (svc-db-{memcached,redis},
web-app-{bigbluebutton,matrix,oauth2-proxy}) get sizing tiers matched
to their workload: BBB heavy (4 cpus / 6g / 4096 pids), Matrix mid-high
(1.5 cpus / 2g / 2048 pids), redis / memcached / oauth2-proxy small.
Native-on-host roles (desk-*, drv-*, gen-*, svc-bkp-*, svc-net-*,
svc-opt-*, web-opt-rdr-*, web-svc-legal, web-app-socialhome) carry
approximate footprints of the underlying package(s); the keys do not
bind a container at runtime (no compose template includes them) but
document expected host consumption and satisfy the strict lint.
Pre-existing curated min_storage values stay (BBB's 50GB recording
budget, redis 1GB, oauth2-proxy 1GB, drv-intel 0GB, …) — only
placeholder shapes ("0GB" / "1GB" with quotes) get overwritten.
Restructures tests/lint/ansible/ into five concern-keyed subdirectories
(jinja/, modules/, services/, shell/, sql/) plus the pre-existing roles/
and variables/. Each new subdir ships its own __init__.py exporting
PROJECT_ROOT and a scope-focused README.md following the project's
documentation conventions (RFC 2119 keywords, emoji-after-headline,
one-sentence-per-line, no historical context).
12 existing tests move into their concern bucket via git mv so history
follows the file. Adds tests/lint/ansible/variables/test_dotted_path_keys_exist.py
which catches the bug class where a Jinja dotted path's head resolves
via group_vars/all/ or the consuming role's own vars/defaults but a
deeper segment does not exist as a key in the dict tree (e.g. the
previous `LDAP.SERVER.REACH_DOMAIN` typo in web-app-odoo).
Touches three external references so cross-links keep resolving:
- docs/contributing/actions/testing/suppression.md (run-once, shared)
- roles/sys-svc-compose/tasks/utils/network/README.md (network-via-util)
- tests/lint/ansible/sql/test_no_inline_multiline_sql.py (self-path exclude)
Adds tests/lint/ansible/test_no_jinja_default_on_spot_path.py: scans every
Jinja `{{ ... | default(...) }}` block under roles/, tasks/, group_vars/,
identifies the actual default target via a paren-aware left-walk from the
matched `|`, and fails when the target's dotted path resolves cleanly
against any group_vars/all/*.yml dict tree. Role vars/defaults are
intentionally out of scope (Ansible loads them only when the owning role
is in the play, so cross-role defaulting is a legitimate caller guard,
not SPOT decoupling). Per-line opt-out: `# nocheck: spot-default`.
Fixes the 21 violations the test surfaces:
- group_vars/all/00_general.yml: drop `| default('')` on CAPTCHA self-checks
- group_vars/all/02_tls.yml: drop `| default(...)` on DOMAIN_PRIMARY in CA subject
- roles/web-app-{listmonk,keycloak,espocrm}/meta/schema.yml: drop defaults on CAPTCHA.*
- roles/web-app-suitecrm/templates/env.j2: map empty SECURITY -> 'none' via `or`,
not via the default filter (SPOT value `""` is always set)
- roles/web-app-opentalk/tasks/main.yml: drop default on MODE_ASSERT (always bool)
- roles/user-workstation/tasks/01_core.yml: drop default on MASK_CREDENTIALS_IN_LOGS
- roles/test-e2e-playwright/tasks/run_one.yml: drop defaults on TLS_MODE + CA_TRUST.*
Also corrects the dotted-path bugs in roles/web-app-odoo/vars/main.yml that
referenced non-existent LDAP keys (`LDAP.SERVER.REACH_DOMAIN`,
`LDAP.USERS.SERVICE_ACCOUNT.*`, `LDAP.ATTRIBUTES.ID`, `LDAP.SERVER.SSL`) to
the canonical paths actually defined in group_vars/all/12_ldap.yml.
Two follow-ups after c7bacbab5 removed `inventory/variants.py`:
* docs/contributing/design/variants.md and docs/requirements/020-*.md
still referenced the deleted submodule, breaking
`test_markdown_relative_links_resolve`. The design doc drops the
removed entries from the reference table; the postmortem turns the
historical mention into plaintext with a pointer to the removal
commit and the current package layout.
* tests/lint/repository/test_markdown_links.py and
test_project_root_import.py walked the worktree via
`iter_project_files`, which does NOT honour `.gitignore`. In the
`make exec` container path the markdown lint's primary
`git ls-files` codepath fails and falls back to that walker, surfacing
19 broken links inside vendored skill READMEs under `.agents/` and
`.claude/skills/` plus 2 `parents[N]` violations in installed skill
scripts. Both lints now use `iter_non_ignored_files` so gitignored
trees (skills install output, build artefacts) are skipped in every
environment.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Commit 16f323319 added a strict variant-only include rule + conflict
detector in `inventory/variants.py`, gated behind the presence of any
primary's `meta/variants.yml`. In practice the rule never paid off:
* The CI failure that motivated it (matomo's matrix transition leaving
a stale dashboard nginx vhost -> 502) is already addressed by the
nginx-purge primitive shipped in commit bfd04e4a1.
* The existing `CombinedResolver` path already implements the "any
role says true or `in group_names` -> pull" semantics via
`utils.roles.applications.services.registry.is_explicit_truth`, so
every needed service still gets pulled and services that all roles
mark False stay out of the include.
* The strict literal-equality conflict detector was a false-positive
factory: 79/80 variant-equipped primaries failed it against the
current repo state because roles legitimately disagree on `enabled`
flags for shared services.
This commit removes the entire variant-only branch and restores the
single-path planner. Round-index clamping for non-primary deps is left
exactly as before 16f323319 -- only the additions are reverted.
The file-split refactor from 16f323319 is intentionally kept; only the
`variants.py` submodule, the corresponding `test_variants.py`, and the
variant-only gate test class in `test_planner.py` are removed. The
`__init__.py` re-exports + `__all__` are pruned to match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
A set_fact run_once: true at role scope in svc-db-postgres/tasks/01_core.yml
did not survive the matrix-deploy round split: rounds whose include set did
not contain svc-db-postgres failed at template-render time with
'POSTGRES_ALLOWED_AVG_CONNECTIONS' is undefined.
- Hoist POSTGRES_VCPUS, POSTGRES_TOTAL_RAM_MB, POSTGRES_MAX_CONNECTIONS,
POSTGRES_ALLOWED_AVG_CONNECTIONS, POSTGRES_SUPERUSER_RESERVED_CONNECTIONS,
POSTGRES_SHARED_BUFFERS*, POSTGRES_WORK_MEM*, POSTGRES_MAINTENANCE_WORK_MEM*,
POSTGRES_DELAY to group_vars/all/17_resource.yml (lazy Jinja evaluation
per round).
- Drop the set_fact from tasks/01_core.yml; the value is now derived
globally on every play.
- Relocate the split_postgres_connections filter plugin from
roles/svc-db-postgres/filter_plugins/ to plugins/filter/ so it is always
loaded via the global filter_plugins = ./plugins/filter declaration in
ansible.cfg.
- Move the matching unit test from
tests/unit/roles/svc-db-postgres/filter_plugins/ to
tests/unit/plugins/filter/ to mirror the plugin's new location.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two changes ship together because the variant-only matrix-deploy logic
was written into the legacy 697-line inventory.py and then re-split:
* Source package (max 240 lines per submodule, well under the 300 budget):
- spec.py DevInventorySpec + PlanEntry data types
- payload.py _resolve_variant_payloads + _bake_overrides
- legacy_resolver.py CombinedResolver fallback (default-mode path)
- variants.py strict variant-only include resolution
- planner.py plan_dev_inventory_matrix + filter_plan_to_variant
- builder.py build_dev_inventory + build_dev_inventory_matrix
- __init__.py public-API re-exports + back-compat shims
* Test package mirrors the source 1:1 with a shared _fixtures.make_spec
to avoid duplicating the DevInventorySpec defaults across files.
* New variant-only matrix mode (gated on any primary shipping
meta/variants.yml): only services whose merged variant payload pins
both enabled and shared to literal True become inventory groups;
Jinja flags never qualify (group_names does not exist yet at build
time); run_after only orders the include, never expands it;
cross-primary disagreement on a (service-key, round) raises ValueError.
Default mode (no variants.yml on any primary) keeps the legacy
CombinedResolver path untouched.
* Docs in design/{inventory,variants}.md updated to point at the new
per-submodule paths; test_nginx_vhosts.py's Test classes now also
inherit directly from unittest.TestCase so the AST-level lint in
test_all_test_files_have_tests.py picks them up (transitive base via
NginxVhostsTestBase alone is not enough).
Coverage: 458 cli unit tests, 349 utils unit tests, 99 dev-deploy tests,
all green; ruff check clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Matrix-deploy variant transitions left per-domain vhost files behind:
when a service dropped out of a round's include set, sys-svc-compose's
docker prune removed its container but the vhost under
<NGINX_DIR>/conf.d/servers/{http,https}/<domain>.conf survived, making
the next round's HTTPS probe see 502 instead of TLS-error 000 (5 failed
test-development jobs on run 25705903504).
The inter-round purge in cli.administration.deploy.development.deploy
already invokes scripts/tests/deploy/local/purge/entity.sh with the
previous round's full include — the missing piece was an entity-keyed
nginx primitive next to db/compose/dir. Add it:
- utils.roles.entity_apps.apps_for_entity: reverse of get_entity_name,
walks roles/*/vars/main.yml to map entity -> [application_id]
- utils.cleanup.nginx_vhosts: resolves entity -> apps -> canonical+alias
domains (via build_applications_from_roles + iter_app_domains so it
stays ansible-free) and removes the matching vhost files
- scripts/container/purge/entity/nginx.sh: thin bash wrapper,
symmetric to db/compose/dir
- scripts/container/purge/apps.sh: drive nginx.sh inside the entity
loop, alongside db/compose/dir
Tests: 14 new unit tests covering the reverse-lookup edge cases and
the purge iterator (existing-only / aliases / multi-entity / no-op CLI).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds tests/lint/ansible/roles/meta/test_oauth2_contract.py — when a
role's meta/services.yml declares oauth2 with `enabled` not literal
false (Jinja-templated `"{{ 'web-app-keycloak' in group_names }}"`
counts as truthy), the test now requires:
* oauth2.origin.host (non-empty string)
* oauth2.origin.port (non-empty scalar)
* <entity>.ports.local.oauth2 (int)
These three fields are consumed at deploy time by
sys-stk-front-proxy and web-app-oauth2-proxy. Their absence
previously surfaced only as opaque `lookup('config', ...) failed`
errors mid-deploy — exactly the trap that hit web-app-shopware on the
Bundle 2 V0 deploy. web-app-shopware's services.yml is updated to
satisfy the new contract (oauth2: 16503 + origin: { host: web, port:
"8000" }).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add tests/lint/ansible/roles/test_compose_no_raw_image.py (strict mode):
every `image:` line in roles/<role>/templates/compose.yml.j2 must
resolve through a Jinja substitution so the underlying registry /
repository / tag is declared in the role's meta/services.yml instead of
being pinned in the compose template.
Fix the 8 pre-existing violations the lint surfaces:
- web-svc-coturn: gateway nginx → services.gateway.{image,version}
- web-svc-simpleicons: local build tag → services.application.{image,version}
- web-app-mailu: clamav antivirus → services.antivirus.{image,version}
- web-app-taiga (5×): events, async-rabbitmq, events-rabbitmq, protected,
gateway → top-level services.<name>.{image,version}
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Filter no longer takes a `database_volume=` parameter. The volume name is
derived from the central provider's `services.<dbtype>.name` (shared) or
`<entity>_database` (dedicated) — mirroring `plugins/lookup/database.py`'s
`volume_prefix + host` construction. Simultaneous postgres + mariadb on
the same role now raises an AnsibleFilterError with a clear message;
support is deferred per the docstring TODO.
The integration test (renamed from `test_shared_service_volumes.py` to
`test_compose_volumes_call_required.py`) now requires every
`roles/*/templates/compose.yml.j2` to call `compose_volumes`, so the
top-level `volumes:` block is always rendered. Without it, flipping a
shared service to dedicated mode adds a named-volume reference that
Compose rejects with `service "<name>" refers to undefined volume`.
Adds the call to 14 templates that were missing it (dashboard, hugo,
sphinx, bridgy-fed, fediwall, littlejs, mig, mini-qr, navigator,
roulette-wheel, simpleicons, collabora, logout, openresty) and strips
the `database_volume=...` argument from every other role plus Taiga's
now-unused `TAIGA_DATABASE_VOLUME` var.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`lookup('database', <consumer>, 'image')` previously returned the bare
dbtype (`postgres`), so an embedded dedicated database
(`postgres.shared=false`) rendered `image: postgres:<svc-db-postgres
version>` even when the central provider shipped a different image
(`svc-db-postgres` declares `postgis/postgis`). `compose pull` then
failed with `manifest unknown` because `postgres:17-3.5` does not
exist on Docker Hub. Look the image up from the central provider's
`services.<dbtype>.image` and fall back to the bare dbtype only when
the provider declares none, so dedicated instances ship the same
image as the shared provider.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The env-keys lint previously matched only direct `process.env.<KEY>`
references. Persona helpers under roles/test-e2e-playwright/files/
personas/ read most env keys via a shared `readEnv("KEY")` wrapper, so
keys consumed only through the wrapper were flagged as unreferenced.
Extend the pattern so the wrapper counts as a real consumer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Splits the network-create helper out of the all-in-one wrapper so roles
can pick the right entry point:
utils/network/create.yml network-only, derives name + subnet from
`network_role_id` (use when the role still
needs to do work between the network create
and the compose-up, or when pre-creating
foreign networks in a loop).
utils/network/routine.yml former wrapper, now delegates to create.yml
and re-flushes sys-svc-compose right after
(use for self-contained single-container
services).
A new lint
`tests/lint/ansible/test_network_create_via_util.py` forbids ad-hoc
`community.docker.docker_network` calls or `docker network create` /
`container network create` shell-outs anywhere in `roles/` outside
`utils/network/create.yml`. The existing 5 callers (mariadb, postgres,
ollama, openldap, web-app-prometheus) move to the new layout and shed
the now-redundant `docker_network_name` / `docker_network_subnet`
caller-side vars; that retires 5 unused role constants
(MARIADB_NETWORK / MARIADB_SUBNET / POSTGRES_SUBNET / OLLAMA_NETWORK /
OPENLDAP_NETWORK).
The `web-app-prometheus` task gains a pre-create loop over
`lookup('native_metrics_apps')` so the deploy unblocks the bootstrap
deadlock observed on test-deploy-server CI run 25634428082 for
web-app-gitea and web-app-matrix (`network <X> declared as external,
but could not be found`): prometheus is loaded as a shared service
*before* the target role, so its `external: true` references to the
target's network needed pre-creation to satisfy compose-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
