Adds a SPOT-aware lint that walks the repo via the cached
project-tree iterator (no `git` subprocess) and fails on any tracked
non-test Python file longer than 500 lines. Files under `tests/` are
intentionally excluded — coverage breadth there reflects feature
matrix, not design weight. A per-file `# nocheck` (or `#nocheck`)
marker in the first 30 lines opts the file out; the marker is
intentionally visible so the cost of carrying long modules is not
silently buried.
Gitignore-aware filtering moves to its own SPOT
-----------------------------------------------
The fnmatch-based `.gitignore` parser/matcher previously lived inline
in `tests/lint/repository/test_test_files_location.py` (and in
near-duplicates across other tests). Pull it into
`utils.cache.gitignore` as a single source of truth:
* `load_gitignore_patterns(root)` — cached pattern loader.
* `is_path_gitignored(rel_path, patterns)` — pure matcher.
* `iter_non_ignored_files(...)` — convenience iterator in
`utils.cache.files` that composes the cached project walk with the
matcher.
`utils.cache._reset_cache_for_tests` now resets the gitignore module
too, so test fixtures that mutate the tree see consistent state.
Refactor / opt-out for existing offenders
-----------------------------------------
Three production files exceeded the cap; addressed each:
* `roles/sys-svc-compose-ca/files/compose_ca.py` — opted out via
`# nocheck: file-size`. The script is delivered to target hosts
via `ansible.builtin.copy` and runs with only the stdlib; splitting
would require also delivering a package, which the role does not
do.
* `tasks/utils/migrate_meta_layout.py` — converted to a package
(`tasks/utils/migrate_meta_layout/`) with focused submodules:
- `__main__` — entrypoint + run-orchestration.
- `yaml_io` — load/dump/empty-dir/deep-merge helpers.
- `credentials` — schema-vs-config collision detection +
runtime-to-schema conversion.
- `role_builder` — per-role meta file emission.
- `indices` — networks + ports + relays index builders.
Run via `python3 -m tasks.utils.migrate_meta_layout`. Largest
submodule is `role_builder.py` at 250 lines.
* `cli/fix/move_unnecessary_dependencies/__main__.py` — split
alongside two new sibling modules in the same package:
- `analysis` — provider/consumer static analysis.
- `apply` — meta rewrite + guarded include-block prepending.
- `yaml_io` — ruamel round-trip helpers + role-tree discovery.
Largest submodule is `analysis.py` at 174 lines.
`make test` green (162 tests, 1 skipped); the new lint passes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Single SPOT for matrix-deploy topology
=======================================
Before this change two paths derived the deploy plan from different
sources and could disagree:
* Group inclusion (`--include` / `group_names`): driven by
`cli.meta.applications.resolution.combined.CombinedResolver`,
which read each role's `meta/services.yml` raw and ignored
`meta/variants.yml` entirely. A variant that flipped
`services.<X>.enabled` to a literal `true` did NOT pull `<X>`'s
provider role into the deploy plan.
* Inventory bake (`host_vars/applications.<app>.*`): driven by the
variant-aware payload merge in `cli.deploy.development.inventory`,
so the role saw the variant-merged services map at deploy time.
The split surfaced concretely for `web-app-wordpress` variant 0
(`services.discourse.enabled: true` in `meta/variants.yml`): the
inventory baked `discourse.enabled = true`, the wp-discourse plugin
task gate fired, and the install loop tried `docker exec discourse`
against a container the variant-blind resolver had never pulled.
This commit collapses both consumers onto the same per-round,
variant-merged services map. The variant-aware resolver (Phase 1+2)
is the additive plumbing, but the deploy/init wiring (Phase 3-5) is
what actually changes behaviour at runtime.
Variant-aware resolver (Phase 1+2)
----------------------------------
`role_introspection.load_shared_service_roles_for_app` accepts a
`services_override` kwarg; `CombinedResolver.__init__` accepts a
`{role_name: services_dict}` map and routes it through the loader.
A `CombinedResolver` instance is round-specific when overrides are
supplied — its per-role edge cache is variant-resolved, do NOT reuse
across rounds. The combined-resolver CLI gains
`--services-overrides <path>`.
`cli.deploy.development.deps.resolve_run_after` threads the
in-container path of the overrides JSON into the resolver
invocation.
Per-round include planner (Phase 3)
-----------------------------------
`cli.deploy.development.inventory.plan_dev_inventory_matrix` now
takes `primary_apps` (just the user-named apps) instead of a
pre-resolved `include` list and returns a 4-tuple per round
`(round_index, inv_dir, round_variants, include_R)`. Per round it:
1. Computes `round_variants` clamped to each primary's variant
count.
2. Builds variant-merged services overrides for every role with a
`meta/variants.yml`, picking that role's variant for the round
(clamped) — so transitive deps with their own variants follow
the round.
3. Runs `CombinedResolver(services_overrides=...)` per primary and
unions the result with the primary apps in user order.
4. Extends `round_variants` with the discovered deps' clamped
variant indices so the deploy step's per-round purge logic stays
accurate.
`build_dev_inventory_matrix` keeps its name but signature changes:
it now takes `primary_apps` and an optional `include_filter` for
SERVICES_DISABLED-style consumer filters.
Init / deploy wiring (Phase 4-5)
--------------------------------
`init.py` and `deploy.py` consume the per-round `include_R`
directly, drop the old `resolve_deploy_ids_for_apps` pre-resolution
step, and apply the SERVICES_DISABLED provider filter on the
include set so the inventory `--include` flag and the final ansible
deploy ids stay in lockstep.
Behaviour invariance for non-variant roles
------------------------------------------
Apps with no `meta/variants.yml`, or without a `services.<X>.enabled`
override on any variant, behave identically to before:
* The override map for those roles is empty — resolver falls
through to the disk-only path.
* Per-round `include_R` collapses to the same set every round.
* Inventory bake outputs the same `applications.<app>.*` shape.
The behaviour change is scoped to roles that legitimately use
variants to flip topology. In the current repo only
`web-app-wordpress` is affected (variant 0 pulls
`web-app-discourse` in; variant 1 leaves it out, exercising the
multisite-without-discourse path).
Test surface
------------
* `tests/unit/cli/deploy/development/test_inventory.py` updated
for the new planner signature, mocking
`_resolve_round_include` / `_build_services_overrides_for_round`
so the planner's tuple shape is asserted independently of the
live resolver.
* `tests/unit/cli/deploy/development/test_matrix_deploy.py` adds a
`_entry()` helper that defaults `include` to the keys of
`round_variants`, so existing fixtures stay terse — round 0 of
the deploy loop deploys whatever is in `include_R`, and the
variant-aware planner naturally emits the same set.
* `tests/unit/utils/cache/test_data.py` pins the planner contract
(`primary_apps=`, 4-tuple plan entries with `include_R`).
* `tests/utils/__init__.py` added so the
`tests.lint.filesystem.test_init_files` lint test passes
end-to-end (the directory existed in the tree without an
__init__.py and tripped the lint).
Verification
------------
* `make test` green (162 tests, 1 skipped) on the merged tree.
* `make deploy-fresh-kept-apps APPS="web-app-wordpress"
FULL_CYCLE=true SERVICES_DISABLED="matomo,email"` end-to-end:
round 0 (variant 0, single-site + Discourse integration) and
round 1 (variant 1, multisite) each ran PASS 1 (sync) and PASS 2
(async), all four passes with `failed=0 rescued=0`. Round 0's
per-round include contained `web-app-discourse`; round 1's did
not — exactly the variant-driven topology the resolver is
supposed to produce.
The unrelated Joomla CLI install race (CI run 25079786137 `Deploy
web-app-joomla`) is parked separately as infinito-nexus/core#209;
its candidate fix is in that issue body, not in this commit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`utils/service_registry.discover_role_services` requires `shared: true`
on the role's primary services entry to register the role as a
provider. The `discourse:` entry in
`roles/web-app-discourse/meta/services.yml` only carried image /
plugin / port metadata, so the registry never mapped the `discourse`
service key to `web-app-discourse`.
Effect of the miss: any consumer role that declared
`services.discourse.enabled: ...` (e.g. web-app-wordpress'
wp-discourse plugin gate) could not be resolved to
`web-app-discourse` by the deploy planner. Discourse was never
pulled into the inventory, the wp-discourse plugin task fired on a
host where no `discourse` container existed, and Ansible aborted
with `docker exec discourse … Error response from daemon: No such
container: discourse`. Reproduced on test-deploy-server matrix run
25079786137 (`Deploy web-app-wordpress`).
Add `shared: true` and the matching `enabled: true` so the registry
recognises the provider. The lint test
`tests/lint/ansible/roles/test_service_core_first_task_run_once.py`
flips on for shared-provider roles and requires
`utils/once/flag.yml` to be the first task in `tasks/01_core.yml`
(prevents recursion in dynamic service loading via load_app.yml when
a second load attempt evaluates the run_once flag before the role
sets it). Move the include up accordingly.
`make test` green after the change.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The orchestrator at scripts/tests/environment/00_orchestrator.sh
unsets GITHUB_ACTIONS to simulate local dev, which flips
Profile.registry_cache_active() to True and activates the `cache`
compose profile (registry-cache + package-cache) even on hosted
runners. Without explicit caps, the half-of-free heuristic in
scripts/meta/env/cache/{registry,package}.sh would claim ~7g+~7g on
the runner's small / partition (~14 GB free after free-disk-space
step), competing with /home/runner/work and other ephemera.
Override INFINITO_REGISTRY_CACHE_*/INFINITO_PACKAGE_CACHE_* in the
test-environment.yml job env so:
- Bind paths land on /mnt/cache/... — shares the 74 GB ephemeral
disk with /var/lib/docker (already at /mnt/docker) instead of
stressing /.
- Caps sized for in-run post-prune re-pulls: rpardini=8g (fits one
deploy's in-use image set), Nexus blobstore=4g (pip/npm/apt
artefacts), heap=1g/direct=1g (OSS minimum is enough for a
pull-through proxy). Leaves ~62 GB headroom on /mnt for Docker
layer storage.
The cache.sh scripts gate every default with `if [[ -z … ]]` /
`: \"\${VAR:=…}\"`, so pre-set values bypass the heuristics; local
dev behaviour is unchanged.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Multi-round / re-deploy scenarios that change MariaDB credentials
between deploys (matrix-deploy round 2 against the same persistent
volume; rotated vault) deadlocked the matomo install at
"Wait for Matomo container to be healthy". The persistent
`config.ini.php` still carries the previous round's credentials,
matomo loops on `Could not connect to the database: Database access
denied`, the healthcheck never goes green, and the play aborts
before `04_patch_database.yml` can run.
Reorder `roles/web-app-matomo/tasks/01_core.yml`:
- Replace the early wait-for-healthy with a lightweight
wait-for-running (`Running: true`) so subsequent tasks have a
container they can `docker exec` into without depending on a DB
connection.
- Bootstrap (first install) and `04_patch_database.yml`
(re-deploy) run as before, in the same mutually-exclusive shape
driven by `matomo_config_exists`.
- Move wait-for-healthy AFTER the post-patch `flush_handlers`,
so the API-config phase (`06_config.yml`) is gated by an
explicit healthcheck on the patched container — strictly
better than the previous flow, where the API phase implicitly
relied on the initial healthcheck "lasting" through the
patch-driven restart.
Verified end-to-end via `deploy-fresh-purged-apps APPS="…" FULL_CYCLE=true`
matrix run with both rounds and both PASS 1/PASS 2 cycles green
(`failed=0 rescued=0` per round/pass) including round 2 / variant 1
(multisite WordPress) which exercises the cred-sync-after-restart
path that previously deadlocked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tracks the promotion of 14 sub-beta web-* roles to lifecycle:
beta. Fixes the in-scope set, per-role OIDC/LDAP/RBAC integration
notes, the Playwright testing gate (oidc/ldap variants, multi-app
fresh deploy, capstone full-cycle), and the autonomous execution
procedure with a single final commit on feature/alpha-to-beta.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
phpldapadmin has no upstream OIDC/LDAP RBAC integration path that
fits the project's beta criteria, so it is moved off the linear
lifecycle axis onto the parallel `unsupported` tier alongside the
other proprietary/demo roles.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a dedicated lifecycle reference and back it with concrete role
metadata changes:
- `docs/contributing/design/services/lifecycle.md`: enumerate the
values `meta/services.yml.<entity>.lifecycle` MAY take and the
criteria each value commits the role to. Linear axis runs from
`planned` -> `pre-alpha` -> `alpha` -> `beta` -> `rc` -> `stable`
-> `maintenance` -> `deprecated` -> `eol`, with the inner five
forming the tested envelope (matrix-deploy + Playwright gate
applies). `beta` requires single sign-on against
web-app-keycloak / svc-db-openldap whenever the upstream software
ships an OIDC or LDAP adapter; `rc` additionally requires a live
burn-in deploy on infinito.nexus.
- Introduce a parallel `unsupported` tier off the linear axis for
roles the project ships without test or maintenance commitment
(proprietary products, demo prototypes). The role's `README.md`
MUST state the unsupported status explicitly.
- `docs/contributing/design/services/layout.md`: cross-link the new
lifecycle.md as the semantic counterpart to the on-disk shape
documented under `run_after` and `lifecycle`.
- Move web-app-confluence, web-app-jira, web-app-magento from
`alpha`/`planned` to `lifecycle: unsupported`. They are commercial
proprietary products / catalogue demonstrations the project does
not commit to maintain. Their meta entries flip to `unsupported`
to match.
- `tests/lint/repository/test_role_meta_layout.py`: extend
`ALLOWED_LIFECYCLES` to cover the full set introduced in
lifecycle.md (`rc`, `maintenance`, `eol`, `unsupported`) so the
lint guard accepts the documented values without false positives.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The shared `msmtp_curl.yml.j2` healthcheck template ran an msmtp test
send unconditionally and `&&`-chained it with the HTTP probe. Whenever
the role had no working SMTP host (operator opted out of the email
provider, mailu disabled via `SERVICES_DISABLED=email`, transient SMTP
outage), msmtp returned non-zero, the chain short-circuited, and the
container flapped permanently to `unhealthy` even though the HTTP
service itself was fine.
Wrap the msmtp branch in
`{% if (lookup('config', application_id, 'services.email.enabled', false) | bool) -%}`
so it disappears entirely from the rendered shell when email is not
declared as enabled. The HTTP probe stays the canonical liveness
signal; msmtp remains a once-per-container diagnostic ping when SMTP
is part of the role's deploy contract.
Affected consumers:
- web-app-wordpress: declares no `email:` block, so the lookup default
resolves to false and the gate suppresses msmtp. Live-verified after
iterC (fresh-purged FULL_CYCLE matrix): wordpress container reports
Health.Status=healthy with the rendered command
`curl -f http://localhost:80/ || exit 1` (msmtp branch absent).
- web-app-friendica: explicitly declares `email.enabled: true`, so
`{% if true -%}` is a no-op and the rendered output is byte-equivalent
to the previous template (Jinja whitespace-control with `-%}` only
trims the wrapper lines). Behaviour unchanged on the enabled side.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`tests/integration/roles/test_playwright_env_service_flags.py` enforces
that every `<SERVICE>_SERVICE_ENABLED=...` line in a role's
`templates/playwright.env.j2` resolves to the literal `"true"` or
`"false"` strings demanded by requirement 006. The heuristic previously
recognised only two shapes:
1. A ternary that ends with `'true'`/`'false'` literals.
2. The bare strings `true` / `false`.
Commit cac52b74a introduced a third equivalent form,
`{{ <bool> | string | lower }}`, which Jinja renders to "true"/"false"
when the operand is a Python boolean. The old regex did not match it,
so the test went red even though the render-output guarantee still
held.
Add a third recognised shape via a `\|\s*string\s*\|\s*lower\b` regex
and update the inline doc to spell out all three. Both forms remain
permitted in role templates; the compact one stays valid alongside the
explicit ternary.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `templates/playwright.env.j2` and `files/playwright.spec.js` so
the role is auto-discovered by `roles/test-e2e-playwright`. The suite
asserts two contracts the deploy MUST guarantee at this stage:
- the front page is served under the role's canonical domain with
TLS, HSTS is emitted, and the response code is `< 400`,
- the response carries an HTML content-type.
Body-text or canonical-landing-page assertions are intentionally
omitted: the role today serves a Dockerfile-built mig image whose
front page is a generic listing, so a strict text-content assertion
would be brittle. The deploy contract validated here matches what
the role actually guarantees today; a richer suite belongs in a
follow-up once the role ships a stable landing page.
Verified end-to-end: fresh-purged FULL_CYCLE matrix over the four
target apps (discourse + mig + sphinx + wordpress), both rounds,
both sync and async passes — mig 2/2 tests PASS in every PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `templates/playwright.env.j2` and `files/playwright.spec.js` so
the role is auto-discovered by `roles/test-e2e-playwright`. The suite
asserts two contracts the deploy MUST guarantee at this stage:
- the front page is served under the role's canonical domain with
TLS, HSTS is emitted, and the response code is `< 400`,
- the response carries an HTML content-type.
Theme/chrome assertions for Sphinx-rendered docs are intentionally
omitted: the deploy stage spins up an empty docs root (the Sphinx
build runs in a separate periodic job, not at deploy time), so the
front page resolves to an `Index of /` listing rather than a
Sphinx-themed page. A docs-build-aware suite belongs in a follow-up
once the build step is part of the role contract.
Verified end-to-end: fresh-purged FULL_CYCLE matrix over the four
target apps (discourse + mig + sphinx + wordpress), both rounds,
both sync and async passes — sphinx 2/2 tests PASS in every PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The shared `msmtp_curl.yml.j2` healthcheck template ran an msmtp test
send unconditionally and `&&`-chained it with the HTTP probe. Whenever
the role had no working SMTP host (operator opted out of the email
provider, mailu disabled via `SERVICES_DISABLED=email`, transient SMTP
outage), msmtp returned non-zero, the chain short-circuited, and the
container flapped permanently to `unhealthy` even though the HTTP
service itself was fine.
Wrap the msmtp branch in
`{% if (lookup('config', application_id, 'services.email.enabled', false) | bool) -%}`
so it disappears entirely from the rendered shell when email is not
declared as enabled. The HTTP probe stays the canonical liveness
signal; msmtp remains a once-per-container diagnostic ping when SMTP
is part of the role's deploy contract.
Affected consumers:
- web-app-wordpress: declares no `email:` block, so the lookup default
resolves to false and the gate suppresses msmtp. Live-verified after
iterC (fresh-purged FULL_CYCLE matrix): wordpress container reports
Health.Status=healthy with the rendered command
`curl -f http://localhost:80/ || exit 1` (msmtp branch absent).
- web-app-friendica: explicitly declares `email.enabled: true`, so
`{% if true -%}` is a no-op and the rendered output is byte-equivalent
to the previous template (Jinja whitespace-control with `-%}` only
trims the wrapper lines). Behaviour unchanged on the enabled side.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`tests/integration/roles/test_playwright_env_service_flags.py` enforces
that every `<SERVICE>_SERVICE_ENABLED=...` line in a role's
`templates/playwright.env.j2` resolves to the literal `"true"` or
`"false"` strings demanded by requirement 006. The heuristic previously
recognised only two shapes:
1. A ternary that ends with `'true'`/`'false'` literals.
2. The bare strings `true` / `false`.
Commit cac52b74a introduced a third equivalent form,
`{{ <bool> | string | lower }}`, which Jinja renders to "true"/"false"
when the operand is a Python boolean. The old regex did not match it,
so the test went red even though the render-output guarantee still
held.
Add a third recognised shape via a `\|\s*string\s*\|\s*lower\b` regex
and update the inline doc to spell out all three. Both forms remain
permitted in role templates; the compact one stays valid alongside the
explicit ternary.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `templates/playwright.env.j2` and `files/playwright.spec.js` so
the role is auto-discovered by `roles/test-e2e-playwright`. The suite
asserts two contracts the deploy MUST guarantee at this stage:
- the front page is served under the role's canonical domain with
TLS, HSTS is emitted, and the response code is `< 400`,
- the response carries an HTML content-type.
Body-text or canonical-landing-page assertions are intentionally
omitted: the role today serves a Dockerfile-built mig image whose
front page is a generic listing, so a strict text-content assertion
would be brittle. The deploy contract validated here matches what
the role actually guarantees today; a richer suite belongs in a
follow-up once the role ships a stable landing page.
Verified end-to-end: fresh-purged FULL_CYCLE matrix over the four
target apps (discourse + mig + sphinx + wordpress), both rounds,
both sync and async passes — mig 2/2 tests PASS in every PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `templates/playwright.env.j2` and `files/playwright.spec.js` so
the role is auto-discovered by `roles/test-e2e-playwright`. The suite
asserts two contracts the deploy MUST guarantee at this stage:
- the front page is served under the role's canonical domain with
TLS, HSTS is emitted, and the response code is `< 400`,
- the response carries an HTML content-type.
Theme/chrome assertions for Sphinx-rendered docs are intentionally
omitted: the deploy stage spins up an empty docs root (the Sphinx
build runs in a separate periodic job, not at deploy time), so the
front page resolves to an `Index of /` listing rather than a
Sphinx-themed page. A docs-build-aware suite belongs in a follow-up
once the build step is part of the role contract.
Verified end-to-end: fresh-purged FULL_CYCLE matrix over the four
target apps (discourse + mig + sphinx + wordpress), both rounds,
both sync and async passes — sphinx 2/2 tests PASS in every PASS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
shopware.com/en/ exceeds the test's 30-redirect limit when probed by
test_urls_reachable.py but is reachable interactively. Suppresses the
URL via the documented `<\!-- nocheck: url -->` marker.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Touches wordpress, nextcloud, yourls, and dashboard. Drops redundant
| default(false) where the value is contracted-set.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Multisite variant explicitly sets `services.discourse.enabled:
false` in `meta/variants.yml`, but the matrix-deploy init bakes a
single include set across rounds, so `web-app-discourse` stays in
`group_names` for both variants. The previous gate
`'web-app-discourse' in (group_names | default([]))` therefore
reported `DISCOURSE_SERVICE_ENABLED=true` on the multisite variant
too, the round-trip test ran instead of skipping, and the deploy
failed because the variant deliberately did not deploy Discourse.
Switch the gate to `lookup('config', application_id,
'services.discourse.enabled')` so the env reflects the role's
variant-resolved view, mirroring how `wp-discourse` plugin enable
is gated. Apply the same swap to `DISCOURSE_BASE_URL` so the
URL is empty on the disabled variant and any accidental access
fails fast instead of pointing at a non-existent host.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The plugin-enable lookups for `wp-discourse` and
`daggerhart-openid-connect-generic` previously used `application_id`
as the lookup target. That is a play-time variable that resolves to
the role currently running, not to the role that owns the
`meta/services.yml` file. When `applications` materialises during
another role's task (for example
`roles/web-app-dashboard/tasks/02_validate.yml:13` triggers
`get_merged_applications` to render the loop expression), the
templar renders every embedded lookup with that role's
`application_id`. The wordpress lookup then asks for
`services.discourse.enabled` on `web-app-dashboard`, which has no
`discourse` service, and the deploy aborts with
`The lookup plugin 'config' failed: Key 'discourse' not found`.
Pin the lookup app id to the literal `'web-app-wordpress'` so the
nested resolution always targets the role that actually owns the
config, mirroring the convention used by
`roles/web-app-gitea/meta/services.yml`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both `roles/web-app-mig/vars/main.yml` and
`roles/web-app-sphinx/vars/main.yml` resolved `container_port` via
`lookup('config', application_id, 'services.' ~ entity_name ~ '.port')`,
but neither role's `meta/services.yml` declares a flat `port` key —
they declare `services.<entity>.ports.{inter, local.http}`. Deploy on
arch crashed at compose-template render time with
`The lookup plugin 'config' failed: Key 'port' not found in dict`
(GHA universal-deploy run 25028370090 jobs 73306260785 + 73306260788).
Match the convention used by web-app-listmonk / -gitea / -shopware
and read `services.<entity>.ports.inter` instead.
The existing `tests/integration/lookups/test_config_paths.py`
literal-path scanner only captures the first quoted segment of a
`lookup('config', ...)` call, so `~`-concatenated paths (where a
Jinja variable expands inside the dotted path) were validated only
as wildcard prefixes. That contract is too loose: any role with a
top-level `services` dict satisfied
`'services.' ~ entity_name ~ '.port'`, even when the resolved path
did not exist for the specific role.
Add a `test_wildcard_paths` method that:
- captures the entire path expression in any
`lookup('config', <app>, <expr>)` whose `<expr>` contains `~`,
- parses the expression token by token, treating quoted strings as
literals and any bareword between `~` markers as `*`,
- joins the parts and splits on `.` into a wildcard path
(for example `'credentials.' ~ X ~ '_jwt_secret'` becomes
`credentials.*_jwt_secret`),
- resolves the role from the file's `roles/<role>/...` path when
the lookup's app argument is a Jinja variable,
- validates the wildcard path through the same fallback chain as
the literal-path test (application defaults -> users defaults
-> credentials in defaults / schema -> images presence), with
per-segment fnmatch globbing so partial-segment wildcards like
`*_jwt_secret` resolve correctly.
Verified: reverting the two role fixes makes the new test report
`'services.*.port' has no match in application defaults / schema for
role 'web-app-mig'` (and the same for sphinx) with file/line refs.
Restoring the fixes turns it green again.
Also drop one extra `..` segment from three relative links in
`docs/contributing/design/inventory.md` so the markdown lint check
passes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two design docs under docs/contributing/design/:
- variants.md (renamed from matrix-deploy.md): explains the
folder-per-round wrapper that consumes per-role meta/variants.yml
and iterates variants at deploy time. Headings normalised to
sentence-case; the "legacy unsuffixed inventory path" wording
rewritten to drop the historical-context phrasing forbidden by
documentation.md.
- inventory.md (new): explains how a single round's inventory is
assembled out of meta/services.yml plus the variant payloads,
which inclusion mechanisms exist (operator --apps, run_after walk,
shared-service auto-include via strict `is True`), what lives in
the inventory file vs. host_vars/<host>.yml, and when Jinja
actually gets rendered (only at play runtime, via
utils.cache.applications._render_with_templar). Worked example
documents the WordPress + Discourse wiring as a reference case.
Cross-links updated wherever they pointed at matrix-deploy.md or
its concept name as a file:
- docs/agents/action/iteration/role.md
- docs/contributing/actions/deploy.md
- docs/contributing/artefact/files/plugins/lookup/applications.md
- docs/contributing/artefact/files/role/variants.md
- scripts/tests/deploy/local/deploy/reuse-kept-app.sh (path comment)
Plain-text occurrences of "matrix-deploy" as a concept term inside
Python comments, log messages, and test docstrings stay as-is; only
file-link references are migrated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
services.yml:
- Add web-app-discourse to wordpress.run_after, so the wp-discourse
plugin install in tasks/plugins/wp-discourse.yml finds the discourse
container running when its `container exec discourse rake
api_key:create_master[...]` and friends fire.
- Make wp-discourse.enabled and daggerhart-openid-connect-generic.enabled
read their flag from services.{discourse,oidc}.enabled via
`lookup('config', application_id, ...)`, replacing the inline
conditional / hardcoded literal. Single source of truth per
shared-service flag, matching the pattern in
roles/web-app-gitea/meta/services.yml.
- Replace the literal `discourse.enabled: false` baseline with
`"{{ 'web-app-discourse' in group_names }}"`. The inventory-builder
ignores Jinja-string `enabled` values (strict `is True` check in
utils/service_registry.py:203), so this is a play-time fallback only:
if a variant does not override the flag and the operator still adds
web-app-discourse to the inventory groups, the conditional resolves
to true at runtime so the wp-discourse plugin enables itself.
variants.yml:
- Variant 0 (canonical Single-Site): set
`services.discourse.enabled: true` so the Service-Registry pulls
web-app-discourse in via the shared-service auto-include path and
the matrix-deploy round exercises the wp-discourse round-trip.
- Variant 1 (Multisite): set `services.discourse.enabled: false` so
the Multisite Playwright scenarios run isolated from the Discourse
edge cases.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Discourse must come up before Wordpress, not after: the wp-discourse
plugin install in roles/web-app-wordpress/tasks/plugins/wp-discourse.yml
runs `container exec discourse rake api_key:create_master[...]` and
related rails commands during the Wordpress play, so the Discourse
container has to exist by then. The previous declaration
`discourse.run_after: [web-app-wordpress]` produced the opposite order
and crashed the GHA universal-deploy with
"Error response from daemon: No such container: discourse" before the
Discourse role itself ever ran (job 73306260674).
The matching wordpress-side run_after addition (which now lists
web-app-discourse as a Wordpress prerequisite) lands in a separate
commit so the two halves of the order flip stay independently
revertable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cover all five default CI distributions (Archlinux, Debian, Ubuntu,
Fedora, CentOS) from a single role. tasks/main.yml dispatches via
ansible_facts['distribution'] to one of three task files grouped by
package-manager family — Debian and Ubuntu both use apt, Fedora and
CentOS both use dnf, so each pair shares a single file:
| distribution | task file | module |
|---------------------|------------------|----------------------------|
| Archlinux | arch.yml | community.general.pacman |
| Debian / Ubuntu | debian.yml | ansible.builtin.apt |
| Fedora / CentOS | fedora.yml | ansible.builtin.dnf |
| other | unsupported.yml | ansible.builtin.fail w/msg |
The dispatcher is wrapped in `when: run_once_update is not defined`
and finishes by calling tasks/utils/once/flag.yml so subsequent
invocations within one play are no-ops.
Migration impact: only one external caller exists
(tasks/stages/01_constructor.yml line 143, "update device") and is
migrated to the new name. The auto-generated tasks/groups/update-roles.yml
(prefix-matched on `update-*`) becomes empty; it had no consumers.
Closes the GHA universal-deploy `update-apt on arch` failure path,
since update-apt no longer exists as a standalone role and the new
dispatcher routes Arch to pacman.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Documentation MUST describe the current state of the subject only.
Phrases like "this replaces X", "previously X did Y", "formerly known
as X", "the legacy X", or "after the refactor" rot as soon as the next
refactor lands and confuse readers who arrive in the present. Anchor
historical context in commit messages, pull request descriptions, and
changelogs instead, where it is tied to a point in time.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes a pattern-match loophole I exploited earlier in this session:
the existing `Bash(git commit*)` ask-rule only matches commands that
literally start with `git commit ...`, so `git -c commit.gpgsign=false
commit ...` (top-level config override before the subcommand) silently
fell through to the `Bash(*)` allow rule. Adding `Bash(git -c *)` to
deny means *any* one-shot git config override is now hard-blocked,
removing the bypass entirely.
Symmetric with the existing `Bash(git -C *)` deny entry: lowercase -c
(config override) joins uppercase -C (working-dir override) in the
deny list. Both flags are sufficiently rare in regular use that
losing autonomous access to them is acceptable; the operator can
still run them outside the sandbox when needed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three orthogonal fixes that together turn the wp-discourse round-trip
test from a 10-min hang into a 1.1-min green run:
1. Body fill: keyboard.press("Tab") from the title block lands on
the WP block-editor command-palette trigger, so the subsequent
keyboard.type(...) ends up in the palette's search field instead
of the post body. Use titleBox.press("Enter") — the natural
Gutenberg flow that splits a new paragraph block below, focused.
2. wp-discourse Gutenberg layout: the plugin (v2.6.x) registers a
PluginSidebar (toolbar icon, separate sidebar), NOT a
PluginDocumentSettingPanel inside the post sidebar. Click the
"Discourse" toolbar toggle first, then look for the publish
checkbox. The checkbox itself is `<input
className="wpdc-publish-topic-checkbox">` with no aria-label,
no name, no id — match by className.
3. finally-cleanup time budget: Playwright counts the finally block
toward the test timeout. The trash-link click's
waitForLoadState("domcontentloaded") hangs on some
WP/Gutenberg combos because move-to-trash is an XHR, not a
navigation. Cap the WP cleanup at 60s with Promise.race + add
per-op timeouts; the assertions all complete in <2 min so the
60s cap leaves comfortable margin.
Also bump test.setTimeout to 600s for this single test — two browser
contexts + two OIDC logins + WP editor + Discourse polling +
post-status cleanup across 5 statuses needs more than the default
300s budget on cold caches; individual step timeouts (60s OIDC, 30s
editor expects, 60s snackbar, 60s Discourse poll) keep regressions
fast-failing.
Validated locally: reuse-kept VARIANT=0 wordpress, 7/7 tests pass in
1.1 min, role re-deploy failed=0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
First-time visits to the Gutenberg post editor show a "Welcome to the
editor" guide modal that hides the title textbox from the
accessibility tree. Scope the close to the dialog (its button only
carries aria-label="Close", which also matches unrelated toolbar
buttons via /close/i) with an Escape fallback.
Replace the brittle empty-block aria-label cascade for the body field
("Empty block" / "Type / to choose a block" / "Block: Paragraph" / …
churn across WP versions) with a single keyboard.press("Tab") from
the title — same path a real user takes, robust against future
aria-label renames.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the direct /oauth/gitlab/login navigation with a real click on
the "SSO with Infinito.Nexus" button injected by javascript.js.j2, so
the test now exercises (a) login-form mount, (b) MutationObserver
injection, (c) rendered href and (d) the OIDC round-trip in one path
— a button regression now fails before the OIDC code is reached.
Wait for #input_loginId to mount before querying the button (the
observer only fires once the form anchor exists; without the wait the
button query races mount and times out on fresh contexts).
Add dismissal for the "No thanks, I'll figure it out myself" Welcome
card and re-run dismissMattermostPopups after navigating to the DM
channel — opening a DM for the first time triggers a fresh onboarding
overlay that intercepts Enter-key submission of the post composer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`cli.deploy.development.init` plans matrix rounds with the FULL include
set (`deploy_ids` = primary apps + transitive run_after deps); deploy
was planning with `primary_app_ids` (just what the user named). When
the only multi-variant role was a transitive dep, init created
`<dir>-0` / `<dir>-1` while deploy looked at `<dir>` and the inventory
file silently went missing. Fix passes `deploy_ids` to
`plan_dev_inventory_matrix` so both stages agree on the round count.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings up the package-cache compose service alongside the existing
registry-cache, gated by the same `cache` profile. Bootstrap helper
(scripts/docker/cache/package.sh) auto-rotates the admin password from
the well-known default, accepts the Sonatype CE EULA (3.71+ blocks
artifact downloads with HTTP 403 until /v1/system/eula is POSTed —
disclaimer must be sent verbatim with its unicode quotes), enables
anonymous read, creates one default blobstore, and registers the MVP
proxy set: apt-debian, apt-ubuntu, pypi-proxy, npm-proxy, helm-bitnami,
raw-githubusercontent. The runner gets pip.conf / .npmrc /
sources.list snippets bind-mounted in (each defaulting to /dev/null
when the cache profile is inactive) so the deploy is byte-identical
with cache off.
Cache plumbing:
- scripts/meta/env/cache/{registry,package}.sh: per-cache env defaults,
hostname-derived admin password seed (uses `hostname` command, not
$HOSTNAME shell var, so the value is identical across interactive
shells, non-interactive subshells, and Python subprocesses).
- scripts/docker/cache/{registry-ca,package}.sh: container-side
bootstrap helpers, both moved under scripts/docker/cache/.
- scripts/system/cache/clean.sh + `make cache-clean`: wipes both
/var/cache/infinito/core/cache/{registry,package} dirs and stops
the cache containers.
- cli/deploy/development/compose.py: Profile gate exports the three
new INFINITO_PACKAGE_CACHE_* client-snippet paths and runs the
bootstrap helper after the package-cache reports healthy.
Compose service uses sonatype/nexus3:3.91.1-ubi pinned, runs as
root (UBI image refuses to chown a fresh bind-mount), and forces
NEXUS_SECURITY_RANDOMPASSWORD=false so the bootstrap can log in with
admin/admin123 before rotating; the random-password mode otherwise
auto-completes onboarding silently and leaves no readable secret on
disk for an automated bootstrap to pick up.
Tests: tests/unit/cli/deploy/development/test_compose.py grows two
new assertions (cache services listed in `up` args when local; build
tests pin CI=true to retain ci-only assertions).
Closes req-012.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds Sonatype Nexus 3 OSS as a second service under the existing
`cache` compose profile, complementing (not replacing) the rpardini
docker registry-cache. Covers apt / pypi / npm / helm / raw via
proxy repos. Mirrors the registry-cache wiring conventions: env
script under scripts/meta/env/, strict ${VAR:?…} contract, Profile
gate, /dev/null bind-mount fallback when the profile is inactive.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The body of registry_cache.sh is already idempotent: every assignment
uses `: "${VAR:=default}"`, the `df`-based size calculation is gated
on `[[ -z "${INFINITO_REGISTRY_CACHE_MAX_SIZE:-}" ]]`, and re-sourcing
re-evaluates the same values to the same result. The
INFINITO_ENV_REGISTRY_CACHE_LOADED guard was pattern-mimicry from
defaults.sh / runtime.sh / etc., where it serves a real purpose
(skipping non-trivial init blocks). Here it skipped a no-op and just
added noise to the env-var namespace.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The three INFINITO_REGISTRY_CACHE_* variables previously carried their
defaults in compose.yml as `${VAR:-...}`, which baked /tmp paths and
a static 32g cache size into the YAML. Move the defaults into a new
shell module that is sourced via `scripts/meta/env/all.sh` and tighten
compose.yml to require the variables strictly via `${VAR:?...}`.
Why
* Adapt cache size to the host. Hard-coding 32g hurts hosts where
`/tmp` is a small tmpfs (typical Linux: ~half of RAM); the cache
would either be capped above the partition or starve other tmpfs
users. Computing the default as half the free disk space at the
cache path leaves a usable budget for the rest of the system on
small hosts and grows on large ones.
* Single source of values. With defaults in shell, the compose YAML
no longer carries duplicated tunables; `compose.yml` just consumes
what the env layer provides. Same pattern as the existing
INFINITO_CONTAINER strict consumption.
Changes
* New scripts/meta/env/registry_cache.sh exports the three vars.
CACHE_MAX_SIZE = `df --output=avail -B1G "$path" / 2`, walking up to
the closest existing ancestor when the cache path itself does not
exist yet (it is created on first start by `bind:
create_host_path: true`). Minimum 1g, fallback 2g if `df` fails.
* scripts/meta/env/all.sh sources the new module so the Makefile-driven
flow picks it up automatically.
* compose.yml: three `${VAR:-default}` -> `${VAR:?Source scripts/meta/env/registry_cache.sh ...}`
so the absence of the env layer fails fast with a precise message.
* Stale block comment above the registry-cache service (CI-only,
/tmp-defaults, etc.) collapsed to a one-line pointer to the README,
per documentation.md "default to no comments / explain only the
non-obvious why".
* docs/contributing/environment/variables.md and
docs/contributing/artefact/files/compose.yml.md describe the new
defaults source and the half-free-disk computation.
Verified
`bash -c 'source scripts/meta/env/all.sh && docker compose --profile ci --profile cache config'`
resolves the three vars (CACHE_MAX_SIZE: 3g on a host with 7G free
in /tmp). Without sourcing, compose fails with the exact
"Source scripts/meta/env/registry_cache.sh ..." message.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move the registry-cache pull-through proxy out of the always-on `ci`
profile into its own `cache` profile, and select it based on the runtime
environment instead of statically.
Why
The proxy only earns its keep when image pulls amortize across many
runs on the same host. That holds on developer machines (cross-run
image dedup, persistent host disk) but not on GitHub-hosted runners
(fresh disk per job), where the proxy startup overhead is net loss
without the corresponding payoff.
How
* Compose: split `registry-cache` into its own `cache` profile;
`infinito.depends_on.registry-cache` gets `required: false` so Compose
does not auto-start the proxy via the dependency when the profile is
inactive; the systemd drop-in bind-mount source is gated by the new
`INFINITO_REGISTRY_CACHE_PROXY_CONF` env var with `/dev/null` default,
so dockerd loads an empty drop-in (no `HTTP_PROXY`, no `ExecStartPre`)
when the proxy is not running and pulls images directly.
* New `cli/deploy/development/profile.py` `Profile` class is the SPOT
for which compose profiles activate. KISS + SRP: `is_ci()` reads the
three CI signals (`GITHUB_ACTIONS`, `RUNNING_ON_GITHUB`, `CI`),
`registry_cache_active()` is its strict inverse, and `args()` returns
the matching `--profile ...` flag list.
* `Compose` and `down.py` consume the same `Profile` so up/down cannot
drift on which services are part of the stack.
* Documentation under `compose/registry-cache/README.md`,
`docs/contributing/artefact/files/compose.yml.md` and
`docs/contributing/environment/variables.md` describes the new toggle
and the `INFINITO_REGISTRY_CACHE_PROXY_CONF` env var. Pre-existing
documentation.md violations in those files (one SPOT self-label, one
em dash, two path-as-link-text instances) corrected in the same pass
per the fix-nearby rule.
* Tests: 14 new `Profile` unit tests in `test_profile.py` plus two
scenario tests in `test_compose.py` that lock down the
`--profile ci` (CI runner) vs `--profile ci --profile cache` (local)
argument shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
rpardini/docker-registry-proxy resolves upstream hosts like
mcr.microsoft.com via AAAA and tries to connect to the v6 address
first. On hosts without IPv6 outbound (developer laptops behind v4-
only NAT, CI runners with disabled IPv6), every cached pull through
the proxy fails with "502 Bad Gateway":
connect() to [2603:1061:f:101::10]:443 failed
(101: Network unreachable) while connecting to upstream
Set DISABLE_IPV6=true so the entrypoint appends `ipv6=off` to nginx's
resolver directive, forcing A-record fallback. Documented inline so
the next reader does not have to chase the rpardini entrypoint to
understand the intent.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cache key was just the absolute path, so every read after the first
returned a cached dict regardless of subsequent file rewrites. The
assumption "the file does not change while my process runs" held for
short-lived CLI tools but not for an Ansible play, where roles like
sys-token-store rewrite tokens.yml multiple times via the `copy:`
module (a path the in-process Python cache cannot observe). Two
seemingly unrelated regressions traced back to the same stale cache:
* sys-svc-mail's "Mailu was preloaded" assertion read users.no-reply
via a stale tokens.yml and aborted the play with `failed=1`.
* sys-front-inj-matomo disabled `inj_enabled.matomo` for every domain
whose Domain-Config-Phase ran after the matomo bootstrap had
persisted users.administrator.tokens['web-app-matomo'] but before
the cache could see the new mtime, dropping the matomo.js script
from those pages.
Switch the cache key to `(path, st_mtime_ns, st_size)` so an
out-of-band `copy:` write surfaces on the very next read in the same
process. Stale entries for the same path are evicted on miss to keep
the cache bounded under repeated rewrites. Per-hit cost adds two
stat() calls (sub-µs each on a warm inode); negligible compared to
the YAML parse it short-circuits.
Verified: dashboard "matomo tracking assets" Playwright test went
from failing in baseline to passing in iter2 of the role-meta-refactor
deploy loop.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduces a `registry-cache` service (rpardini/docker-registry-proxy) in
the `ci` profile that MITM-caches every outbound docker registry pull
made by `infinito`'s inner dockerd, drastically cutting bandwidth on
re-deploys, parallel CI shards, and local fresh-purges.
Wiring (all bind-mounted via compose.yml, no Dockerfile or entry.sh
staging, so edits hot-reload on next container start):
- CA bundle at /opt/registry-cache-ca (read-only),
- systemd drop-in at /etc/systemd/system/docker.service.d/registry-cache-proxy.conf,
- registry-cache-ca.sh at /usr/local/bin/registry-cache-ca.sh.
The drop-in registers registry-cache-ca.sh as ExecStartPre. Compose
gates `infinito` on `registry-cache` `service_healthy` so the CA is on
disk before dockerd boots; the install script supports both Debian
(`update-ca-certificates`) and Arch/Fedora (`update-ca-trust extract`)
trust stores, detected by command availability.
Also reorganizes compose/ into per-service subdirectories
(compose/coredns/, compose/registry-cache/) with README.md per directory
following the documentation guidelines, adds scripts/docker/README.md,
and nests the registry-cache host paths under /tmp/infinito/core/
(mirror/ for blobs, ca/ for the MITM CA bundle).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The req-008/009/010 refactor's mechanical migration left several gaps
the test suite could not catch because the test suite itself was
silently broken in spots. Audit-and-fix pass closes those.
1. Production bug: backup discovery filter
==========================================
roles/sys-ctl-bkp-docker-2-loc/filter_plugins/find_dock_val_by_bkp_entr.py
still read `app.compose.services` from the materialised application
payload. After req-008 the file root IS the services map (no
`compose.services` envelope), so the filter walked an empty dict and
silently returned `[]` for every backup discovery call. Rerouted to
the new `app.services` path. Test fixtures updated to match.
2. External test broken by source_file mismatch
===============================================
tests/external/docker/test_image_versions.py:61 filtered for
`ref.source_file == "config/main.yml"`, but
utils/docker/image/discovery emits `meta/services.yml` since req-008.
The filter rejected every candidate → `entries == []` →
`self.assertTrue(entries)` failed. Switched the filter to the new path.
3. Five lint/integration tests silently iterating empty dicts
=============================================================
The following tests still read `cfg.get("compose").get("services")`
(the legacy compose-wrapper path). After req-008 every meta/services.yml
file root IS the services map, so these tests have been iterating
empty dicts and quietly passing without checking anything:
* tests/lint/ansible/test_service_shared_consistency.py:72
* tests/integration/backups/test_enabled.py:33
* tests/integration/backups/test_no_stop_required.py:35
* tests/integration/docker/test_services_image_version_valid.py:28
* tests/integration/oauth2_oidc/test_mutual_exclusive.py:26
All five rerouted to the file-root convention. test_service_shared_consistency
immediately surfaced 24 missing `# noqa: shared` markers (see below).
4. Lost suppression markers (51 total)
======================================
The migration script (tasks/utils/migrate_meta_layout.py) used
ruamel.yaml round-trip which preserved most comments but lost
trailing-comment markers on inline list items (a known ruamel edge
case). Restored verbatim from main:
* 13 `# nocheck: url` markers on meta/server.yml CSP-whitelist entries
(web-app-bluesky, web-app-flowise, web-app-matomo, web-svc-onlyoffice).
These cover CDN root URLs that legitimately return 404 for the bare
domain (Google Fonts, Matomo CDN, BlueSky API endpoint).
* 1 `# nocheck: docker-version` on web-app-moodle/meta/services.yml
(LTS pin for OIDC compatibility).
* 20 `# noqa: shared` markers on meta/services.yml — only the markers
that are still NEEDED. Initial restoration mirrored all 24 markers
from main, but 4 of them turned out redundant against the current
shape: bigbluebutton/postgres, bookwyrm/postgres, discourse/postgres,
magento/mariadb already declare `shared:` (true or false), so the
lint rule does not fire and the marker is dead weight. Skipped them.
* 13 `# noqa: email` markers on meta/services.yml — every covered
`email:` block has `{enabled: false, shared: false}`, exactly the
pattern the email-integration lint expects to suppress.
5. Test-fixture cleanups for backwards-compat clarity
=====================================================
Several unit tests still wrapped synthetic application payloads in
`{"compose": {"services": {...}}}`. Harmless because the production
code paths under test were already migrated, but misleading for future
contributors. Reshaped to match the post-req-008 file-root
convention:
* tests/unit/plugins/filter/test_node_autosize.py
* tests/unit/plugins/filter/test_docker_service_enabled.py
* tests/unit/utils/manager/test_inventory_manager.py (5 occurrences)
* tests/unit/cli/deploy/development/test_inventory.py
* tests/unit/roles/sys-ctl-bkp-docker-2-loc/filter_plugins/test_find_dock_val_by_bkp_entr.py
Validation
==========
* Unit: 1572 passed.
* Lint: 56 passed.
* Integration: 161 passed (1 skipped).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New lint guard
==============
tests/lint/repository/test_no_direct_yaml_calls.py AST-walks the
production tree (utils/, cli/, plugins/, filter_plugins/,
lookup_plugins/, roles/, scripts/, tasks/) and flags any direct
`yaml.safe_load` / `yaml.safe_dump` / `yaml.load` / `yaml.dump` /
`yaml.safe_load_all` / `yaml.dump_all` call. Aliased imports are
tracked: `import yaml as Y` ⇒ `Y.safe_load(...)` is flagged;
`from yaml import safe_load` ⇒ bare `safe_load(...)` is flagged.
Per-line opt-out: `# noqa: direct-yaml` on the call's physical line
(or any line spanned by a multi-line call). No central exemption
list — every legitimate exception lives next to its call with a
human-readable rationale in the surrounding comment.
Cache module additions
======================
* `dump_yaml_str(data)` — symmetric to `safe_dump(data)` with no file
path. Used for stdout output and filter return values where the
cache cannot help but the API needs to stay consistent.
* `load_yaml_str(text)` — symmetric to `safe_load(text)`. Used when
YAML comes from an HTTP body, a mutated buffer, or a mock-patched
source whose content does not correspond to a stable on-disk file.
Migrated production call sites (33 → 0 direct-yaml violations)
==============================================================
* utils/{entity_name_utils, valid_deploy_id, invokable, service_registry,
domains/list, applications/{config, in_group_deps},
docker/image/discovery, meta/{port_bands, scan},
roles/{dependency_resolver, meta_lookup, resource_validation}}
* cli/{build/{graph, inventory/full/command}, create/inventory/{host_vars,
mirror_overrides}, fix/vars_main_files, meta/{applications/in_group_deps,
callorder}, mirror/resolver, validate/inventory}
* plugins/{filter/{compose_volumes, get_all_application_ids, get_role,
invokable_paths, role_path_by_app_id}, lookup/applications_current_play}
* roles/{svc-db-postgres/filter_plugins/split_postgres_connections,
web-app-bigbluebutton/filter_plugins/compose_mods,
web-app-dashboard/lookup_plugins/docker_cards}
Behaviour adjustments
=====================
* utils/roles/dependency_resolver — `safe_load_all` (multi-doc) →
single-doc `load_yaml_any` wrapped in a list. Ansible task files are
single-doc by convention; the downstream multi-doc loop stays intact
for forward compatibility.
* roles/sys-svc-compose-ca/files/compose_ca.py — KEEPS direct
yaml.safe_load / yaml.safe_dump (annotated with noqa). The script is
copied to the target host via `ansible.builtin.copy` and the target
has no `utils/` package on PYTHONPATH.
noqa-annotated direct-yaml call sites (with rationale)
======================================================
* utils/cache/yaml.py — IS the cache; would recurse.
* utils/handler/yaml.py — vault-aware Loader subclass that wraps
\!vault scalars in VaultScalar; cache-by-path collapses the wrapping.
* cli/encrypt/inventory/__main__.py — vault-aware Dumper subclass that
emits \!vault literal blocks; the cached dump_yaml writes plain YAML
which is the wrong output here.
* roles/sys-svc-compose-ca/files/compose_ca.py — runtime-deployed.
* tasks/utils/migrate_meta_layout.py — once-off migration script.
Validation
==========
* Unit tests: 1573 passed.
* Lint: 56 passed (incl. new test_no_direct_yaml_calls).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
req-011 — descriptive role-level metadata
* Move logo / homepage / video / display out of
meta/main.yml.galaxy_info into a new project-owned
roles/<role>/meta/info.yml (file-root convention, no `info:` wrapper).
* 78 role meta/main.yml files migrated via ruamel round-trip
(key-order / quoting / inline-comments preserved). Roles with none
of the four fields are left untouched (no empty info.yml created).
* web-app-dashboard/lookup_plugins/docker_cards.py reads logo +
display from meta/info.yml; description / galaxy_tags continue to
come from galaxy_info (Galaxy-spec fields).
* docker_cards.py now routes ALL its file reads through the
process-wide caches: load_yaml_any for vars/main.yml,
meta/main.yml, meta/info.yml; read_text from utils.cache.files for
README.md. The orphan `import yaml` is gone.
New utils/cache/files (renamed from tests/utils/fs)
* The process-wide filesystem cache (read_text, iter_project_files,
iter_project_files_with_content, _all_project_files) is not
test-specific; it now lives under utils/cache/files.py next to the
other process-wide caches. 34 consumer imports rewritten across
tests/ and docs/. The empty tests/utils/ package is removed.
* New unit suite tests/unit/utils/cache/test_files.py — 12 cases
covering read_text, iter_project_files (skip-dirs, extensions,
exclude_tests, exclude_dirs, walk-cache invariants), the combined
iterator, non-UTF-8 silent skip, and _reset.
utils/cache redundancy cleanup
* applications.py + users.py + cli/create/inventory/services_disabler.py
drop their direct yaml.safe_load calls and their private
_load_yaml_mapping/_load_yaml_variant_list copies; everything routes
through utils.cache.yaml.load_yaml / load_yaml_any. The dead
utils.cache.base._load_yaml_mapping is removed.
* users.py:_load_store_users (tokens.yml read) likewise migrated.
* _load_yaml_variant_list moves out of base.py to applications.py
(its only consumer) and is renamed _load_variants_overrides — also
routes through the cached loader now.
* utils.cache.yaml._reset_cache_for_tests is renamed _reset (kept as
alias) for parity with the per-domain reset helpers.
* utils.cache.files grows a _reset() that clears its two LRU caches.
* utils.cache.__init__._reset_cache_for_tests now orchestrates ALL
six caches (applications, users, domains, base, yaml, files) so
tests get clean state across cache layers — needed because
applications/users now share the yaml cache and would otherwise
serve stale parses across reset boundaries.
* utils.cache.files imports PROJECT_ROOT from base instead of
redefining it.
* tests/unit/utils/cache/test_data.py + test_base.py: obsolete
TestLoadYamlMapping / TestLoadYamlVariantList classes deleted
(covered by test_yaml.py + test_variants.py).
* README.md updated for the new module list (files, yaml) and the
six-module orchestrator.
Galaxy schema lint
* tests/lint/repository/test_role_meta_main_galaxy_schema.py: split
galaxy_info field requirements into hard-required {author,
description, license} and soft-recommended {min_ansible_version,
platforms}. Soft-recommended -> emit a GitHub Actions ::warning::
annotation via utils.annotations.message.warning (titled
"meta/main.yml galaxy_info" with file= pointing at the offender),
no test failure. Rationale: min_ansible_version + platforms are
only enforced by Galaxy when *publishing*, which Infinito.Nexus
does not do.
* Galaxy lint goes green (was 226 offenders before req-011 +
license_url/repository/documentation strip; was 62 before this
relaxation; now 0 hard offenders, 62 soft warnings).
* roles/svc-opt-keyboard-color/meta/main.yml: backfill `license`
(only role missing the hard-required field).
New lint
* tests/lint/repository/test_role_meta_info.py — enforces meta/info.yml
file-root convention, the four-field allowlist (logo, homepage,
video, display), and shape (logo.class string, homepage/video
non-empty strings, display bool). Also fails if any
meta/main.yml.galaxy_info still carries one of the four migrated
fields.
Docs
* docs/contributing/design/services/layout.md: new section
"Descriptive Role-Level Metadata — meta/info.yml" with allowed-
fields table, optionality rule, materialised path, and consumer
note. File-Layout table grows the meta/info.yml row;
meta/main.yml row gets the explicit "no logo/homepage/video/display"
note. req-011 added to header and Related Pages.
* utils/cache/README.md: files.py + yaml.py rows added; six-module
reset orchestrator documented.
Validation
* Unit: 1573 passed.
* Lint: 55 passed.
* Integration: 161 passed (1 skipped) via pre-commit hook.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Specifies the move of project-internal descriptive role-level fields
(`logo`, `homepage`, `video`, `display`) out of
`meta/main.yml.galaxy_info` into a new project-owned
`roles/<role>/meta/info.yml` (file-root, no `info:` wrapper).
Same anti-pattern that req-010 fixed for `run_after` / `lifecycle` and
that the strip in 0ab817a83 already handled for
`license_url` / `repository` / `documentation`. After this requirement
lands, the Galaxy-schema lint
(`tests/lint/repository/test_role_meta_main_galaxy_schema.py`) will
go green for every remaining offender.
Includes:
* target layout + file-root convention;
* known consumer (`web-app-dashboard/lookup_plugins/docker_cards.py`);
* acceptance criteria for file migration, schema rules, loader,
consumer rewrites, lint, docs, atomicity;
* validation app set + final iteration loop.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
