`web-opt-rdr-www` generates a redirect vhost
`<NGINX_DIR>/conf.d/servers/{http,https}/www.<domain>.conf` for every
deployed app's canonical and alias domains. Those redirect vhosts are
not part of any app's `server.domains.{canonical,aliases}` list, so the
entity-keyed purge primitive in `utils/cleanup/nginx_vhosts.py` did
not yield them and `iter_vhost_files_for_entity` left them on disk.
Between matrix-deploy rounds this leaked the redirect vhosts of the
previous round's broader app set into the next round's narrower one.
On run 26451828677 job 77879571696 (web-opt-rdr-www on debian) the
round 1 -> round 2 transition
round 1/2 inv=github-0 apps=[15 apps incl. web-svc-cdn,
web-svc-simpleicons, web-svc-css, ...]
round 2/2 inv=github-1 apps=[web-opt-rdr-www]
dropped the bare `cdn.infinito.example.conf` etc. via the existing
purge but kept the matching `www.cdn.infinito.example.conf`,
`www.icon.infinito.example.conf`, ... behind. Round 2 only redeploys
web-opt-rdr-www, whose `www_redirect_domains` lookup now only emits
its own `www.w3redirect.<primary>` redirect, so nothing reissues the
stale www-prefixed vhosts.
`sys-ctl-hlth-csp` discovers domains by scanning .conf filenames in
the nginx servers dir, so it still probed the stale `www.<x>.conf`
hosts. nginx served them with a cert whose SAN no longer covered the
hostname (the per-domain cert path baked into the conf had been
removed alongside the bare vhost), the TLS handshake returned
`net::ERR_SSL_UNRECOGNIZED_NAME_ALERT`, the CSP probe exited 11, and
`sys-service-terminator : Apply state` failed after 10 retries.
Extend `iter_vhost_files_for_entity` to also yield the
`www.<domain>.conf` variant for every domain it already yields,
skipping domains that already begin with `www.` so a role that
declares its canonical with a leading `www.` (web-opt-rdr-www's own
`www.w3redirect.<primary>`) does not produce `www.www.<x>.conf`. The
existing two- and three-step purge primitives (db / compose / dir /
nginx) and the orchestrator in `scripts/container/purge/apps.sh` keep
their entity-keyed contract; only the per-entity yield set widens.
Coverage:
- test_yields_www_redirect_variant: per-app www-prefixed file is
yielded alongside the bare vhost.
- test_www_prefixed_domain_not_double_prefixed: a www-prefixed
canonical does not synthesize `www.www.<x>` even when such a file
exists on disk (defense in depth against a future regression that
prepends unconditionally).
- test_removes_www_redirect_variant: end-to-end purge removes both
bare and www variant, leaves an unrelated www-prefixed file alone.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* TC003: move `pathlib.Path` into `TYPE_CHECKING` blocks across the
new `requirements/archive` package and its completeness lint test.
* TC001: same treatment for `VaultHandler` in
`credentials/emit.py` (annotation-only).
* SIM105/S110: replace `try/except/pass` in
`current_play_redirect_domains.py` with
`contextlib.suppress(Exception)`.
* `archive/__main__.py`: mark `README_PATH.read_text(...)` with
`# nocheck: cache-read` — CLI rewrites the README in the same flow,
mirroring the `changelog/archive` pattern.
* Drive-by ruff-format condensations in the archive unit tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* `python -m cli.contributing.requirements.archive` moves the H1 of
every fully-checked `docs/requirements/NNN-*.md` into the
`## Archive` block of the directory README and deletes the file.
Files with any remaining `- [ ]` are skipped. Split SRP-wise into
discovery / inspect / readme / __main__.
* New lint `test_requirements_completeness` warns per unchecked
acceptance criterion (GitHub annotation) and fails when any
requirement is ready for archival, pointing at the CLI. Keeps the
requirements directory small so AI agents process less context.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The plugin's previous
domain_primary = str(vars_.get("DOMAIN_PRIMARY") or "")
domain_homepage = str(vars_.get("DOMAIN_HOMEPAGE") or "")
read inventory values that may carry Ansible 2.18+'s
``_AnsibleTaggedStr`` with the ``TrustedAsTemplate`` tag — exactly
the shape ``inventories/development/default.yml:20`` produces with
DOMAIN_PRIMARY: "{{ lookup('env', 'INFINITO_DOMAIN') }}"
A bare ``str(value)`` casts the tag away and leaves the literal Jinja
text in the plugin output. The output then flows into
``CURRENT_PLAY_REDIRECT_DOMAINS`` via the set_fact in
``tasks/stages/01_constructor.yml`` and lands in
``roles/web-opt-rdr-domains/tasks/main.yml`` loop items as
item={'source': "{{ lookup('env','INFINITO_DOMAIN') }}", 'target': "{{ … }}"}
The ``lookup('tls', domain, 'enabled')`` call in
``roles/web-opt-rdr-domains/tasks/redirect-domain.yml:5`` then
receives the literal Jinja string and raises
The lookup plugin 'tls' failed: tls: application_id
'{{ lookup('env', 'INFINITO_DOMAIN') }}' not found in domains mapping
(job 77797371504 on run 26428080957). That CI ran on SHA d1410c0be,
which already shipped this plugin — so the upstream replacement of
the inline two-stage set_fact merges with the lookup plugin did not
fix the underlying trust-tag-strip bug; it just relocated it from
Jinja to Python.
Add ``_resolve_str`` that runs the value through ``self._templar``
before string-coercion. Idempotent for already-resolved strings,
exception-safe (falls back to ``str(raw)`` if the templar is
unavailable, e.g. on hosts without rendered vars). Both
``DOMAIN_PRIMARY`` and ``DOMAIN_HOMEPAGE`` route through it.
Defense-in-depth with the existing ``plugins/lookup/tls.py``
defensive templating (commit f32266d53): even if a future code path
hands ``lookup('tls', …)`` an untemplated string, the tls plugin
itself now templates the first term. Either layer alone suffices for
the rdr-domains failure; both together guarantee the symptom cannot
re-surface from a single regression elsewhere.
Coverage: two regression tests in
``tests/unit/plugins/lookup/test_current_play_redirect_domains.py``
— ``test_jinja_tagged_domain_primary_is_templated`` proves the
templar call happens and produces the resolved domain, and
``test_templar_failure_falls_back_to_raw_value`` proves a raising
templar does not crash the plugin. The 10 existing tests stay green.
``make test`` green across all 5 targets.
Class follow-up: 11 further lookup plugins read inventory values
without ``templar.template`` first (see git log for ``vars_.get``
call sites). Not in scope for this fix; logged for a separate
audit pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Six ``uri_retry`` tasks under ``roles/web-app-xwiki/tasks/`` carried
their XML payload as inline ``body: |`` blocks (CheckExtension Groovy
page, InstallExtensions Groovy page, the administrator user-page
PUT/POST pair, and the post-Fix-#8 XWikiAdminGroup PUT/POST pair).
Inline XML mixes Ansible YAML indentation with Jinja control flow
(``{% raw %}``-escaping for ``{{groovy}}`` delimiters), defeats
editor XML support, and silently rots when a future contributor
copy/edits the snippet without realising the surrounding ``body: |``
context.
Split by concern:
- ``templates/xml/*.xml.j2`` — six XML files that embed Jinja:
``user_page``, ``user_xwikiusers_object``,
``xwiki_admin_group_page``, ``xwiki_admin_group_member_object``,
``install_extensions_page``, ``main_webhome_page`` (the last one
was already a template; moved here for symmetry with the new five).
- ``files/xml/check_extension_page.xml`` — fully static; rewritten
without ``{% raw %}{{groovy}}{% endraw %}`` because the file is
served via ``lookup('file', …)`` (no Jinja pass), so ``{{groovy}}``
reaches XWiki literally.
Task references migrate to ``body: "{{ lookup('template',
'xml/<name>.xml.j2') }}"`` (or ``lookup('file', 'xml/<name>.xml')``
for the static case), normalising the previously bespoke long-form
``roles/web-app-xwiki/templates/main_webhome_page.xml.j2`` lookup in
the bargain.
SPOT-extract the magic string ``XWikiAdminGroup``:
- ``XWIKI_ADMIN_GROUP_PAGE: "XWikiAdminGroup"`` in vars/main.yml,
documented as the page-name part used by REST URLs, the
XAR-shipped page, the LDAP ``group_mapping`` argument, and the
OIDC ``groups.mapping`` argument.
- Five call sites updated to ``{{ XWIKI_ADMIN_GROUP_PAGE }}`` or
bare ``XWIKI_ADMIN_GROUP_PAGE`` inside list contexts:
``tasks/03_administrator.yml`` (×4 — two task names, the PUT URL,
the POST URL), ``templates/xml/xwiki_admin_group_page.xml.j2``,
``templates/compose.yml.j2`` (LDAP group_mapping), and
``templates/xwiki.properties.j2`` (OIDC groups.mapping, in a
commented example).
Lint guard: new ``tests/lint/ansible/tasks/test_no_inline_xml_body.py``
scans every ``roles/`` and ``tasks/`` YAML for ``body: |`` (or
``body: >``) folded scalars whose first content line starts with
``<`` and fails with a pointer at the matching ``templates/xml/...``
or ``files/xml/...`` alternative. JSON / shell / SQL bodies stay
allowed. Per-task opt-out: ``# nocheck: inline-xml-body`` on the
``body:`` line or the line above. Five unit tests cover the
scanner: matches XML body, ignores JSON body, honours same-line
nocheck, honours line-above nocheck, repo-wide sweep clean after
this refactor.
XWiki v17.10.8 distribution-merge behaviour was cross-checked
against upstream (``AbstractXarEntryType.upgradeType =
THREEWAYS``); the empty ``<member/>`` placeholder in the
XAR-shipped ``XWikiAdminGroup`` cannot overwrite our explicitly-added
member (different GUID / number).
``make test`` green across all 5 targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The inventory validator at cli/administration/inventory/validate
walked `recursive_keys(host_vars) ⊆ recursive_keys(application_defaults)`
and flagged any extra key with "Missing default for <app>: <key>".
After commit 10cb02461 ("bake only variants.yml overrides into
matrix-deploy --vars") host_vars carry every key the active
variants.yml overlay declares. Since 29377e1bd
(`_build_application_defaults` reads only `meta/<topic>.yml`, not
variants.yml — to keep the variant-0 service-key leak closed), the
defaults' allow-set is artificially narrower than the role contract:
roles/web-opt-rdr-www declares `services.dashboard.{enabled,shared}`
exclusively in `meta/variants.yml`, so the validator rejects the
baked host_vars with three "Missing default" entries before
ansible-playbook even starts (job 77797115291 on run 26428083128,
distro=debian; distro-agnostic per the matrix).
Widen the validator's legal-key set per app to
recursive_keys(defaults[app]) ∪ ⋃ recursive_keys(variants[app][i])
variants.yml IS part of the role contract; the validator's previous
"defaults only" view was incomplete. Runtime
`_build_application_defaults` stays narrow (so 29377e1bd's
variant-0-leak fix stays in place) — only the validator widens. Typos
not declared by any variant remain caught: a new
`test_key_typo_in_variants_still_rejected` unit test pins that.
Package split (KISS/DRY/SPOT), mirroring the credentials/ refactor:
`validate/__main__.py` carried six orthogonal concerns plus
orchestration in 175 LoC. Split:
- `validate/keys.py`: `recursive_keys` (pure helper).
- `validate/loaders.py`: `load_yaml_file` (with vault-block stripping)
and `load_inventory_files` (directory scan).
- `validate/applications.py`: `compare_application_keys` (the widened
variants-aware logic).
- `validate/users.py`: `compare_user_keys`.
- `validate/hosts.py`: `validate_host_keys`.
- `validate/__main__.py`: thin orchestrator — argparse + load defaults/
variants/users + invoke compare_* + exit code. 75 LoC.
`validate/__init__.py` re-exports the public API
(`main`, `compare_application_keys`, `compare_user_keys`,
`load_inventory_files`, `load_yaml_file`, `recursive_keys`,
`validate_host_keys`, `PROJECT_ROOT`). External callers
(`tests/unit/cli/administration/inventory/validate/test_main.py`,
`tests/unit/cli/administration/inventory/validate/test_hosts.py`)
keep working unchanged via this re-export.
`__main__.py` uses ABSOLUTE imports (`from cli.administration.inventory.
validate.applications import ...`) instead of relative because it's
invoked as a subprocess by tests and by the deploy entrypoint; the
sys.path bootstrap at the top of the file makes `cli.…` resolvable
in that mode.
Test layout follows the split — one test file per module:
- `test_keys.py` (6 tests: empty/non-dict/flat/nested/deep/prefix).
- `test_loaders.py` (8 tests: vault stripping, missing file, broken
YAML, top-level + recursive directory scan, empty-block skip).
- `test_applications.py` (9 tests: all-keys-match, unknown app, extra
key, credentials skip, variants legal, variants union, no-variants
fallback, source path in error, typo still rejected).
- `test_users.py` (5 tests: match, unknown user warn, missing default,
password skip, credentials skip).
- `test_hosts.py` (renamed from test_inventory_hosts.py — unit tests
for `validate_host_keys`).
- `test_main.py` (renamed from test_inventory.py — end-to-end
subprocess tests for `main()`, including the
`test_variants_only_key_is_accepted` regression guard for this fix).
Coverage: 36 tests, all green. `make test` green across all 5 targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The credentials baker at cli/administration/inventory/credentials/__main__.py
walked `schema_creds.items()` one level deep and called `str(value)` on
every leaf. For nested credential groups like
``credentials.recaptcha = {key, secret}`` (declared in
roles/web-app-espocrm/meta/schema.yml and
roles/web-app-listmonk/meta/schema.yml) this collapsed the inner dict
into its Python repr, vault-encrypted that blob, and wrote it as a
single scalar at ``credentials.recaptcha``. The runtime ``config``
lookup then choked on the descent into ``recaptcha.key``:
The lookup plugin 'config' failed: Expected dict for 'key', got
_AnsibleTaggedStr at 'key'
Full path so far: applications['web-app-espocrm'].credentials.recaptcha.key
Current object: "{'key': '{{ CAPTCHA.RECAPTCHA.KEY }}', 'secret':
'{{ CAPTCHA.RECAPTCHA.SECRET }}'}"
(jobs 77797371397 espocrm and 77797371442 listmonk on run 26428080957.)
Introduce ``emit_credentials`` (new module: ``credentials/emit.py``)
that walks the schema recursively. Scalar leaves still vault-encrypt
via ``to_vault_block``; dict nodes recurse into a nested CommentedMap
via ``ensure_map``. The walker also tracks dotted full-key paths so
the existing ``--force``-mode override-apply logic sees the same dotted
keys it always saw.
Auto-healing for previously-corrupted host_vars: ``ensure_map``
overwrites non-CommentedMap placeholders, so the next bake replaces a
stringified ``credentials.recaptcha`` blob with the correct nested
structure. Operators with corrupted entries simply re-run the
credentials generator; no manual cleanup needed.
Package refactor (KISS/DRY/SPOT):
The 450+ LoC ``__main__.py`` was carrying four orthogonal concerns
(vault, --set parsing, recursive emission, prompts) plus orchestration.
Split by concern:
- ``credentials/vault.py``: ``is_vault_encrypted``,
``is_ruamel_vault``, ``to_vault_block`` (+ private helpers
``_vault_body``, ``_make_vault_scalar_from_text``).
- ``credentials/overrides.py``: ``parse_overrides``, ``override_for``.
- ``credentials/emit.py``: ``ensure_map``, ``emit_credentials``.
- ``credentials/prompts.py``: ``ask_for_confirmation``.
- ``credentials/__main__.py``: argparse + ``main`` + two thin mode
handlers (``_run_snippet_mode``, ``_run_default_mode``) + the
force-override helper.
``credentials/__init__.py`` re-exports the public API
(``main``, ``ask_for_confirmation``, ``parse_overrides``, etc.) so
external callers (notably ``tests/unit/cli/administration/inventory/
test_credentials.py``) keep working unchanged.
Test layout follows the split:
- ``tests/unit/cli/administration/inventory/credentials/test_vault.py``
(20 tests: vault detection, body extraction, scalar construction,
to_vault_block paths, plus the VaultHandler subprocess coverage).
- ``tests/unit/cli/administration/inventory/credentials/test_overrides.py``
(13 tests: parse_overrides + override_for incl. dotted nested keys).
- ``tests/unit/cli/administration/inventory/credentials/test_emit.py``
(10 tests: ensure_map placeholder replacement, recursive emission,
none → empty, skip_existing semantics, override precedence,
dotted-path tracking, the on-disk corruption recovery case).
- ``tests/unit/cli/administration/inventory/credentials/test_prompts.py``
(5 tests: ask_for_confirmation yes/no/garbage/case/whitespace).
``tests/unit/cli/administration/inventory/test_credentials.py`` keeps
only the end-to-end ``main()`` integration coverage (including
``test_nested_credentials_dict_recurses_instead_of_stringifying`` —
the explicit regression guard for the original failure mode).
Coverage: 47 new per-module unit tests + the 5 existing main()
integration tests, all green. ``make test`` green across all five
targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CI job 77797371504 on run 26428080957 crashed with:
tls lookup: application_id '{{ lookup('env', 'INFINITO_DOMAIN') }}'
not found in domains mapping
`DOMAIN_PRIMARY` is defined in inventories/development/default.yml:20
as `"{{ lookup('env', 'INFINITO_DOMAIN') }}"`. On Ansible 2.18+ that
string carries a `TrustedAsTemplate` tag (it is an
`_AnsibleTaggedStr`). The constructor at tasks/stages/01_constructor.yml:64
wrapped it in `[ DOMAIN_PRIMARY ] | flatten | map('string') | first`
— the explicit `str()` cast strips the trust tag. The resulting plain
string flowed into `CURRENT_PLAY_REDIRECT_DOMAINS.source` as the
literal text `{{ lookup('env', 'INFINITO_DOMAIN') }}`, was passed
through `domain: "{{ item.source }}"` in
roles/web-opt-rdr-domains/tasks/main.yml:14 (one render pass that
produces untrusted content), and finally reached `as_str(terms[0])`
in plugins/lookup/tls.py:39 unchanged. The tls plugin never templated
its arg, so `resolve_term` looked up the literal Jinja string in the
domains mapping and raised.
Two layers, both necessary:
1. plugins/lookup/tls.py:39 — defensively template `terms[0]` through
`self._templar` before string-coercion. Mirrors the idiom in
plugins/lookup/unit_name.py:33 and plugins/lookup/email.py:61. Any
future caller that hands the plugin an unresolved expression now
gets it resolved before the domains-map lookup.
2. tasks/stages/01_constructor.yml:64-65 — drop the
`[ X ] | flatten | map('string') | first` ritual for
`DOMAIN_PRIMARY` and `DOMAIN_HOMEPAGE`. Both are scalar strings in
every known inventory (group_vars/all/00_general.yml:41,43 +
inventories/development/default.yml:20); the over-defensive
wrapper was the trust-tag stripper. Direct reference preserves the
tag through `CURRENT_PLAY_REDIRECT_DOMAINS` and every downstream
consumer.
Downstream consumers audited; all transparent to `_AnsibleTaggedStr`
(it inherits from `str`):
- merge_mapping filter: content-based dict-key compare
- domain_mappings filter: takes `DOMAIN_PRIMARY` directly, no cast
- items2dict(key_name='source', value_name='source'): hashable
- generate_all_domains filter: `isinstance(item, str)` matches
- sys-stk-front-base `when: ... | length > 0`: only counts
- web-opt-rdr-domains -> tls lookup (the failing path): now templates
Coverage: new `test_term_with_unresolved_jinja_is_templated` in
tests/unit/plugins/lookup/test_tls.py verifies the lookup feeds its
first term through the templar. The 12 existing tls tests stay
green. `make test` green across all 5 targets.
Class follow-up (not addressed here): the
`| string` / `| map('string')` cast still strips `TrustedAsTemplate`
elsewhere if a future commit reintroduces it; 12 other lookup
plugins do not yet defensively template their args. A lint guard and
plugin-wide audit will come in separate PRs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pruned per project convention "no unnecessary code comments":
- module docstring: 23 lines → 1 line (the file name + symbol already
state intent; full incident write-up belongs in the commit log)
- wipe_database_entries docstring: 11 lines → 2 lines
- six inline rationale comments in tests/__main__ removed (test
names already encode the intent)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
After 10cb02461 switched `payload._resolve_variant_payloads` from
`get_variants` to `get_variant_overrides_only`, matrix-deploy bakes
only the raw `meta/variants.yml` slice into host_vars (typically just
`services:`). Lookups that read `variables['applications']` raw no
longer see content from `meta/rbac.yml`, `meta/server.yml`, etc.
`rbac_group_path` was the last lookup still going through the raw
host_vars read. For joomla (whose `meta/rbac.yml` declares the
`editor` role) the post-10cb02461 host_vars carried no `rbac` block
at all, the lookup walked an empty `rbac.roles` map, and the deploy
crashed with "rbac_group_path: role 'editor' is not declared under
applications[web-app-joomla].rbac.roles. Declared roles:
['administrator']" (job 77797371433 on run 26428080957).
Route `_get_application` through `get_merged_applications` so the
meta-merged view (services + server + rbac + volumes + schema
credentials + variant overrides) is restored. This mirrors the
existing pattern in plugins/lookup/{config,tls,nginx,domain}.py and
keeps the mirror-override intent of 10cb02461 intact: the bake step
still emits only the variant overrides; the merged cache assembles
the full view at lookup time.
Test updates:
- TestRbacGroupPathLookup.setUp patches get_merged_applications to
read from variables['applications'] so the existing 14 tests stay
hermetic without disk-reading roles/*/meta/*.
- New TestRbacGroupPathReadsMergedApplications regression case
exercises the exact 10cb02461 scenario: host_vars only carry a
variants.yml slice, the merged cache supplies the rbac block, the
lookup returns the correct path.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Matrix-deploy variant transitions wipe a consumer's compose stack and
its named DB volume (`compose down -v` in
scripts/container/purge/entity/compose.sh) but `databases.csv` at
$FILE_DATABASE_SECRETS survives the purge. When the next round drops
the consumer from its inventory, the stale baudolo-seed row remains
behind. The subsequent backup run reads that row, attempts
`mariadb-dump -u<entity> -p<old_pw>` against the freshly initialised
RDBMS user that no longer exists, and the systemd unit fails with
"Access denied for user 'matomo'@'127.0.0.1' (using password: YES)"
(job 77797356000 on run 26428080957: matomo dump in
svc-bkp-container-2-local variant 1, where round 1 seeded matomo but
round 2 only deploys postgres + mariadb + svc-bkp itself).
Add `utils.cleanup.databases_csv` mirroring the existing
`utils.cleanup.tokens` pattern: parse the baudolo-seed CSV format
(';'-separated, four columns instance/database/username/password, with
header — verified against baudolo.seed.__main__:check_and_add_entry)
and strip every row whose `database` or `username` column matches an
entity derived from the supplied app ids via
utils.roles.entity_name.get_entity_name. Header and blank lines are
preserved verbatim; the file is rewritten only when at least one row
was removed.
Wire the helper into scripts/container/purge/apps.sh right after the
token-store wipe so every matrix-deploy inter-round purge invokes it
alongside the other cleanups. Operator-triggered purges
(`INFINITO_APPS=<app> bash scripts/container/purge/apps.sh`) also get
the cleanup — that is a behaviour change relative to before, where
the row would survive an operator-triggered entity teardown.
Coverage: 12 unit tests in tests/unit/utils/cleanup/test_databases_csv.py
covering the happy path, the multi-app case, the
`database='*'`/username-match case, the unknown-app no-op, header and
blank-line preservation, the all-rows-removed case, malformed short
rows, and the legacy-header-less file. `make test` green across all 5
targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- web-opt-rdr-www canonical: rdr-www.{DOMAIN_PRIMARY} -> w3redirect.{DOMAIN_PRIMARY}.
- New lookup `www_redirect_domains` (role-scoped): returns full {source, target}
mapping list, resolves DOMAIN_HOMEPAGE target for the role's own canonical via
lookup('domain', 'web-opt-rdr-www'). REDIRECT_WWW_DOMAINS now carries dicts;
tasks/main.yml and _01_cloudflare_edge_redirect.yml updated to consume dicts.
- New lookup `current_play_redirect_domains`: replaces the two inline
set_fact merges in tasks/stages/01_constructor.yml. Merges user-supplied
redirect_domain_mappings, primary-redirect (conditional on web-opt-rdr-domains
in deployed), and per-app domain mappings (inlined from the now-removed
domain_mappings filter).
- domain_redirect_mappings filter removed; web-app-keycloak RedirectUrisTest
pollution of utils.roles.applications.config in sys.modules fixed via
tearDownClass cleanup.
- csp_skip_domains lookup: csp.health.skip opt-out path removed (architecture
no longer needs it).
- Lint test_only_web_app_roles_integrate_dashboard expanded from web-app-* to
web-*; test_role_and_group_vars_used now scans .py files too so vars consumed
only inside plugins (DOMAIN_HOMEPAGE, AUTO_BUILD_ALIASES) are seen.
- New role-scoped variants.yml on web-opt-rdr-www with dashboard on/off.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Since 576fad84f (2026-05-10) moved role images from
`defaults/main.yml` into `meta/services.yml`, matrix-deploy's variant
baker has been writing the FULL effective config (services.yml +
variants.yml deep-merged via `get_variants`) into host_vars through
the `--vars` JSON consumed by `apply_vars_overrides`. That populates
`applications.{role}.services.{svc}.image` (and `.version`, .cpus,
etc.) with the upstream value, which then silently disables
`apply_mirror_overrides` for every role with an explicit image
declaration: the default `if_missing` policy short-circuits when the
field is already non-blank, so the rendered compose / Dockerfile FROM
keeps pulling from the upstream registry rather than the GHCR mirror.
The regression went unnoticed because Docker Hub usually serves the
pull, but it eventually surfaced as a BuildKit 429 on
`docker.io/library/php:8.2-apache` during the suitecrm Dockerfile
build (job 77602110531 on run 26362676803). Nextcloud, moodle,
mariadb consumers, postgres consumers, simpleicons, phpmyadmin,
phpldapadmin, jenkins, baserow, fider, ... all share the same latent
defect — only suitecrm happened to hit the rate limit first.
Fix the root cause instead of papering over per-role with
`mirror_policy: force`: introduce `get_variant_overrides_only` next
to `get_variants` and switch payload.py to it. The new function
returns the raw `meta/variants.yml` entries per role — no merge with
`meta/services.yml`. matrix-deploy now bakes only the variant-
specific overrides into host_vars; the role's services.yml fields
flow through the normal Ansible lookup chain at deploy time, leaving
`image`/`version` blank in host_vars so `apply_mirror_overrides`
populates them from `mirrors.yml` under its default `if_missing`
policy.
Behavior preserved for the other `get_variants` consumers:
* utils/manager/inventory.py:104 (schema-driven credentials) keeps
the full merged variant payload.
* planner.py:58 (variant-count only) is unaffected.
* legacy_resolver.py:42 (CombinedResolver runtime services_overrides)
keeps the full merged view.
Coverage:
* 3 new tests in test_applications.py for the new helper (single
empty variant when no meta/variants.yml, raw override without base
merge, cache invariance).
* test_payload.py + test_builder.py mocks repointed from `get_variants`
to `get_variant_overrides_only`.
* `make test` green across all 5 targets (lint, unit, integration,
external, test-lint) — 188 tests total.
Side-effects assessed:
* Repo-wide grep confirms no template / task accesses
`applications.X.services.Y.image` directly via hostvars — every
consumer goes through `lookup('config', …)` which still deep-merges.
* Roles previously pulling upstream now resolve via the GHCR mirror
whenever the image is already synced (the `images-mirror-missing.yml`
workflow keeps the destination populated).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`_apply_one_role_special_rules()` in utils/manager/inventory.py used
`bool(resolve_database_service_key(...))` to decide whether to
auto-generate a `database_password` credential for a role. The
resolver raises when multiple RDBMS keys are enabled — which is the
correct contract for compose_volumes / container_depends_on /
plugins.lookup.database (they need an unambiguous single key) but the
wrong contract for inventory's credential-generation pass: variant 1
of svc-bkp-container-2-local intentionally enables both mariadb and
postgres to exercise the engine-side dump path against both engines
at once, and the deploy aborted with `ValueError: multiple direct
database services are enabled: mariadb, postgres` during inventory
generation (job 77602095377 on run 26362676803).
Add a non-raising sibling helper `has_single_database_service()` that
returns True iff exactly one RDBMS service is enabled, and switch
inventory's call site to it. Multi-DB roles now skip the auto-
generated `database_password` (which was a phantom credential for
svc-bkp anyway — no schema.yml, no cross-role lookup; the role's own
`04_seed-database-to-backup.yml` receives `database_password` as a
caller-passed task var, not as a self-credential lookup).
`resolve_database_service_key` keeps its strict raise semantics for
the compose_volumes / container_depends_on / database-lookup callers,
which still need a single unambiguous key.
Add unit coverage for both functions (3 + 4 tests) in
tests/unit/utils/roles/applications/services/test_database.py.
Static check only (per operator request).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`generate_all_domains(include_www=True)` synthesizes `www.X` for every
canonical domain X and pushes them into CURRENT_PLAY_DOMAINS_ALL, but
they are never registered in the global canonical_domains_map or in
applications[*].server.domains. The strict `tls`/`cert` lookups
therefore raised `domain 'www.X' not found (domains/applications)`
when web-opt-rdr-www iterated REDIRECT_WWW_DOMAINS and called
`lookup('tls', 'www.X', 'enabled')` in the redirect-domain.yml TLS
gate — failing job 77602110568 on run 26362676803 (and cascading
through sys-front-tls/vars/main.yml, sys-front-tls/tasks/02_http.yml
and sys-front-tls-selfsigned/tasks/01_core.yml, all of which call
`lookup('tls', application_id, …)` with the same leaked www.X).
Add a single fallback branch to `resolve_term`: if the strict lookup
fails AND the term starts with `www.`, retry both resolver paths
(legacy reverse + applications) against the bare domain. The fix is
strictly additive — every previously successful term and every non-www
miss preserves its current behavior.
Coverage:
* test_www_prefix_falls_back_to_bare_domain
* test_www_prefix_with_unknown_bare_still_raises
* test_non_www_unregistered_still_raises (regression guard)
Verified statically via the in-container `resolve_term`:
www.infinito.example -> (svc-prx-openresty, infinito.example)
www.rdr-www.infinito.example -> (web-opt-rdr-www, rdr-www.infinito.example)
The global self-signed cert SAN list (cert.py:193) is built from
CURRENT_PLAY_DOMAINS_ALL, so it already covers the www.X variants —
nginx serves the redirect vhost with that cert and the TLS handshake
on the hlth-webserver check matches.
Static check only (per operator request).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Flatten records from all summaries and sort by ISO-Z timestamp so the
step-summary table is chronological across apps/variants/passes instead
of grouped per file.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`_build_application_defaults` returned variant 0 of every role with a
meta/variants.yml, which then deep-merged with the variant-N override
that build_dev_inventory bakes into the inventory. Any service flag
that variant N did not explicitly disable carried through from
variant 0 into the runtime applications map. The most visible casualty
was svc-bkp-container-2-local variant 1 (postgres + mariadb only),
where gitea + keycloak leaked in via variant 0, sys-service-loader
pulled in svc-db-openldap via keycloak.services.ldap, and the deploy
crashed in sys-svc-compose/04_files.yml on openldap's env.j2 with
"credentials is defined in schema at roles/svc-db-openldap/meta/schema.yml
but not set in application 'svc-db-openldap'".
Defaults now come from each role's variant-free base config
(meta/<topic>.yml plus schema-defaults, no variants overlay). Variants
stay accessible via get_variants for any caller that explicitly needs
a specific variant. Inventory-level applications.<app> overrides win
exactly as before.
Supporting changes that keep the variant matrix consistent under the
new defaults model:
- New lint guard tests/integration/roles/meta/variants/
test_non_baseline_explicit_disables.py: for every dynamic
service-key declared in a role's meta/services.yml that variant 0
pins to literal true, every non-baseline variant MUST explicitly
pin the same key to either literal true or to literal
enabled: false + shared: false. Statically-enabled keys are exempt
(same rule as test_auth_isolation.py).
- roles/web-app-{fusiondirectory,lam,nextcloud,phpldapadmin}/meta/
variants.yml: explicit `false` pins for dashboard / email / logout /
matomo / prometheus / talk in restrictive variants so deep-merge no
longer hands them the deleted variant-0 inheritance.
- roles/web-app-moodle/templates/playwright.env.j2: gate the LAM
block on `lookup('config', application_id, 'services.lam.enabled',
false)`. LAM is a transitive helper, not a moodle-owned service, so
the default-bearing lookup tolerates the missing path when the
variant does not deploy LAM.
- tests/unit/utils/cache/test_variants.py: rename the legacy
"always variant zero" coverage to TestMergedApplicationsUsesBaseConfig
and add test_get_application_defaults_omits_variant_zero_only_keys
as the regression guard for the leak this commit fixes.
- docs/contributing/artefact/files/role/variants.md +
docs/contributing/design/variants.md: variant 0 now manually
enumerates every dynamic services.yml key with
enabled: true + shared: true; `{}` is only permitted when the role
has no dynamic-enabled service-key.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
InventoryManager.resolve_schema_includes_recursive previously read the
root role's base meta/services.yml when discovering shared providers,
regardless of which matrix variant the planner had pinned for the
round. For a role whose variants.yml enables additional services per
variant (e.g. svc-bkp-container-2-local variant 2 enabling
services.ldap), the base config had no `enabled+shared` flag for those
services, so the recursive resolver never visited the variant's
shared providers — their credentials were never generated, and the
deploy crashed in the provider's env.j2 with "credentials is defined
in schema at roles/<provider>/meta/schema.yml but not set in
application '<provider>'".
Wires the active variant through the existing inventory pipeline:
- utils/manager/inventory.py: InventoryManager(variant=None|int). When
set, load_role_config_by_path overlays variants.yml[N] onto the root
role's meta config (only for the root — transitive providers keep
their own base lookup). variant=None preserves the pre-refactor base
behavior for every standalone caller.
- cli/administration/inventory/credentials: new --variant N flag,
forwarded to InventoryManager.
- cli/administration/inventory/provision: new --app-variants <JSON>
flag ({app_id: variant_index}), forwarded per-app to the credentials
subprocess.
- cli/administration/deploy/development/inventory/builder.py: emits
--app-variants from spec.variant_selectors() when the round pins any
variant. Apps without variants.yml collapse to a single empty
overlay in get_variants so this is a no-op for them.
10 unit tests cover: variant overlay for root role, variant=None
fallback to base, out-of-range fallback, no overlay on non-root
paths, --variant CLI forwarding (set + default), --app-variants
generator forwarding (set + omit), builder --app-variants emission
(set + omit).
Caveat: bkp-c2l CI variant 1 still loads svc-db-openldap transitively
at runtime (outside the planner's variant closure) — a separate
sys-service-loader / role-meta bug, not addressed here.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
keepass.info responds with 415 Unsupported Media Type to the probe's
GET; the previous 25a77bc7e widening landed 418/429/451 but missed 415.
Treat 415 like 401/403/405 (server alive, request format rejected).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Commit f63118f34 ("feat(lint): enforce auth-isolation in
variants.yml") added `enabled: false, shared: false` overrides for
mariadb / postgres to every auth-isolated LDAP-only variant. That
disables the role's hard-required DB at deploy time, so e.g. the
moodle baseline blows up with `Required env var MOODLE_DB_HOST is
not set` as soon as round 2 of the matrix hits variant 1.
The auth-isolation rule's premise — "auth-isolated rounds must trim
non-auth services from the include closure" — only buys closure
savings for services whose `meta/services.yml` base declares
`enabled` as a dynamic Jinja expression. DB-engine entries are
already a static `enabled: true`, so the resolver pulls them in
regardless of the variant pin; the variant `false` only breaks the
deploy without trimming anything.
Two changes:
- `test_auth_isolation`: exempt service-keys whose `services.yml`
base has a static literal `enabled: true` from the per-variant
pin obligation. Such keys should not be declared in auth-isolated
variants at all; the inherited base value is correct.
- 10 web-app roles in the active deploy scope (akaunting, baserow,
bookwyrm, espocrm, friendica, gitea, moodle, nextcloud, odoo,
openproject): drop the entire mariadb/postgres stanza from the
LDAP-only variant — both `enabled` and `shared` inherit from
`meta/services.yml`. Matches the pre-f63118f34 state.
- 6 more roles out of the current iteration scope (funkwhale,
opentalk, shopware, snipe-it, suitecrm, taiga) carry the same
broken `enabled: false` line: removed it, kept the pre-existing
`shared: false` to preserve the variant's own-DB-container
topology intent.
`make test` is green; `variants-auth-isolation` and
`variants-static-override` both pass with no per-variant nochecks
required.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The compose templates appended additional depends_on entries (e.g. taiga
rabbitmq services, openproject cache/seeder, mailu resolver/front) on
column 6 directly after the lookup call. When the lookup returned ""
(no DB and no redis), those entries dangled without a depends_on header
and broke YAML parsing.
Move every such trailing block into the lookup's `extra` kwarg so the
header is always emitted as soon as any entry exists. Adds a regression
test for the extras-only case.
Affected: taiga, mailu, magento, pixelfed, openproject, espocrm.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Eliminate include-style depends_on assembly. The lookup resolves DB/redis
inductively from applications config and returns indented YAML (or "" when
no entries), removing the empty `depends_on:` block bug in libretranslate.
Migrates 43 compose templates from `{% include … dmbs_*.yml.j2 %}` to
`{{ lookup('container_depends_on', application_id) }}`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`status` was too vague for what the lookup returns (apps in current
play). Rename to `deployment` and add two new fields:
* `runtime` — RUNTIME variable value
* `deployed` — runtime-aware authoritative set: equals `running`
(whitelist or groups) for ephemeral CI runtimes (act, github),
falls back to `groups` for persistent runtimes (host, dev). This
matches reality: in CI the container is fresh so only whitelisted
apps exist; in production a partial deploy still leaves prior
apps live, so groups is the right set.
Switch the two webserver/csp healthchecks to `lookup('deployment').deployed`
so they only probe domains for apps that actually exist in the current
deploy. In production behaviour is unchanged.
Update remaining consumers (cli/build/include, web-app-keycloak,
test-e2e-playwright) for the rename. Existing `status` semantics
(`whitelist`, `running`, `groups`, `all`) preserved under the new
name.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cluster 2a (hlth-msmtp failed=65 in universal single-app deploys):
the health-check script sent "To: administrator" but the administrator
Unix user is only provisioned in server-pass, so postfix rejected the
recipient. Replace the inline Jinja conditional with a role-local
lookup plugin that picks administrator.email (external), "administrator"
(local + user exists) or "root" (local + missing).
Side change: SIM105 cleanup in utils/install/lint/__main__.py.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fixes Cluster 3a ca-inject YAML parse: when a role variant disables its
DB (enabled=false), the depends_on template emitted ` :` because
the existing `not shared` guard fired for a non-existing DB host.
The new derived flag mirrors `service.required` (= enabled AND shared)
and is consumed in:
- depends_on/dbms_base.yml.j2, dmbs_excl.yml.j2 (the actual fix)
- sys-svc-rdbms postgres/mariadb service templates
- web-app-discourse, web-app-snipe-it (latent same-pattern defensives)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Renames the shared-services loader role from sys-utils-service-loader to
sys-service-loader (shorter, no orphaned "utils" prefix; entity name in
meta/services.yml realigns to service-loader per get_entity_name).
Introduces an explicit two-phase contract in the loader: after the
ordered preload pass, flush a `_service_loader_post_run` queue that
service roles register into during preload via the new helper
`list_or_shoot.yml` (list during preload, shoot directly outside it).
Replaces the `when: not _service_loader_preloading` guards in
sys-svc-compose and sys-svc-webserver-core with helper calls. Closes
the gap where non-web deploys would never load these post-load
dependencies; in normal web deploys behavior stays the same thanks to
each sub-role's `run_once_<role>` gate.
Docs, comments, test docstring, and constructor-stage invocation
follow the rename. Historical changelogs (CHANGELOG.md, packaging/*)
intentionally left at the original name.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Trailing textual references missed in the prior cmd-migration commit:
three requirement docs and two lint-test docstrings still pointed at
`make exec`, but the target has always been `make compose-exec`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the gap left by test_image_versions, which silently skips any
non-semver pin (latest, stable, branch tags). The new test_image_reachable
walks every entry from iter_role_images and probes the Docker Registry v2
manifest endpoint via HTTPS with anonymous bearer-token negotiation
(docker.io, ghcr.io, quay.io, mcr.microsoft.com). HTTP 404 hard-fails as
a broken pin; auth/rate-limit/5xx/network errors warn. Bounded by per-
request, per-probe, and global deadlines and parallelized via
INFINITO_WORKER_FETCH.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
INFINITO_CMD is a per-invocation argument, not a config value — drop it
from default.env, the passthrough STATIC_KEYS, and the dotenv-generator
test, and rename to lowercase `cmd` in Makefile targets, exec scripts,
internal callers, and user-facing docs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`categories_of()` now emits the sub-level key with the same hyphen
the role-id already uses (`web-app`, `svc-bkp`, `sys-ctl`) instead of
the dot-form (`web.app`, `svc.bkp`, `sys.ctl`). Reads cleanly against
the role naming convention and removes the only place in the
matching layer where role-id segments got rewritten.
Updates the matching unit tests and the only consumer annotation
that referenced the dot form (`sys-ctl-cln-bkps`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Auth-isolated matrix variants (exactly one of LDAP / OIDC-OAuth2
enabled, and no non-auth `enabled: true` alongside) must explicitly
pin every non-auth service-key in the role's variant union to literal
`enabled: false, shared: false`. Without this, the variant's
auth-only test scope balloons into a full integration deploy via the
services.yml-default propagation, wasting CI minutes and obscuring
the regression signal.
Adds `tests/integration/roles/meta/variants/test_auth_isolation.py`
(rule key `variants-auth-isolation`) plus the matching catalog entry.
The trigger is narrow on purpose: a variant that enables auth
alongside other services is a deliberate integration test, not an
auth-isolated one, and the rule does not fire there.
Mass-fixes the 24 legacy auth-isolated variants the rule turned up:
each gets the missing `enabled: false, shared: false` declarations
appended to its `services:` block. Where the false-override clashes
with a static-literal base in meta/services.yml, the override line
also picks up a `# nocheck: variants-static-override` marker. The
svc-bkp-container-2-local variant 2 additionally carries
`# nocheck: variants-services-match` on the new keys because they
are resolver-only matrix hooks with no consumer contract in the base
meta.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The variants → services match guard (`test_services_match.py`) used to
have no exemption mechanism, forcing every variant service-key to also
appear as a top-level entry in `meta/services.yml`. That contract is
correct for application roles whose variants pin per-app service
flags, but it forces noise on invokable infrastructure roles whose
variants exist only to pin the round's companion topology — there the
service-key is a resolver hook, not a consumer-side contract, and
adding a stub in `meta/services.yml` just duplicates the variant
declaration.
Add a `# nocheck: variants-services-match` opt-out (same line or line
above the key) so legitimate resolver-only matrix hooks can stay in
variants.yml without a synthetic services.yml stub. Mirrors the
`variants-static-override` parser pattern (text walk + per-leaf line
indices + is_suppressed_at).
Adopts the new exemption on `svc-bkp-container-2-local`:
meta/services.yml drops the gitea / keycloak / postgres / mariadb /
ldap stubs, meta/variants.yml carries the nocheck marker on each
variant service-key entry. The resolver closure per round stays
identical (16 / 3 / 2 roles).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Consolidates the backup-role topology around four directional roles
under the existing svc.bkp category:
sys-ctl-bkp-docker-2-loc → svc-bkp-container-2-local
svc-bkp-loc-2-usb → svc-bkp-local-2-device
svc-bkp-rmt-2-loc → svc-bkp-remote-2-local
sys-bkp-provider → deleted (purely a bundler with no own logic;
the two pieces it pulled in — user-backup
and sys-timer-cln-bkps — are now declared
directly by callers that need them)
Every renamed role moves into svc.bkp (already invokable per
roles/categories.yml), gets an `application_id` in vars/main.yml, and
declares the per-app primary-entity service flags + resource limits the
invokable-role lint requires.
Companion changes:
- svc-bkp-container-2-local: meta/variants.yml declares three matrix
rounds (gitea+keycloak, postgres+mariadb, openldap) that pin the
companion topology via service-key flags the CombinedResolver
translates into provider-role includes.
- sys-svc-compose: container-2-local is no longer in the docker-mgmt
include loop; it joins via a separate `'svc-bkp-container-2-local'
in group_names` gate so the 3 backup roles are deployable
independently (each addresses its own failure mode).
- The three `04_seed-database-to-backup.yml` includes (web-app-matrix,
web-app-bigbluebutton, sys-svc-rdbms) are gated by the same
group_names check so consumer apps only seed when the container
backup is actually in scope.
- web-app-gitea picks up `enabled: true, shared: true, provides:
gitea` on its primary entity so the service registry can pull it in
via a `services.gitea.enabled+shared` flag (needed for the
container-backup variant 0).
- group_vars constants and the pyproject ruff per-file-ignore + the
.ansible-lint-ignore paths follow the role-id rename.
- The remote-2-local script picks up the snake_case rename
(`pull-specific-host.py` → `pull_specific_host.py`) and the
`os.makedirs` → `Path.mkdir` conversion so the role no longer needs
the PTH/N999 ruff escape hatch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous regex anchored the `name` capture at line start and used
lazy `.+?`, which in production captured the full
`<ts> p=<pid> u=<user> n=ansible INFO| <role>` prefix as the role name
instead of just `<role>`. The CI step summary then rendered every row
with the timestamped log header and the `total` filter silently failed
(because `name` was no longer equal to `"total"`).
Add an optional non-capturing `(?:.*\bINFO\|\s+)?` group so the
ANSIBLE_LOG_PATH prefix is consumed when present and `name` captures
just the role token (or `role : task`). Test sample didn't carry the
log prefix, so the bug slipped through — add a prefixed-format test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Move role_is_invokable / role_has_required_by / role_has_nocheck out
of plugins/filter (where they were flagged as test-only filters) into
utils/roles/required_by_coverage.py. All three use load_yaml_any,
read_text, and ROLE_FILE_META_SERVICES from the canonical SPOTs.
- Refactor the verifier (cli.meta.roles.services.called) the same way:
iterate roles/*/ directly, look up meta/services.yml via the constant,
parse with load_yaml_any, and tolerate a missing roles_dir (unit tests
point at a fake /repo).
- Make the mailu-dependency integration test yaml-aware for meta/
services.yml: walk the parsed tree and skip values inside any
`required_by` subtree. `required_by` is inverse coverage metadata for
the deploy verifier, not a "consumer of mailu" reference, so previous
regex-on-text matching produced false positives for cert-deploy.
- sys-ctl-mtn-cert-renew is gated by `lookup('tls', application_id,
'mode') == 'letsencrypt'`, which the verifier can't express via
role-id/category match — mark nocheck with the reason inline.
- Add README.md for cli/meta/roles/services (CLI-category convention).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a static lint that hard-fails when a non-invokable role's
meta/services.yml declares neither `required_by` nor `nocheck:
required-by-coverage`. This protects the runtime verifier
(cli.meta.roles.services.called) from silently losing coverage when new
infrastructure roles ship without a coverage contract.
New filter plugins (reusable from Ansible + Python tests):
- role_is_invokable: walks categories.yml; true if any node on the
role's category path is `invokable: true`.
- role_has_required_by: true if any entity in meta/services.yml declares
a non-empty `required_by.categories` or `required_by.roles`.
- role_has_nocheck: true if meta/services.yml carries
`# nocheck: <check_id>` (whole-word match).
New test: tests/lint/ansible/roles/meta/test_required_by_for_non_invokable.py
fails CI with a per-file list and a fix recipe (declare required_by OR add
the nocheck marker — both already shipped in Bundle B and Bundle D).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Asserts at deploy time that every role declaring `required_by` in its
meta/services.yml actually executed (at least one TASK block ending with
a non-skip status of `ok` / `changed` / `fatal` / `included`). Catches
silent coverage regressions where a once-flag swallows an include before
the role's body can run.
CLI: `python -m cli.meta.roles.services.called --logfile X --apps a,b,c`
Library: `cli.meta.roles.services.called.verify(...)` supports per-pass
slicing via `log_byte_offset` and reads either host-side log files or
in-container logs when `container=` is given.
Hooked into `cli.administration.deploy.development.deploy._run_deploy`
so matrix-deploy runs (CI + local make compose-deploy mode=initialize/
reinstall) snapshot the log offset before each compose.exec and verify
the resulting slice after rc=0. Defaults ANSIBLE_LOG_PATH to
/tmp/infinito-deploy.log so the check also runs locally without env-var
setup.
Includes 35 self-contained unit tests covering category derivation,
TASK-block status parsing (incl. ANSI escape stripping + substring
role-id guard + multi-skip edge case), required_role_ids YAML scan,
host log slicing with byte offsets, end-to-end verify, and CLI argparse.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reworks the REPL into a stateful shell that tracks a category path: typing a bare category name cd's into it, `/` returns to root, `..` walks one level up (chained as `../..` etc.), path tokens with slashes or a leading `..` navigate, `infinito [path]` always jumps absolute, and `ls` renders the current folder with a 🗂️/⚙️ emoji column, name, and a 30-char-truncated description (folders sorted before commands). Executable leaves (sub-packages with `__main__.py`) run instead of cd-ing into them. The prompt prints `🖥️ infinito <space-separated path>> `; the screen clears (incl. scrollback via `\033[3J`) before each command run, and the run echoes the resolved invocation in the fresh frame so the user sees what is about to execute.
Splits `repl.py` (~250 lines) into focused modules: `constants.py` (sentinels), `navigation.py` (path/category resolve + prompt + normalize), `ls.py` (folder rendering + truncate), `runner.py` (subprocess dispatch + clear_screen), `banner.py` (startup banner). `repl.py` is now a thin `main()` loop with backward-compat re-exports so external consumers keep the `repl.<symbol>` surface. Mirrors the split on the test side into `test_navigation.py`, `test_runner.py`, `test_ls.py`, and a `test_repl.py` that owns only main-loop integration with a shared `_MainLoopHarness` mixin.
Adds `scripts/tests/environment/11_console.sh` as a CI smoke that feeds a navigation sequence through the REPL and asserts every prompt shape plus the `ls` emoji output, so a regression in the prompt format, the path resolver, or the folder rendering is caught by the environment-test orchestrator.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Per-role Playwright runs no longer call `npm install` against
registry.npmjs.org on every invocation. A new `01_setup.yml` runs once
per host/version, populates `_shared/node_modules/` inside the Playwright
container, and stamps the installed version in `.installed-version`.
Each `02_run_one.yml` mounts that tree read-only at `/e2e/node_modules`
and the container command becomes `npx --no-install playwright test`.
This eliminates the npmjs.org TLS-handshake flake that took out the
keycloak playwright in a recent variant-2 update, saves ~4 s per role,
and gives version bumps a single rebuild point.
Three retry layers cover transient registry failures: Ansible (3× with
5 s delay), npm fetch (5× with 10-60 s backoff), and image pull (3×).
Stage-base / shared-dir / stamp / node_modules paths live as SPOT
constants in `vars/main.yml` so call sites never reconstruct them.
Renames `tasks/run_one.yml` → `tasks/02_run_one.yml` (and adds
`01_setup.yml`) to make the include order visible by filename; updates
cross-references in `AGENTS.md`, the two `docs/contributing/...` pages,
`dump_disk_diagnostics.sh`, and the lint test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`role_summary.py` no longer truncates to a configurable top-N: drops
`DEFAULT_TOP_N`, `TOP_N_ENV_KEY`, and `_read_top_n`; `_format_table`
renders all rows from `parse_role_times`. Renames the Markdown
heading and the corresponding GitHub Actions step in the four
`test-deploy-*` workflows from `Top role runtimes` to
`Role runtimes` for accuracy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Dashboard card icons were rendered with `http://172.17.0.1:8044/<slug>.svg`,
the in-cluster sync URL that only the Ansible runner can reach. The
browser saw `NS_ERROR_CONNECTION_REFUSED` for every Simple Icons card.
The `add_simpleicon_source` filter now takes two URLs: it probes the
local sync URL (plain HTTP, no TLS, no redirect chain — reachable
from the deploy container's `requests` without a CA bundle) and
writes the public `lookup('tls', 'web-svc-simpleicons', 'url.base')`
URL into `icon.source`. Cards render `<img src="https://icon.<DOMAIN>/<slug>.svg">`,
which is reachable from the browser and CSP-permitted (`img-src *`).
Drops the dead local-cache scaffolding (static-subdir vars, volume
mount, `Ensure directory` task, refresh-download step) since the
browser now fetches Simple Icons directly. Renames the
previously-static `templates/style.css.j2` to `files/style.css`
(no Jinja markers, matches the new template-uses-jinja lint warning).
Adds a Playwright spec `test-icons.js` that asserts the regression:
no card may still reference `172.17.0.1`, at least one card must
resolve under `icon.<DOMAIN>`, with a favicon fallback path when the
Simple Icons service is disabled.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a warning-only lint that walks every `roles/<role>/templates/**/*.j2`
file (via the cached `iter_project_files_with_content` helper) and flags
those that contain no `{{ }}`, `{% %}`, or `{# #}` markers. Each
offender gets a GitHub Actions `::warning ::` annotation pointing at
the file path, plus a consolidated stdout list for local runs, with
the suggestion to move the file under `roles/<role>/files/` and load
it via `copy:` / `script:`. The test does not assert — purely warning
surface so we can drain the backlog incrementally.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`lookup('image', '<role>', '<svc>', 'ref')` only consulted
`images_overrides.*` plus role defaults, which the mirror pipeline
never populates — the pipeline writes into
`applications.<role>.services.<svc>.{image,version}`. As a result,
`sys-svc-container`'s `BUSYBOX_IMAGE` / `NODE_IMAGE`,
`sys-ctl-hlth-csp`'s `HEALTH_CSP_IMAGE`, and `test-e2e-playwright`'s
`TEST_E2E_PLAYWRIGHT_IMAGE` always resolved to the raw upstream ref
and hit Docker Hub directly. In CI this surfaced as
`toomanyrequests: ...unauthenticated pull rate limit` on
`busybox:1.37`.
Switches every productive `lookup('image', ...)` to
`lookup('config', '<role>', 'services.<svc>.image|version')` — the
same path templates like `web-app-oauth2-proxy` already use, which
resolves through the merged `applications.*` map and so transparently
picks up the GHCR mirror override written by the inventory creator.
Removes the now-unused `plugins/lookup/image.py` plugin and its unit
test (zero remaining consumers across roles, scripts, and tests).
Rewrites the image-origin / image-mirror / dockerfile docs to teach
the `lookup('config', ...)` pattern, dropping references to the
deleted plugin and the dead `images_overrides.*` variable. Doc edits
follow the `documentation.md` rules (no em dashes, one sentence per
line, no historical phrasing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `make compose-playwright role=<role> [pw="..."] [keep=true]` as the
canonical inner-loop entry point for rerunning a role-local Playwright
spec against the live stack — replaces the raw
`scripts/tests/e2e/rerun-spec.sh` invocation across docs.
Defines a small comment-block schema for Makefile targets (description
line on top, followed by `Usage:` / `Example:` / `Note:` / `Param <name>:`
markers). Lint enforces it; `make help` shows the description per target
in the short list and `make help target=<name>` prints the full block.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Catches refactor drift across `.sh`, `Makefile` recursive calls, and
GitHub workflows. Includes fixes for the 5 bugs the new test surfaced
on first run: `make -C ROOT down`/`up` in the network stack refresh
helper (missed during the compose-* rename) and four
`make compose-deploymode=…` / `compose-deployapps=…` typos in
07/08/09_deploy_*.sh plus three docstring/echo references
(missing space after `compose-deploy`, introduced by the
deploy → compose-deploy sed-rename).
Reorder every target block in the Makefile so the definitions land in
strict ASC order — the order broke during the cascading domain-prefix
renames (install-agent, security-apparmor-*, etc.) and now matches what
test_makefile_targets_sorted_ascending requires.
Rewrite the two for-loops in test_phony_declarations as list-comprehension /
list.extend forms so ruff PERF401 stops complaining and the file passes
autoformat.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Place a single `.PHONY: <target>` line directly above each target's
comment block instead of clustering all phony declarations at the top
of the Makefile. The `help.sh` parser already resets its comment state
on lines beginning with `.`, so the PHONY entry must sit ABOVE the
comment (not between comment and target) to keep `make help` accurate.
Add the previously-missing PHONY entries for `dotenv` and `dotenv-force`
so every task target carries one.
Introduce `tests/lint/filesystem/makefile/test_phony_declarations.py`
to enforce two invariants going forward:
- every target has a matching `.PHONY: <target>` above its comment, and
- every `.PHONY: <target>` references an actually-defined target (no orphans).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the seven separate deploy-* make targets with a single `make
deploy` recipe that maps short Make variables to the canonical
INFINITO_* environment variables and dispatches via the new
scripts/tests/deploy/local/deploy/main.sh router. main.sh and all
sub-scripts continue to read the long-form names, so Ansible passthrough
and container forwarding stay unchanged.
Short Make CLI now exposes apps, mode, purge, type, bundles, disabled,
and full_cycle as aliases. The router picks the verb folder
(initialize, reinstall, update) and scope (all, selection) based on the
chosen mode plus the presence of INFINITO_APPS or INFINITO_BUNDLES.
Also folds in the global INFINITO_TEST_DEPLOY_TYPE -> INFINITO_DEPLOY_TYPE
rename across default.env, passthrough handler, dotenv test, CI deploy
workflows, discovery scripts, and the server stage task.
Active contributor and agent-iteration docs, role-level Administration
and README files, the bundles README, and the deploy reset helper are
all retargeted to the new short CLI. docs/requirements/ entries stay
untouched as historical audit logs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
