The plugin's previous
domain_primary = str(vars_.get("DOMAIN_PRIMARY") or "")
domain_homepage = str(vars_.get("DOMAIN_HOMEPAGE") or "")
read inventory values that may carry Ansible 2.18+'s
``_AnsibleTaggedStr`` with the ``TrustedAsTemplate`` tag — exactly
the shape ``inventories/development/default.yml:20`` produces with
DOMAIN_PRIMARY: "{{ lookup('env', 'INFINITO_DOMAIN') }}"
A bare ``str(value)`` casts the tag away and leaves the literal Jinja
text in the plugin output. The output then flows into
``CURRENT_PLAY_REDIRECT_DOMAINS`` via the set_fact in
``tasks/stages/01_constructor.yml`` and lands in
``roles/web-opt-rdr-domains/tasks/main.yml`` loop items as
item={'source': "{{ lookup('env','INFINITO_DOMAIN') }}", 'target': "{{ … }}"}
The ``lookup('tls', domain, 'enabled')`` call in
``roles/web-opt-rdr-domains/tasks/redirect-domain.yml:5`` then
receives the literal Jinja string and raises
The lookup plugin 'tls' failed: tls: application_id
'{{ lookup('env', 'INFINITO_DOMAIN') }}' not found in domains mapping
(job 77797371504 on run 26428080957). That CI ran on SHA d1410c0be,
which already shipped this plugin — so the upstream replacement of
the inline two-stage set_fact merges with the lookup plugin did not
fix the underlying trust-tag-strip bug; it just relocated it from
Jinja to Python.
Add ``_resolve_str`` that runs the value through ``self._templar``
before string-coercion. Idempotent for already-resolved strings,
exception-safe (falls back to ``str(raw)`` if the templar is
unavailable, e.g. on hosts without rendered vars). Both
``DOMAIN_PRIMARY`` and ``DOMAIN_HOMEPAGE`` route through it.
Defense-in-depth with the existing ``plugins/lookup/tls.py``
defensive templating (commit f32266d53): even if a future code path
hands ``lookup('tls', …)`` an untemplated string, the tls plugin
itself now templates the first term. Either layer alone suffices for
the rdr-domains failure; both together guarantee the symptom cannot
re-surface from a single regression elsewhere.
Coverage: two regression tests in
``tests/unit/plugins/lookup/test_current_play_redirect_domains.py``
— ``test_jinja_tagged_domain_primary_is_templated`` proves the
templar call happens and produces the resolved domain, and
``test_templar_failure_falls_back_to_raw_value`` proves a raising
templar does not crash the plugin. The 10 existing tests stay green.
``make test`` green across all 5 targets.
Class follow-up: 11 further lookup plugins read inventory values
without ``templar.template`` first (see git log for ``vars_.get``
call sites). Not in scope for this fix; logged for a separate
audit pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The inventory validator at cli/administration/inventory/validate
walked `recursive_keys(host_vars) ⊆ recursive_keys(application_defaults)`
and flagged any extra key with "Missing default for <app>: <key>".
After commit 10cb02461 ("bake only variants.yml overrides into
matrix-deploy --vars") host_vars carry every key the active
variants.yml overlay declares. Since 29377e1bd
(`_build_application_defaults` reads only `meta/<topic>.yml`, not
variants.yml — to keep the variant-0 service-key leak closed), the
defaults' allow-set is artificially narrower than the role contract:
roles/web-opt-rdr-www declares `services.dashboard.{enabled,shared}`
exclusively in `meta/variants.yml`, so the validator rejects the
baked host_vars with three "Missing default" entries before
ansible-playbook even starts (job 77797115291 on run 26428083128,
distro=debian; distro-agnostic per the matrix).
Widen the validator's legal-key set per app to
recursive_keys(defaults[app]) ∪ ⋃ recursive_keys(variants[app][i])
variants.yml IS part of the role contract; the validator's previous
"defaults only" view was incomplete. Runtime
`_build_application_defaults` stays narrow (so 29377e1bd's
variant-0-leak fix stays in place) — only the validator widens. Typos
not declared by any variant remain caught: a new
`test_key_typo_in_variants_still_rejected` unit test pins that.
Package split (KISS/DRY/SPOT), mirroring the credentials/ refactor:
`validate/__main__.py` carried six orthogonal concerns plus
orchestration in 175 LoC. Split:
- `validate/keys.py`: `recursive_keys` (pure helper).
- `validate/loaders.py`: `load_yaml_file` (with vault-block stripping)
and `load_inventory_files` (directory scan).
- `validate/applications.py`: `compare_application_keys` (the widened
variants-aware logic).
- `validate/users.py`: `compare_user_keys`.
- `validate/hosts.py`: `validate_host_keys`.
- `validate/__main__.py`: thin orchestrator — argparse + load defaults/
variants/users + invoke compare_* + exit code. 75 LoC.
`validate/__init__.py` re-exports the public API
(`main`, `compare_application_keys`, `compare_user_keys`,
`load_inventory_files`, `load_yaml_file`, `recursive_keys`,
`validate_host_keys`, `PROJECT_ROOT`). External callers
(`tests/unit/cli/administration/inventory/validate/test_main.py`,
`tests/unit/cli/administration/inventory/validate/test_hosts.py`)
keep working unchanged via this re-export.
`__main__.py` uses ABSOLUTE imports (`from cli.administration.inventory.
validate.applications import ...`) instead of relative because it's
invoked as a subprocess by tests and by the deploy entrypoint; the
sys.path bootstrap at the top of the file makes `cli.…` resolvable
in that mode.
Test layout follows the split — one test file per module:
- `test_keys.py` (6 tests: empty/non-dict/flat/nested/deep/prefix).
- `test_loaders.py` (8 tests: vault stripping, missing file, broken
YAML, top-level + recursive directory scan, empty-block skip).
- `test_applications.py` (9 tests: all-keys-match, unknown app, extra
key, credentials skip, variants legal, variants union, no-variants
fallback, source path in error, typo still rejected).
- `test_users.py` (5 tests: match, unknown user warn, missing default,
password skip, credentials skip).
- `test_hosts.py` (renamed from test_inventory_hosts.py — unit tests
for `validate_host_keys`).
- `test_main.py` (renamed from test_inventory.py — end-to-end
subprocess tests for `main()`, including the
`test_variants_only_key_is_accepted` regression guard for this fix).
Coverage: 36 tests, all green. `make test` green across all 5 targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The credentials baker at cli/administration/inventory/credentials/__main__.py
walked `schema_creds.items()` one level deep and called `str(value)` on
every leaf. For nested credential groups like
``credentials.recaptcha = {key, secret}`` (declared in
roles/web-app-espocrm/meta/schema.yml and
roles/web-app-listmonk/meta/schema.yml) this collapsed the inner dict
into its Python repr, vault-encrypted that blob, and wrote it as a
single scalar at ``credentials.recaptcha``. The runtime ``config``
lookup then choked on the descent into ``recaptcha.key``:
The lookup plugin 'config' failed: Expected dict for 'key', got
_AnsibleTaggedStr at 'key'
Full path so far: applications['web-app-espocrm'].credentials.recaptcha.key
Current object: "{'key': '{{ CAPTCHA.RECAPTCHA.KEY }}', 'secret':
'{{ CAPTCHA.RECAPTCHA.SECRET }}'}"
(jobs 77797371397 espocrm and 77797371442 listmonk on run 26428080957.)
Introduce ``emit_credentials`` (new module: ``credentials/emit.py``)
that walks the schema recursively. Scalar leaves still vault-encrypt
via ``to_vault_block``; dict nodes recurse into a nested CommentedMap
via ``ensure_map``. The walker also tracks dotted full-key paths so
the existing ``--force``-mode override-apply logic sees the same dotted
keys it always saw.
Auto-healing for previously-corrupted host_vars: ``ensure_map``
overwrites non-CommentedMap placeholders, so the next bake replaces a
stringified ``credentials.recaptcha`` blob with the correct nested
structure. Operators with corrupted entries simply re-run the
credentials generator; no manual cleanup needed.
Package refactor (KISS/DRY/SPOT):
The 450+ LoC ``__main__.py`` was carrying four orthogonal concerns
(vault, --set parsing, recursive emission, prompts) plus orchestration.
Split by concern:
- ``credentials/vault.py``: ``is_vault_encrypted``,
``is_ruamel_vault``, ``to_vault_block`` (+ private helpers
``_vault_body``, ``_make_vault_scalar_from_text``).
- ``credentials/overrides.py``: ``parse_overrides``, ``override_for``.
- ``credentials/emit.py``: ``ensure_map``, ``emit_credentials``.
- ``credentials/prompts.py``: ``ask_for_confirmation``.
- ``credentials/__main__.py``: argparse + ``main`` + two thin mode
handlers (``_run_snippet_mode``, ``_run_default_mode``) + the
force-override helper.
``credentials/__init__.py`` re-exports the public API
(``main``, ``ask_for_confirmation``, ``parse_overrides``, etc.) so
external callers (notably ``tests/unit/cli/administration/inventory/
test_credentials.py``) keep working unchanged.
Test layout follows the split:
- ``tests/unit/cli/administration/inventory/credentials/test_vault.py``
(20 tests: vault detection, body extraction, scalar construction,
to_vault_block paths, plus the VaultHandler subprocess coverage).
- ``tests/unit/cli/administration/inventory/credentials/test_overrides.py``
(13 tests: parse_overrides + override_for incl. dotted nested keys).
- ``tests/unit/cli/administration/inventory/credentials/test_emit.py``
(10 tests: ensure_map placeholder replacement, recursive emission,
none → empty, skip_existing semantics, override precedence,
dotted-path tracking, the on-disk corruption recovery case).
- ``tests/unit/cli/administration/inventory/credentials/test_prompts.py``
(5 tests: ask_for_confirmation yes/no/garbage/case/whitespace).
``tests/unit/cli/administration/inventory/test_credentials.py`` keeps
only the end-to-end ``main()`` integration coverage (including
``test_nested_credentials_dict_recurses_instead_of_stringifying`` —
the explicit regression guard for the original failure mode).
Coverage: 47 new per-module unit tests + the 5 existing main()
integration tests, all green. ``make test`` green across all five
targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CI job 77797371504 on run 26428080957 crashed with:
tls lookup: application_id '{{ lookup('env', 'INFINITO_DOMAIN') }}'
not found in domains mapping
`DOMAIN_PRIMARY` is defined in inventories/development/default.yml:20
as `"{{ lookup('env', 'INFINITO_DOMAIN') }}"`. On Ansible 2.18+ that
string carries a `TrustedAsTemplate` tag (it is an
`_AnsibleTaggedStr`). The constructor at tasks/stages/01_constructor.yml:64
wrapped it in `[ DOMAIN_PRIMARY ] | flatten | map('string') | first`
— the explicit `str()` cast strips the trust tag. The resulting plain
string flowed into `CURRENT_PLAY_REDIRECT_DOMAINS.source` as the
literal text `{{ lookup('env', 'INFINITO_DOMAIN') }}`, was passed
through `domain: "{{ item.source }}"` in
roles/web-opt-rdr-domains/tasks/main.yml:14 (one render pass that
produces untrusted content), and finally reached `as_str(terms[0])`
in plugins/lookup/tls.py:39 unchanged. The tls plugin never templated
its arg, so `resolve_term` looked up the literal Jinja string in the
domains mapping and raised.
Two layers, both necessary:
1. plugins/lookup/tls.py:39 — defensively template `terms[0]` through
`self._templar` before string-coercion. Mirrors the idiom in
plugins/lookup/unit_name.py:33 and plugins/lookup/email.py:61. Any
future caller that hands the plugin an unresolved expression now
gets it resolved before the domains-map lookup.
2. tasks/stages/01_constructor.yml:64-65 — drop the
`[ X ] | flatten | map('string') | first` ritual for
`DOMAIN_PRIMARY` and `DOMAIN_HOMEPAGE`. Both are scalar strings in
every known inventory (group_vars/all/00_general.yml:41,43 +
inventories/development/default.yml:20); the over-defensive
wrapper was the trust-tag stripper. Direct reference preserves the
tag through `CURRENT_PLAY_REDIRECT_DOMAINS` and every downstream
consumer.
Downstream consumers audited; all transparent to `_AnsibleTaggedStr`
(it inherits from `str`):
- merge_mapping filter: content-based dict-key compare
- domain_mappings filter: takes `DOMAIN_PRIMARY` directly, no cast
- items2dict(key_name='source', value_name='source'): hashable
- generate_all_domains filter: `isinstance(item, str)` matches
- sys-stk-front-base `when: ... | length > 0`: only counts
- web-opt-rdr-domains -> tls lookup (the failing path): now templates
Coverage: new `test_term_with_unresolved_jinja_is_templated` in
tests/unit/plugins/lookup/test_tls.py verifies the lookup feeds its
first term through the templar. The 12 existing tls tests stay
green. `make test` green across all 5 targets.
Class follow-up (not addressed here): the
`| string` / `| map('string')` cast still strips `TrustedAsTemplate`
elsewhere if a future commit reintroduces it; 12 other lookup
plugins do not yet defensively template their args. A lint guard and
plugin-wide audit will come in separate PRs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pruned per project convention "no unnecessary code comments":
- module docstring: 23 lines → 1 line (the file name + symbol already
state intent; full incident write-up belongs in the commit log)
- wipe_database_entries docstring: 11 lines → 2 lines
- six inline rationale comments in tests/__main__ removed (test
names already encode the intent)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
After 10cb02461 switched `payload._resolve_variant_payloads` from
`get_variants` to `get_variant_overrides_only`, matrix-deploy bakes
only the raw `meta/variants.yml` slice into host_vars (typically just
`services:`). Lookups that read `variables['applications']` raw no
longer see content from `meta/rbac.yml`, `meta/server.yml`, etc.
`rbac_group_path` was the last lookup still going through the raw
host_vars read. For joomla (whose `meta/rbac.yml` declares the
`editor` role) the post-10cb02461 host_vars carried no `rbac` block
at all, the lookup walked an empty `rbac.roles` map, and the deploy
crashed with "rbac_group_path: role 'editor' is not declared under
applications[web-app-joomla].rbac.roles. Declared roles:
['administrator']" (job 77797371433 on run 26428080957).
Route `_get_application` through `get_merged_applications` so the
meta-merged view (services + server + rbac + volumes + schema
credentials + variant overrides) is restored. This mirrors the
existing pattern in plugins/lookup/{config,tls,nginx,domain}.py and
keeps the mirror-override intent of 10cb02461 intact: the bake step
still emits only the variant overrides; the merged cache assembles
the full view at lookup time.
Test updates:
- TestRbacGroupPathLookup.setUp patches get_merged_applications to
read from variables['applications'] so the existing 14 tests stay
hermetic without disk-reading roles/*/meta/*.
- New TestRbacGroupPathReadsMergedApplications regression case
exercises the exact 10cb02461 scenario: host_vars only carry a
variants.yml slice, the merged cache supplies the rbac block, the
lookup returns the correct path.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Matrix-deploy variant transitions wipe a consumer's compose stack and
its named DB volume (`compose down -v` in
scripts/container/purge/entity/compose.sh) but `databases.csv` at
$FILE_DATABASE_SECRETS survives the purge. When the next round drops
the consumer from its inventory, the stale baudolo-seed row remains
behind. The subsequent backup run reads that row, attempts
`mariadb-dump -u<entity> -p<old_pw>` against the freshly initialised
RDBMS user that no longer exists, and the systemd unit fails with
"Access denied for user 'matomo'@'127.0.0.1' (using password: YES)"
(job 77797356000 on run 26428080957: matomo dump in
svc-bkp-container-2-local variant 1, where round 1 seeded matomo but
round 2 only deploys postgres + mariadb + svc-bkp itself).
Add `utils.cleanup.databases_csv` mirroring the existing
`utils.cleanup.tokens` pattern: parse the baudolo-seed CSV format
(';'-separated, four columns instance/database/username/password, with
header — verified against baudolo.seed.__main__:check_and_add_entry)
and strip every row whose `database` or `username` column matches an
entity derived from the supplied app ids via
utils.roles.entity_name.get_entity_name. Header and blank lines are
preserved verbatim; the file is rewritten only when at least one row
was removed.
Wire the helper into scripts/container/purge/apps.sh right after the
token-store wipe so every matrix-deploy inter-round purge invokes it
alongside the other cleanups. Operator-triggered purges
(`INFINITO_APPS=<app> bash scripts/container/purge/apps.sh`) also get
the cleanup — that is a behaviour change relative to before, where
the row would survive an operator-triggered entity teardown.
Coverage: 12 unit tests in tests/unit/utils/cleanup/test_databases_csv.py
covering the happy path, the multi-app case, the
`database='*'`/username-match case, the unknown-app no-op, header and
blank-line preservation, the all-rows-removed case, malformed short
rows, and the legacy-header-less file. `make test` green across all 5
targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- web-opt-rdr-www canonical: rdr-www.{DOMAIN_PRIMARY} -> w3redirect.{DOMAIN_PRIMARY}.
- New lookup `www_redirect_domains` (role-scoped): returns full {source, target}
mapping list, resolves DOMAIN_HOMEPAGE target for the role's own canonical via
lookup('domain', 'web-opt-rdr-www'). REDIRECT_WWW_DOMAINS now carries dicts;
tasks/main.yml and _01_cloudflare_edge_redirect.yml updated to consume dicts.
- New lookup `current_play_redirect_domains`: replaces the two inline
set_fact merges in tasks/stages/01_constructor.yml. Merges user-supplied
redirect_domain_mappings, primary-redirect (conditional on web-opt-rdr-domains
in deployed), and per-app domain mappings (inlined from the now-removed
domain_mappings filter).
- domain_redirect_mappings filter removed; web-app-keycloak RedirectUrisTest
pollution of utils.roles.applications.config in sys.modules fixed via
tearDownClass cleanup.
- csp_skip_domains lookup: csp.health.skip opt-out path removed (architecture
no longer needs it).
- Lint test_only_web_app_roles_integrate_dashboard expanded from web-app-* to
web-*; test_role_and_group_vars_used now scans .py files too so vars consumed
only inside plugins (DOMAIN_HOMEPAGE, AUTO_BUILD_ALIASES) are seen.
- New role-scoped variants.yml on web-opt-rdr-www with dashboard on/off.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Since 576fad84f (2026-05-10) moved role images from
`defaults/main.yml` into `meta/services.yml`, matrix-deploy's variant
baker has been writing the FULL effective config (services.yml +
variants.yml deep-merged via `get_variants`) into host_vars through
the `--vars` JSON consumed by `apply_vars_overrides`. That populates
`applications.{role}.services.{svc}.image` (and `.version`, .cpus,
etc.) with the upstream value, which then silently disables
`apply_mirror_overrides` for every role with an explicit image
declaration: the default `if_missing` policy short-circuits when the
field is already non-blank, so the rendered compose / Dockerfile FROM
keeps pulling from the upstream registry rather than the GHCR mirror.
The regression went unnoticed because Docker Hub usually serves the
pull, but it eventually surfaced as a BuildKit 429 on
`docker.io/library/php:8.2-apache` during the suitecrm Dockerfile
build (job 77602110531 on run 26362676803). Nextcloud, moodle,
mariadb consumers, postgres consumers, simpleicons, phpmyadmin,
phpldapadmin, jenkins, baserow, fider, ... all share the same latent
defect — only suitecrm happened to hit the rate limit first.
Fix the root cause instead of papering over per-role with
`mirror_policy: force`: introduce `get_variant_overrides_only` next
to `get_variants` and switch payload.py to it. The new function
returns the raw `meta/variants.yml` entries per role — no merge with
`meta/services.yml`. matrix-deploy now bakes only the variant-
specific overrides into host_vars; the role's services.yml fields
flow through the normal Ansible lookup chain at deploy time, leaving
`image`/`version` blank in host_vars so `apply_mirror_overrides`
populates them from `mirrors.yml` under its default `if_missing`
policy.
Behavior preserved for the other `get_variants` consumers:
* utils/manager/inventory.py:104 (schema-driven credentials) keeps
the full merged variant payload.
* planner.py:58 (variant-count only) is unaffected.
* legacy_resolver.py:42 (CombinedResolver runtime services_overrides)
keeps the full merged view.
Coverage:
* 3 new tests in test_applications.py for the new helper (single
empty variant when no meta/variants.yml, raw override without base
merge, cache invariance).
* test_payload.py + test_builder.py mocks repointed from `get_variants`
to `get_variant_overrides_only`.
* `make test` green across all 5 targets (lint, unit, integration,
external, test-lint) — 188 tests total.
Side-effects assessed:
* Repo-wide grep confirms no template / task accesses
`applications.X.services.Y.image` directly via hostvars — every
consumer goes through `lookup('config', …)` which still deep-merges.
* Roles previously pulling upstream now resolve via the GHCR mirror
whenever the image is already synced (the `images-mirror-missing.yml`
workflow keeps the destination populated).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`_apply_one_role_special_rules()` in utils/manager/inventory.py used
`bool(resolve_database_service_key(...))` to decide whether to
auto-generate a `database_password` credential for a role. The
resolver raises when multiple RDBMS keys are enabled — which is the
correct contract for compose_volumes / container_depends_on /
plugins.lookup.database (they need an unambiguous single key) but the
wrong contract for inventory's credential-generation pass: variant 1
of svc-bkp-container-2-local intentionally enables both mariadb and
postgres to exercise the engine-side dump path against both engines
at once, and the deploy aborted with `ValueError: multiple direct
database services are enabled: mariadb, postgres` during inventory
generation (job 77602095377 on run 26362676803).
Add a non-raising sibling helper `has_single_database_service()` that
returns True iff exactly one RDBMS service is enabled, and switch
inventory's call site to it. Multi-DB roles now skip the auto-
generated `database_password` (which was a phantom credential for
svc-bkp anyway — no schema.yml, no cross-role lookup; the role's own
`04_seed-database-to-backup.yml` receives `database_password` as a
caller-passed task var, not as a self-credential lookup).
`resolve_database_service_key` keeps its strict raise semantics for
the compose_volumes / container_depends_on / database-lookup callers,
which still need a single unambiguous key.
Add unit coverage for both functions (3 + 4 tests) in
tests/unit/utils/roles/applications/services/test_database.py.
Static check only (per operator request).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`generate_all_domains(include_www=True)` synthesizes `www.X` for every
canonical domain X and pushes them into CURRENT_PLAY_DOMAINS_ALL, but
they are never registered in the global canonical_domains_map or in
applications[*].server.domains. The strict `tls`/`cert` lookups
therefore raised `domain 'www.X' not found (domains/applications)`
when web-opt-rdr-www iterated REDIRECT_WWW_DOMAINS and called
`lookup('tls', 'www.X', 'enabled')` in the redirect-domain.yml TLS
gate — failing job 77602110568 on run 26362676803 (and cascading
through sys-front-tls/vars/main.yml, sys-front-tls/tasks/02_http.yml
and sys-front-tls-selfsigned/tasks/01_core.yml, all of which call
`lookup('tls', application_id, …)` with the same leaked www.X).
Add a single fallback branch to `resolve_term`: if the strict lookup
fails AND the term starts with `www.`, retry both resolver paths
(legacy reverse + applications) against the bare domain. The fix is
strictly additive — every previously successful term and every non-www
miss preserves its current behavior.
Coverage:
* test_www_prefix_falls_back_to_bare_domain
* test_www_prefix_with_unknown_bare_still_raises
* test_non_www_unregistered_still_raises (regression guard)
Verified statically via the in-container `resolve_term`:
www.infinito.example -> (svc-prx-openresty, infinito.example)
www.rdr-www.infinito.example -> (web-opt-rdr-www, rdr-www.infinito.example)
The global self-signed cert SAN list (cert.py:193) is built from
CURRENT_PLAY_DOMAINS_ALL, so it already covers the www.X variants —
nginx serves the redirect vhost with that cert and the TLS handshake
on the hlth-webserver check matches.
Static check only (per operator request).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Flatten records from all summaries and sort by ISO-Z timestamp so the
step-summary table is chronological across apps/variants/passes instead
of grouped per file.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`_build_application_defaults` returned variant 0 of every role with a
meta/variants.yml, which then deep-merged with the variant-N override
that build_dev_inventory bakes into the inventory. Any service flag
that variant N did not explicitly disable carried through from
variant 0 into the runtime applications map. The most visible casualty
was svc-bkp-container-2-local variant 1 (postgres + mariadb only),
where gitea + keycloak leaked in via variant 0, sys-service-loader
pulled in svc-db-openldap via keycloak.services.ldap, and the deploy
crashed in sys-svc-compose/04_files.yml on openldap's env.j2 with
"credentials is defined in schema at roles/svc-db-openldap/meta/schema.yml
but not set in application 'svc-db-openldap'".
Defaults now come from each role's variant-free base config
(meta/<topic>.yml plus schema-defaults, no variants overlay). Variants
stay accessible via get_variants for any caller that explicitly needs
a specific variant. Inventory-level applications.<app> overrides win
exactly as before.
Supporting changes that keep the variant matrix consistent under the
new defaults model:
- New lint guard tests/integration/roles/meta/variants/
test_non_baseline_explicit_disables.py: for every dynamic
service-key declared in a role's meta/services.yml that variant 0
pins to literal true, every non-baseline variant MUST explicitly
pin the same key to either literal true or to literal
enabled: false + shared: false. Statically-enabled keys are exempt
(same rule as test_auth_isolation.py).
- roles/web-app-{fusiondirectory,lam,nextcloud,phpldapadmin}/meta/
variants.yml: explicit `false` pins for dashboard / email / logout /
matomo / prometheus / talk in restrictive variants so deep-merge no
longer hands them the deleted variant-0 inheritance.
- roles/web-app-moodle/templates/playwright.env.j2: gate the LAM
block on `lookup('config', application_id, 'services.lam.enabled',
false)`. LAM is a transitive helper, not a moodle-owned service, so
the default-bearing lookup tolerates the missing path when the
variant does not deploy LAM.
- tests/unit/utils/cache/test_variants.py: rename the legacy
"always variant zero" coverage to TestMergedApplicationsUsesBaseConfig
and add test_get_application_defaults_omits_variant_zero_only_keys
as the regression guard for the leak this commit fixes.
- docs/contributing/artefact/files/role/variants.md +
docs/contributing/design/variants.md: variant 0 now manually
enumerates every dynamic services.yml key with
enabled: true + shared: true; `{}` is only permitted when the role
has no dynamic-enabled service-key.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
InventoryManager.resolve_schema_includes_recursive previously read the
root role's base meta/services.yml when discovering shared providers,
regardless of which matrix variant the planner had pinned for the
round. For a role whose variants.yml enables additional services per
variant (e.g. svc-bkp-container-2-local variant 2 enabling
services.ldap), the base config had no `enabled+shared` flag for those
services, so the recursive resolver never visited the variant's
shared providers — their credentials were never generated, and the
deploy crashed in the provider's env.j2 with "credentials is defined
in schema at roles/<provider>/meta/schema.yml but not set in
application '<provider>'".
Wires the active variant through the existing inventory pipeline:
- utils/manager/inventory.py: InventoryManager(variant=None|int). When
set, load_role_config_by_path overlays variants.yml[N] onto the root
role's meta config (only for the root — transitive providers keep
their own base lookup). variant=None preserves the pre-refactor base
behavior for every standalone caller.
- cli/administration/inventory/credentials: new --variant N flag,
forwarded to InventoryManager.
- cli/administration/inventory/provision: new --app-variants <JSON>
flag ({app_id: variant_index}), forwarded per-app to the credentials
subprocess.
- cli/administration/deploy/development/inventory/builder.py: emits
--app-variants from spec.variant_selectors() when the round pins any
variant. Apps without variants.yml collapse to a single empty
overlay in get_variants so this is a no-op for them.
10 unit tests cover: variant overlay for root role, variant=None
fallback to base, out-of-range fallback, no overlay on non-root
paths, --variant CLI forwarding (set + default), --app-variants
generator forwarding (set + omit), builder --app-variants emission
(set + omit).
Caveat: bkp-c2l CI variant 1 still loads svc-db-openldap transitively
at runtime (outside the planner's variant closure) — a separate
sys-service-loader / role-meta bug, not addressed here.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The compose templates appended additional depends_on entries (e.g. taiga
rabbitmq services, openproject cache/seeder, mailu resolver/front) on
column 6 directly after the lookup call. When the lookup returned ""
(no DB and no redis), those entries dangled without a depends_on header
and broke YAML parsing.
Move every such trailing block into the lookup's `extra` kwarg so the
header is always emitted as soon as any entry exists. Adds a regression
test for the extras-only case.
Affected: taiga, mailu, magento, pixelfed, openproject, espocrm.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Eliminate include-style depends_on assembly. The lookup resolves DB/redis
inductively from applications config and returns indented YAML (or "" when
no entries), removing the empty `depends_on:` block bug in libretranslate.
Migrates 43 compose templates from `{% include … dmbs_*.yml.j2 %}` to
`{{ lookup('container_depends_on', application_id) }}`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`status` was too vague for what the lookup returns (apps in current
play). Rename to `deployment` and add two new fields:
* `runtime` — RUNTIME variable value
* `deployed` — runtime-aware authoritative set: equals `running`
(whitelist or groups) for ephemeral CI runtimes (act, github),
falls back to `groups` for persistent runtimes (host, dev). This
matches reality: in CI the container is fresh so only whitelisted
apps exist; in production a partial deploy still leaves prior
apps live, so groups is the right set.
Switch the two webserver/csp healthchecks to `lookup('deployment').deployed`
so they only probe domains for apps that actually exist in the current
deploy. In production behaviour is unchanged.
Update remaining consumers (cli/build/include, web-app-keycloak,
test-e2e-playwright) for the rename. Existing `status` semantics
(`whitelist`, `running`, `groups`, `all`) preserved under the new
name.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cluster 2a (hlth-msmtp failed=65 in universal single-app deploys):
the health-check script sent "To: administrator" but the administrator
Unix user is only provisioned in server-pass, so postfix rejected the
recipient. Replace the inline Jinja conditional with a role-local
lookup plugin that picks administrator.email (external), "administrator"
(local + user exists) or "root" (local + missing).
Side change: SIM105 cleanup in utils/install/lint/__main__.py.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fixes Cluster 3a ca-inject YAML parse: when a role variant disables its
DB (enabled=false), the depends_on template emitted ` :` because
the existing `not shared` guard fired for a non-existing DB host.
The new derived flag mirrors `service.required` (= enabled AND shared)
and is consumed in:
- depends_on/dbms_base.yml.j2, dmbs_excl.yml.j2 (the actual fix)
- sys-svc-rdbms postgres/mariadb service templates
- web-app-discourse, web-app-snipe-it (latent same-pattern defensives)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements docs/requirements/021-sso-flavor-migration.md atomically:
* Schema: every role's meta/services.yml replaces the parallel
`oauth2:` / `oidc:` top-level blocks with a single `sso:` block
carrying `enabled`, `shared`, and a `flavor` discriminator
(`oidc` | `oauth2` | `saml`; default `oidc`). Flavor-specific
sub-keys live under `sso.<flavor>.*` (`sso.oauth2.{origin,acl,
allowed_groups}`, `sso.oidc.plugin`).
* Provider role: roles/web-app-oauth2-proxy/ is deleted in its
entirety; its 5 sidecar templates move under
roles/web-app-keycloak/templates/sso_proxy/, and the per-consumer
rendering task lives at roles/web-app-keycloak/tasks/sso_proxy.yml.
web-app-keycloak now declares `provides: sso`.
* Runtime: new resolver utils/roles/applications/services/sso.py
centralises the SSO predicates (`is_enabled`, `is_proxy_gated`,
`is_oidc_native`) and the oauth2 sub-values
(`oauth2_origin_{host,port}`, `oauth2_acl`, `oauth2_allowed_groups`).
Three lookup plugins consume the resolver: `sso` (single-app
predicate access), `sso_proxy_consumers` (enumerates oauth2-flavored
consumers), `sso_oidc_plugin` (renamed from `oidc_flavor`, Nextcloud
plugin selector). sys-stk-backend, sys-svc-compose, sys-svc-proxy
and plugins/filter/compose_volumes.py all gate on
`lookup('sso', ..., 'is_proxy_gated')` instead of manually combining
`enabled` and `flavor`.
* Identifier renames repo-wide: `services.<entity>.ports.local.oauth2`
→ `ports.local.sso`; `PORT_BANDS.local.oauth2` → `PORT_BANDS.local.sso`;
`OAUTH2_PROXY_*` env/Jinja vars → `SSO_PROXY_*`;
`oauth2_proxy_cookie_secret` → `sso_proxy_cookie_secret`;
`OAUTH2_SERVICE_ENABLED` + `OIDC_SERVICE_ENABLED` collapse into
`SSO_SERVICE_ENABLED`; Playwright gating helpers
(isServiceEnabled/skipUnlessServiceEnabled/requireService/safeIsEnabled/
safeSkipUnlessEnabled) move from `oidc`/`oauth2` to `sso`.
* Dual-block resolution: per req's Background-Table, web-app-bookwyrm
is `flavor: oauth2`; web-app-gitea and web-app-friendica keep
`flavor: oauth2` with their existing `oauth2.acl.blacklist`
(/user/login) and `oauth2.acl.whitelist` (federation + discovery
endpoints) — preserving the operative oauth2-proxy gating.
* Tests: the two mutual-exclusion guards
(tests/integration/iam/oauth2_oidc/test_mutual_exclusive.py and
test_acl_mutual_exclusive.py) are deleted per Decision #10;
test_oauth2_contract.py is renamed and rewritten to
test_sso_contract.py, asserting that `flavor: oauth2` roles carry
`oauth2.origin.{host,port}` plus `ports.local.sso`; test_sso.py
(web-app integration) now requires the unified `sso:` block with an
explicit flavor or a `# nocheck: sso` opt-out; new
tests/lint/repository/no_legacy_sso_paths/ guards the source tree
against re-introducing the legacy strings.
* Docs: req 21 itself has every AC checked off, and reqs 006, 008,
009, 013, 015, 016, 019, 020 plus 10+ role READMEs and
docs/contributing/actions/testing/suppression.md are updated to the
unified SSO naming.
Empirically validated end-to-end against the validation app set
(web-app-keycloak, web-app-nextcloud, web-app-bookwyrm, web-app-gitea,
web-app-friendica, web-app-dashboard) in matrix variants 0 and 1; v2
also passes the 4 target roles' Playwright after a documented
pre-existing KC_HOSTNAME-drift workaround (separate concern, not
caused by this refactor).
BREAKING CHANGE: every consumer of `services.oauth2.*`,
`services.oidc.*`, `OAUTH2_PROXY_*`, `OAUTH2_SERVICE_ENABLED`,
`OIDC_SERVICE_ENABLED`, `oauth2_proxy_cookie_secret`,
`oauth2_proxy_application_id`, `ports.local.oauth2`, the
`web-app-oauth2-proxy` role, or the legacy gating helper service names
(`oidc`/`oauth2`) must migrate to the unified `sso` shape documented
in docs/requirements/021-sso-flavor-migration.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`categories_of()` now emits the sub-level key with the same hyphen
the role-id already uses (`web-app`, `svc-bkp`, `sys-ctl`) instead of
the dot-form (`web.app`, `svc.bkp`, `sys.ctl`). Reads cleanly against
the role naming convention and removes the only place in the
matching layer where role-id segments got rewritten.
Updates the matching unit tests and the only consumer annotation
that referenced the dot form (`sys-ctl-cln-bkps`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Consolidates the backup-role topology around four directional roles
under the existing svc.bkp category:
sys-ctl-bkp-docker-2-loc → svc-bkp-container-2-local
svc-bkp-loc-2-usb → svc-bkp-local-2-device
svc-bkp-rmt-2-loc → svc-bkp-remote-2-local
sys-bkp-provider → deleted (purely a bundler with no own logic;
the two pieces it pulled in — user-backup
and sys-timer-cln-bkps — are now declared
directly by callers that need them)
Every renamed role moves into svc.bkp (already invokable per
roles/categories.yml), gets an `application_id` in vars/main.yml, and
declares the per-app primary-entity service flags + resource limits the
invokable-role lint requires.
Companion changes:
- svc-bkp-container-2-local: meta/variants.yml declares three matrix
rounds (gitea+keycloak, postgres+mariadb, openldap) that pin the
companion topology via service-key flags the CombinedResolver
translates into provider-role includes.
- sys-svc-compose: container-2-local is no longer in the docker-mgmt
include loop; it joins via a separate `'svc-bkp-container-2-local'
in group_names` gate so the 3 backup roles are deployable
independently (each addresses its own failure mode).
- The three `04_seed-database-to-backup.yml` includes (web-app-matrix,
web-app-bigbluebutton, sys-svc-rdbms) are gated by the same
group_names check so consumer apps only seed when the container
backup is actually in scope.
- web-app-gitea picks up `enabled: true, shared: true, provides:
gitea` on its primary entity so the service registry can pull it in
via a `services.gitea.enabled+shared` flag (needed for the
container-backup variant 0).
- group_vars constants and the pyproject ruff per-file-ignore + the
.ansible-lint-ignore paths follow the role-id rename.
- The remote-2-local script picks up the snake_case rename
(`pull-specific-host.py` → `pull_specific_host.py`) and the
`os.makedirs` → `Path.mkdir` conversion so the role no longer needs
the PTH/N999 ruff escape hatch.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous regex anchored the `name` capture at line start and used
lazy `.+?`, which in production captured the full
`<ts> p=<pid> u=<user> n=ansible INFO| <role>` prefix as the role name
instead of just `<role>`. The CI step summary then rendered every row
with the timestamped log header and the `total` filter silently failed
(because `name` was no longer equal to `"total"`).
Add an optional non-capturing `(?:.*\bINFO\|\s+)?` group so the
ANSIBLE_LOG_PATH prefix is consumed when present and `name` captures
just the role token (or `role : task`). Test sample didn't carry the
log prefix, so the bug slipped through — add a prefixed-format test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Move role_is_invokable / role_has_required_by / role_has_nocheck out
of plugins/filter (where they were flagged as test-only filters) into
utils/roles/required_by_coverage.py. All three use load_yaml_any,
read_text, and ROLE_FILE_META_SERVICES from the canonical SPOTs.
- Refactor the verifier (cli.meta.roles.services.called) the same way:
iterate roles/*/ directly, look up meta/services.yml via the constant,
parse with load_yaml_any, and tolerate a missing roles_dir (unit tests
point at a fake /repo).
- Make the mailu-dependency integration test yaml-aware for meta/
services.yml: walk the parsed tree and skip values inside any
`required_by` subtree. `required_by` is inverse coverage metadata for
the deploy verifier, not a "consumer of mailu" reference, so previous
regex-on-text matching produced false positives for cert-deploy.
- sys-ctl-mtn-cert-renew is gated by `lookup('tls', application_id,
'mode') == 'letsencrypt'`, which the verifier can't express via
role-id/category match — mark nocheck with the reason inline.
- Add README.md for cli/meta/roles/services (CLI-category convention).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Asserts at deploy time that every role declaring `required_by` in its
meta/services.yml actually executed (at least one TASK block ending with
a non-skip status of `ok` / `changed` / `fatal` / `included`). Catches
silent coverage regressions where a once-flag swallows an include before
the role's body can run.
CLI: `python -m cli.meta.roles.services.called --logfile X --apps a,b,c`
Library: `cli.meta.roles.services.called.verify(...)` supports per-pass
slicing via `log_byte_offset` and reads either host-side log files or
in-container logs when `container=` is given.
Hooked into `cli.administration.deploy.development.deploy._run_deploy`
so matrix-deploy runs (CI + local make compose-deploy mode=initialize/
reinstall) snapshot the log offset before each compose.exec and verify
the resulting slice after rc=0. Defaults ANSIBLE_LOG_PATH to
/tmp/infinito-deploy.log so the check also runs locally without env-var
setup.
Includes 35 self-contained unit tests covering category derivation,
TASK-block status parsing (incl. ANSI escape stripping + substring
role-id guard + multi-skip edge case), required_role_ids YAML scan,
host log slicing with byte offsets, end-to-end verify, and CLI argparse.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reworks the REPL into a stateful shell that tracks a category path: typing a bare category name cd's into it, `/` returns to root, `..` walks one level up (chained as `../..` etc.), path tokens with slashes or a leading `..` navigate, `infinito [path]` always jumps absolute, and `ls` renders the current folder with a 🗂️/⚙️ emoji column, name, and a 30-char-truncated description (folders sorted before commands). Executable leaves (sub-packages with `__main__.py`) run instead of cd-ing into them. The prompt prints `🖥️ infinito <space-separated path>> `; the screen clears (incl. scrollback via `\033[3J`) before each command run, and the run echoes the resolved invocation in the fresh frame so the user sees what is about to execute.
Splits `repl.py` (~250 lines) into focused modules: `constants.py` (sentinels), `navigation.py` (path/category resolve + prompt + normalize), `ls.py` (folder rendering + truncate), `runner.py` (subprocess dispatch + clear_screen), `banner.py` (startup banner). `repl.py` is now a thin `main()` loop with backward-compat re-exports so external consumers keep the `repl.<symbol>` surface. Mirrors the split on the test side into `test_navigation.py`, `test_runner.py`, `test_ls.py`, and a `test_repl.py` that owns only main-loop integration with a shared `_MainLoopHarness` mixin.
Adds `scripts/tests/environment/11_console.sh` as a CI smoke that feeds a navigation sequence through the REPL and asserts every prompt shape plus the `ls` emoji output, so a regression in the prompt format, the path resolver, or the folder rendering is caught by the environment-test orchestrator.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`role_summary.py` no longer truncates to a configurable top-N: drops
`DEFAULT_TOP_N`, `TOP_N_ENV_KEY`, and `_read_top_n`; `_format_table`
renders all rows from `parse_role_times`. Renames the Markdown
heading and the corresponding GitHub Actions step in the four
`test-deploy-*` workflows from `Top role runtimes` to
`Role runtimes` for accuracy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Dashboard card icons were rendered with `http://172.17.0.1:8044/<slug>.svg`,
the in-cluster sync URL that only the Ansible runner can reach. The
browser saw `NS_ERROR_CONNECTION_REFUSED` for every Simple Icons card.
The `add_simpleicon_source` filter now takes two URLs: it probes the
local sync URL (plain HTTP, no TLS, no redirect chain — reachable
from the deploy container's `requests` without a CA bundle) and
writes the public `lookup('tls', 'web-svc-simpleicons', 'url.base')`
URL into `icon.source`. Cards render `<img src="https://icon.<DOMAIN>/<slug>.svg">`,
which is reachable from the browser and CSP-permitted (`img-src *`).
Drops the dead local-cache scaffolding (static-subdir vars, volume
mount, `Ensure directory` task, refresh-download step) since the
browser now fetches Simple Icons directly. Renames the
previously-static `templates/style.css.j2` to `files/style.css`
(no Jinja markers, matches the new template-uses-jinja lint warning).
Adds a Playwright spec `test-icons.js` that asserts the regression:
no card may still reference `172.17.0.1`, at least one card must
resolve under `icon.<DOMAIN>`, with a favicon fallback path when the
Simple Icons service is disabled.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`lookup('image', '<role>', '<svc>', 'ref')` only consulted
`images_overrides.*` plus role defaults, which the mirror pipeline
never populates — the pipeline writes into
`applications.<role>.services.<svc>.{image,version}`. As a result,
`sys-svc-container`'s `BUSYBOX_IMAGE` / `NODE_IMAGE`,
`sys-ctl-hlth-csp`'s `HEALTH_CSP_IMAGE`, and `test-e2e-playwright`'s
`TEST_E2E_PLAYWRIGHT_IMAGE` always resolved to the raw upstream ref
and hit Docker Hub directly. In CI this surfaced as
`toomanyrequests: ...unauthenticated pull rate limit` on
`busybox:1.37`.
Switches every productive `lookup('image', ...)` to
`lookup('config', '<role>', 'services.<svc>.image|version')` — the
same path templates like `web-app-oauth2-proxy` already use, which
resolves through the merged `applications.*` map and so transparently
picks up the GHCR mirror override written by the inventory creator.
Removes the now-unused `plugins/lookup/image.py` plugin and its unit
test (zero remaining consumers across roles, scripts, and tests).
Rewrites the image-origin / image-mirror / dockerfile docs to teach
the `lookup('config', ...)` pattern, dropping references to the
deleted plugin and the dead `images_overrides.*` variable. Doc edits
follow the `documentation.md` rules (no em dashes, one sentence per
line, no historical phrasing).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Enable the ansible.posix.profile_roles callback alongside profile_tasks
so every deploy run emits a per-role runtime block. Each test-deploy
workflow now writes the Ansible log to ANSIBLE_LOG_PATH and runs a new
post-step that parses the block, aggregates duplicate role entries,
sorts by total duration, and writes the top-N (default 20, override via
INFINITO_PROFILE_TOP_N) as a Markdown table to $GITHUB_STEP_SUMMARY so
the table appears on the Actions job overview page. Includes 15 unit
tests covering aggregation, ANSI stripping, task/total filtering,
env-knob handling, and step-summary writes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four follow-ups uncovered by \`make test\` after 3c9db6142:
* tests/lint/ansible/variables/test_role_and_group_vars_used:
TLS_ENABLED is declared in group_vars/all/02_tls.yml but never
read via \`{{ TLS_ENABLED }}\`; it is consumed by
plugins/lookup/{tls,email,cert}.py, which the YAML/Jinja-scoped
lint cannot see. Tag the declaration with the project's standard
\`# nocheck: unused-var\` marker and document the consumer.
* tests/lint/filesystem/python/test_no_raw_project_walk: the new
test_no_ansible_env_lookups.py used \`Path.rglob("*")\` for its
walk, which is the exact shape the no-raw-walks lint forbids in
test code. Switch to \`iter_project_files(extensions=_SCAN_EXTS)\`
with a relative-path prefix filter; same coverage, shared cache.
* tests/unit/cli/administration/deploy/development/test_inventory_vars_file_spot:
the new lint test legitimately names the dev inventory path
(\`inventories/development/default.yml\`) in its failure message.
Add it to the SPOT-drift-guard allowlist with a justification.
* tests/integration/ansible/jinja/test_variable_definitions: the
debug task added in 0884eb4e2 prints \`ansible_env\` and \`vars\`,
both Ansible magic variables. Add them to the test's reserved-name
list so the static analyzer stops flagging them as undefined.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wrap `python -m cli` in a host-side REPL launched via `make console`. The
loop normalizes input (strips leading `infinito`/`cli`, treats bare `help`
as `--help`), ignores Ctrl+C so accidental input cancels don't kill the
session, and exits only on `exit`/`quit`/`:q` or Ctrl+D. Includes a banner
with help/web/docs/license/author hints. Drops the now-redundant
`infinito --help` footer from `make help`. Adds 22 unit tests covering
normalization, subprocess dispatch, and main-loop control flow.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause of the DinD DNS-check failure on test-deploy-server
(networks.internet.ip4 falling through to 127.0.0.1 despite
INFINITO_IP4=172.30.0.10 reaching every link of the env pipeline):
cli/administration/inventory/provision/cli.py declared --ip4 with a
hardcoded default of "127.0.0.1" and ensure_host_vars_file wrote that
value unconditionally into host_vars/<host>.yml. group_vars'
defaults_networks.internet.ip4 resolves "172.30.0.10" via
\`lookup('env', 'INFINITO_IP4')\`, but tasks/stages/01_constructor.yml
merges \`defaults_networks | combine(networks | default({}, true))\` so
host_vars wins. INFINITO_IP4 looked broken end-to-end; the breakage
actually lived in the provision-CLI host_vars overlay.
INFINITO_DNS_IP and INFINITO_DOMAIN propagated correctly because
the provision CLI had no --dns and the --primary-domain write was
None-gated and never invoked.
While here, --ssl-disabled and --primary-domain are removed too:
neither had any effect in the dev flow. provision/cli.py applies
\`--vars-file inventories/development/default.yml\` AFTER
ensure_host_vars_file via apply_vars_overrides_from_file, whose
_deep_update_commented_map unconditionally overwrites
(\`target[key] = value\`). The dev vars-file always set
\`TLS_ENABLED: true\`, so \`--ssl-disabled\` (which forced
\`TLS_ENABLED: false\` in host_vars) was systematically reverted to
true. Dev/CI ran with TLS enabled the whole time. Inventory stays at
\`TLS_ENABLED: true\` to preserve that effective behaviour.
Removed from cli/administration/inventory/provision/cli.py:
- --ip4, --ip6: defaulted to 127.0.0.1 / ::1, shadowed group_vars
- --primary-domain: never set by any caller, None-gated write
- --ssl-disabled: always re-overwritten by --vars-file later in the
same provision pass
Removed from cli/administration/inventory/provision/host_vars.py the
matching writes in ensure_host_vars_file; the function now only sets
\`ansible_connection: local\` for localhost-class hosts. Defaults for
DOMAIN_PRIMARY / TLS_ENABLED / networks.internet.{ip4,ip6} live in
group_vars (env-driven) and the inventory vars-file (deployment-wide
flags); host_vars stops duplicating them.
Callers cleaned up:
- cli/administration/deploy/development/inventory/builder.py
- cli/administration/deploy/container/command.py
- scripts/tests/deploy/local/utils/fresh-kept-all-init-and-deploy.sh
- scripts/tests/deploy/local/reset/inventory.sh
Tests + docs follow:
- tests/unit/cli/administration/inventory/provision/test_host_vars.py
now asserts host_vars no longer carries baked DOMAIN_PRIMARY /
TLS_ENABLED / networks defaults
- docs/administration/deploy.md drops --ssl-disabled from the
provision example
- docs/contributing/design/inventory.md re-scopes the host_vars
column: the DOMAIN_PRIMARY / TLS_ENABLED / networks.internet.{ip4,ip6}
values arrive via --vars / --vars-file overlays, otherwise group_vars
provides the env-driven default
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
compose.yml no longer hides services behind \`profiles: ["ci"]\`. The
profile existed solely as a no-op safety net against a bare
\`docker compose up\` in the repo root, but the top-level compose.yml is
already explicitly scoped to local dev / CI testing (production goes
through the Ansible roles under roles/<role>/templates/), so the safety
net is redundant with the documented scope.
Every \`--profile ci\` callsite collapses:
- compose.yml: \`profiles: ["ci"]\` removed from coredns + infinito
- Profile.args() deleted; Profile keeps is_ci() and
registry_cache_active() for the cache-override gating
- compose.py / down.py: profile.args() splat removed from the
docker-compose command builder; down.py drops its now-unused
Profile import
- 5 shell wrappers (install, lint, tests/code, meta/resolve/apps,
meta/resolve/diff/affected_roles): plain \`docker compose\` instead
of \`docker compose --profile ci\`
- tests/unit/.../test_profile.py: TestProfileArgs class removed
- tests/unit/.../test_compose.py: two cmd-shape asserts updated to
reflect the missing --profile pair
Docs:
- docs/contributing/artefact/files/compose.yml.md: the
\"every service MUST be behind a profiles: entry\" rule is gone
- docs/contributing/actions/debugging/ci.md: the
\`docker compose ... --profile ci config\` debug snippet drops the
flag; while touching the file the two em-dash clause separators in
the reproduction-profiles shell block are rewritten to colons
(documentation.md em-dash rule), and one list-item in
compose.yml.md is collapsed to one sentence per line for
consistency with its siblings
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
With env.development and env/ci.env both gone, env/default.env was the
last inhabitant of the directory. Hoisting it to the repo root removes
the empty-shell indirection: the SPOT is now [default.env](default.env),
sitting next to .env (its generated companion) and other repo-root
configuration artefacts like compose.yml, Dockerfile, and Makefile.
Path updates land in:
- generator: cli/meta/env/__main__.py (REPO_ROOT / "default.env")
- workflow: .github/workflows/test-install-make.yml (awk reads from
default.env, error message updated in lockstep)
- lint tests: test_used_keys_declared.py and test_infinito_prefix.py
switched from globbing env/*.env to targeting default.env directly;
test_default_env_keys_handled.py points at the new path
- drift test: test_inventory_vars_file_spot.py whitelist + STATIC_ENV
- smoke test: tests/integration/meta/env/test_dotenv_generator.py
STATIC_ENV + every docstring/error message
Doc-string and prose references in utils/env/*, scripts/*,
cli/administration/deploy/development/{common,coredns}.py,
Makefile, Dockerfile, and every doc page that mentioned env/default.env
are rewritten in the same pass so future readers see one path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The four ci.env keys (INFINITO_DNS_IP, INFINITO_DOMAIN, INFINITO_DOMAIN_RE,
INFINITO_IP4) were only honored by docker compose --env-file, so processes
that sourced .env via load.sh saw INFINITO_IP4 as unset. As a result
networks.internet.ip4 in the Ansible playbook fell through to the
127.0.0.1 default and the DinD DNS-check handler failed on the deploy
server matrix.
Moving the defaults into env/default.env makes them flow through the same
.env / load.sh / os.environ chain as every other INFINITO_* key, so
\${INFINITO_IP4} in compose.yml, lookup('env', 'INFINITO_IP4') in
group_vars, and envsubst over the CoreDNS Corefile template all see the
same value deterministically.
Side fix: utils/env/parser.py now undoes the writer's \\\\/\\" escaping for
quoted values, closing a round-trip gap that surfaced once
INFINITO_DOMAIN_RE (the only value containing a literal backslash) started
flowing through the generator.
All --env-file env/ci.env call sites (compose.py, down.py, install/lint/
test wrappers, resolve scripts, test-dns workflow + script) are removed.
coredns.py now reads .env via parse_static_env, docs and unit/integration
tests follow.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Relocates `cli/meta/{ports,networks/suggest}` to
`cli/contributing/network/{ports,address/suggest}` so port and subnet
allocators live alongside the rest of the contributor-side tooling
(mirror, update, changelog) instead of `meta`. Mirrors the tests
into `tests/unit/cli/contributing/network/` and renames the contract
docs to `docs/contributing/tools/network/{port,address}.md`. Prog
names update from `cli meta ports suggest` / `cli meta networks
suggest` to `cli contributing network ports suggest` / `cli
contributing network address suggest`; the corresponding lint
references and `utils.meta.scan` docstring follow.
Adds a third subcommand: `cli contributing network diagnose`
(stdlib-only DNS / TCP / TLS / PMTU probe against the standard
infrastructure hosts plus interface MTU, `/etc/resolv.conf`,
`/etc/hosts`, proxy env, CA-bundle summaries). Split across `cli.py`
(orchestration), `config.py` (SPOT for hosts / per-distro install
commands), `format.py` (output + subprocess helpers), `tools.py`
(distro-aware auto-install of `iputils-ping` + `iproute2`),
`report.py` (system-info dumps + IPv6-default-route detection), and
`probes.py` (per-host probes). Exposed as `make debug-network`.
Companion `docs/contributing/actions/debugging/network.md` absorbs
the diagnose-tool walkthrough (incl. how to read MTU-induced TLS
handshake timeouts) so the merged page covers DNS / MTU / IPv6 /
connectivity end-to-end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`pull.py` always ran `compose build --pull` so BuildKit refetched the
FROM image even when it was already local. When the registry was
unreachable mid-handshake (sandboxed bridge, MTU mismatch dropping
TLS-ServerHello fragments) the build aborted on
`failed to resolve source metadata` even though the base image was
on disk.
The buildable path now runs `compose build --pull` first and falls
back to plain `compose build` on non-zero exit. The fallback uses the
local FROM image directly; if that build also fails, run_or_fail
still raises with the labelled error.
Test suite reworked to track every `run_cmd` invocation (the build
path no longer routes through run_or_fail on the happy path) and
covers both the fallback branch and the local-images-only pull
tolerance path that was already in main.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`make dotenv` previously copied the static 1500 default from
env/default.env into .env for the compose bridge MTU, regardless of the
host's actual outbound MTU. On hosts where the default-route interface
sits below 1500 (VPN, tunnels, jumbo-frame negotiation failure) the
bridge ended up larger than the path MTU and TLS handshakes silently
lost their ServerHello fragments — every dotenv regeneration without
an explicit INFINITO_OUTER_NETWORK_MTU= override silently re-introduced
the regression.
Replaces the static-passthrough entry with a dedicated handler that
walks /proc/net/route to find the default-route interface, reads
/sys/class/net/<iface>/mtu, and seeds the value via setdefault so any
caller-set env still wins. Falls back to the env/default.env static
default when detection cannot complete (no default route, unreadable
sysfs entry, foreign host layout).
Adds 13 unit tests covering the /proc parse, the sysfs read, the
composition, and the handler's apply contract (detected wins,
static fallback, no-op when both empty, caller-env override).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two independent bugs were compounding into a corrupted NIX_CONFIG that
broke every local image rebuild:
1. scripts/tests/code/run.sh non-idempotently appended
"accept-flake-config = true" to whatever NIX_CONFIG it inherited.
On the next source of .env (which carried the previous value back),
the line accumulated -- "true\naccept-flake-config = true\n..." --
until Nix rejected the Boolean. Fix: pipe through `awk '\!seen[$0]++'`
to dedup lines while preserving order.
2. utils/env/writer.py silently re-encoded literal newlines as `\n`
escape sequences inside double-quoted values. docker-compose's
--env-file loader does not decode them and neither does bash
`source`, so any multi-line value round-tripped into garbage on the
next read. Fix: raise ValueError instead -- producers must collapse
to a single line before handing off to the env-builder.
Test fixtures for the writer adjusted to assert the new raise
semantics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
build/install-lint-<hash>.stamp lives on the host-bind-mounted build/
tree, but the lint tools live inside the docker container. When the
infinito image is rebuilt, the tools are gone but the stamp persists --
so the next install-lint short-circuits and downstream lint targets fail
with exit 127 (command not found).
_stamp_is_fresh() now additionally checks shutil.which() for every tool
the all-mode install produces. A missing tool invalidates the stamp,
forcing the per-tool ensure() chain to run (each ensure is idempotent so
this stays cheap when state is clean).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Promote the key from GHA-only (gha_passthrough) to always-emitted (passthrough)
so the deploy CLI can read it strictly via os.environ[...]. The previous
INFINITO_NO_BUILD layout left the key unset locally, which crashed
cli/.../up.py and cli/.../compose.py with KeyError after the env-pipeline
refactor that introduced gha_passthrough.
New semantics:
INFINITO_BUILD=1 build the local image (default; local mode)
INFINITO_BUILD=0 skip build, pull image (CI overrides via -e)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pulls VENV/PYTHON/PIP/PYTHONPATH out of the dotenv handler pipeline
into scripts/meta/python/detect.sh. load.sh now sources detect.sh and
prepends ${VENV}/bin to PATH above the INFINITO_ENV_LOADED guard, so
every BASH_ENV re-entry re-resolves PYTHON the moment a venv
materialises mid-chain — the same fix as b79ed2869 but at the SPOT,
which retires the reload-trick from ansible.sh, python.sh and
pre-commit.sh.
Also renames INVENTORY_VARS_FILE → INFINITO_INVENTORY_VARS_FILE under
the new env-prefix lint, drops the 5 dead INFINITO_ENV_*_LOADED unsets
left over from the multi-loader era, and adds a lint guarding the
INFINITO_ prefix invariant on env/*.env keys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
scripts/docker/entry.sh's two compile-control reads (INFINITO_COMPILE
and INFINITO_COMPILE_SILENCE) crashed during Dockerfile RUN because
that context has no env-pipeline: env/default.env exists in the repo
but no make/BASH_ENV/load.sh chain materialises it into the running
shell. The bare reads under `set -u` triggered unbound-variable exit
even though the keys were registered in default.env + passthrough.
Move build/silence control from env to script flags so entry.sh is
self-contained at bootstrap time. Each flag now triggers its action
directly inside the parse loop -- no COMPILE/SILENCE state vars:
--compile -> run_local_build (streaming)
--compile-silent -> run_local_build >/dev/null 2>&1
-- -> end flag parsing, rest is the exec target
Drop INFINITO_COMPILE + INFINITO_COMPILE_SILENCE from the whole
registry surface: env/default.env, passthrough.STATIC_KEYS, the
infinito_compile handler (gone entirely), compose.yml's env
injection block, the dotenv-generator baseline test, the
build_env unit-test fixture, and the now-no-op `-e INFINITO_COMPILE=0`
in test-install-{make,pkgmgr} workflows.
Dockerfile RUN switches to `entry.sh --compile -- true`; runtime
ENTRYPOINT receives no flags so the default (no-compile) holds and
exec hands off to systemd via the CMD.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three new lint tests close gaps in the INFINITO_* SPOT contract:
* tests/lint/filesystem/env/test_default_env_keys_handled.py
Every default.env key must be wired through a handler.
* tests/lint/filesystem/env/test_used_keys_declared.py
Every INFINITO_* in .sh/.py must live in env/{default,ci}.env or
a handler; per-line `# nocheck:` suppresses runtime-only refs.
* tests/lint/filesystem/python/test_no_infinito_get_defaults.py
Forbid ANY default in `.get("INFINITO_*", ...)` -- including the
empty-string fallback (which silently masks an unset key). The
nocheck check walks every physical line the call spans so an
autoformat-split call still honours the marker.
Cache namespace flattens to a single prefix:
INFINITO_PACKAGE_CACHE_* -> INFINITO_CACHE_PACKAGE_*
INFINITO_REGISTRY_CACHE_* -> INFINITO_CACHE_REGISTRY_*
The five previously driver-injected paths (APT_LIST, NPMRC,
PIP_CONF, REGISTRY_PROXY_CONF, FRONTEND_CA_FILE) move into
env/default.env and passthrough.STATIC_KEYS; the
`cache_env_overrides()` helper and its two call sites are gone
along with the test-fixture override for FRONTEND_CA_DIR.
SPOT-strict rewrites:
- 19 sites: `something.get("INFINITO_X", "")` -> subscript
`something["INFINITO_X"]` (loud KeyError when env loader did
not run; all keys live in env/default.env).
- 2 handler-internal bootstrap reads suppressed with
`# nocheck: handler-bootstrap-read`.
- 30 runtime-only / driver-injected / test-fixture references
annotated with `# nocheck: <reason>` per-line.
- Unit tests that previously removed SPOT-guaranteed keys via
`{k: v ... if k \!= KEY}` or `clear=True` switch to setting
the key to "" explicitly, matching the new contract.
Side fixes:
- INFINITO_DNS_DOMAIN + INFINITO_DNS_HOSTS_FILE move from
shell-side defaults in common.sh to env/default.env and
passthrough.STATIC_KEYS.
- dns_systemd_is_operational() drops the dead
INFINITO_DNS_FORCE_HOSTS_FALLBACK override branch (the
variable was never set anywhere).
- INFINITO_WORKER_FETCH switches from `os.environ.get(KEY, "1")`
to `os.environ[KEY]`.
- scripts/tests/code/wrapper.sh exec args move into a bash
array so `# nocheck:` on INFINITO_TEST_TYPE can live on its
own element line (SC2215 fix).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rename env/static.env → env/default.env across the entire repo
(string label, doc links, Python file-path constants) — the
`parse_static_env*` function names stay as a stable API surface.
Sort all `*.env` / `*.env.j2` entries alphabetically (52 in
default.env, 4 in ci.env, 46 of 49 j2 templates; 3 templates
with Jinja2 `{% %}` control flow are skipped because conditional
blocks would tear apart under a naive sort). Sort all 83
Makefile target blocks alphabetically; .PHONY group headers
stay in place.
Collapse the two remaining multi-line Makefile recipe comments
(dotenv-force, test-signed) to single-line WHY comments.
Add four lint tests under tests/lint/filesystem/ to keep the
invariants enforced going forward:
- env/test_alphabetical_sort.py
- env/test_single_line_comments.py (# nocheck: markers exempt)
- makefile/test_alphabetical_targets.py
- makefile/test_single_line_comments.py (# nocheck: markers exempt)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
env/static.env:
Drop the stale "and 'make lint-galaxy'" reference from
INFINITO_PARALLEL's comment (lint-galaxy was removed earlier).
tests/external/repository/test_urls_reachable.py:
Make _MAX_WORKERS network-aware. URL reachability is I/O-bound, so
cpu_count() is the wrong proxy; scale by 20x to saturate the wait
budget. Respects INFINITO_PARALLEL=false for sequential fallback.
tests/unit/utils/install/test_github_release.py:
Replace the github.com/x/y/ dummy URLs with example.com/x/y/ so the
URL-reachability test (which scans source for HTTP literals) skips
them via its example.* allowlist. The github.com URLs were probed
for real and returned 404.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
galaxy-importer's legacy-role mode was dropped upstream in 0.5.0
(CLI no longer accepts --legacy-role / --namespace), and the project
does not publish roles to Ansible Galaxy. The check enforces a
publishing schema that never gets exercised, against an abandoned API
that can only be kept alive via pinning galaxy-importer==0.4.39 +
setuptools<81 in an isolated venv. The cost has stopped buying
anything.
Removed:
- scripts/lint/galaxy.sh
- utils/install/lint/galaxy_importer.py
- install_isolated_pip_pkg() in utils/install/pip.py (only galaxy_importer
used it)
- Makefile lint-galaxy target + the entry in the `lint:` aggregate
- pre-commit make-lint-galaxy hook
- The galaxy_importer ensure() call in _install_ansible_tools
- Test assertion against cli.galaxy_importer in test_main.py
- Documentation rows mentioning lint-galaxy / galaxy-importer
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Refactor scripts/install/lint.sh (~870 LOC bash) into a Python package
under utils/install/. scripts/install/lint.sh becomes a 7-line wrapper
that delegates to `python3 -m utils.install.lint`.
Layout:
- utils/install/{primitives,pip,npm,github_release,system_pkg}.py:
channel-agnostic installers
- utils/install/lint/<tool>.py: one ensure() per tool (actionlint,
shfmt, mbake, ruff, ansible_lint, galaxy_importer, markdownlint_cli2,
eslint, shellcheck, ansible_commands, ansible_collections)
- utils/install/lint/__main__.py: CLI + group dispatch + stamp logic
INFINITO_LINT_RUNNER dispatch:
- New env var (host | docker, default docker), tracked in env/static.env
+ passthrough handler + drift test.
- scripts/install/wrapper.sh routes `make install-lint` to host or
container based on the runner.
- scripts/lint/wrapper.sh already dispatched; container branch now uses
bash (not --login) so /etc/profile does not reset PATH, and forwards
BASH_ENV so the .env keys land in the lint script's env.
Stamp logic:
- Per-interpreter stamp at build/install-lint-<sha256(sys.executable)>.stamp
so host venv and container venv track installs independently despite
bind-mounted build/.
- `--force` drops the stamp; group args bypass the stamp.
Tests + docs:
- 43 unit tests under tests/unit/utils/install/{,lint/} cover primitives,
pip/npm/github_release/system_pkg fallback chains, and the main CLI
(stamp logic, group dispatch, error path).
- utils/install/README.md + utils/install/lint/README.md document the
structure and the per-tool ensure() contract.
- docs/contributing/tools/make.md install-lint row updated to mention
the runner dispatch + per-env stamp + install-lint-force.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
