OpenTalk's images (web-frontend, controller, recorder) live on
registry.opencode.de, which was absent from both _MIRRORABLE_REGISTRIES and
_ALL_REGISTRY_PREFIXES in utils/docker/image/discovery.py. So is_mirrorable_image
returned False, iter_role_images skipped them, the GHCR mirror never copied
them, and the deploy pulled direct from registry.opencode.de -> TLS handshake
timeout (deploy failed).
Add the registry to both sets so the single iter_role_images gate enrolls the
3 images: the mirror-sync copies them to GHCR and the resolver rewrites the
deploy to pull from the GHCR mirror. The prefix entry strips the host so the
mirror destination is well-formed (no doubled registry). Add a regression test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The auto-generated "Older Releases" index in CHANGELOG.md emitted dash
bullets while the rest of the file uses asterisks, failing markdownlint
MD004 (ul-style, consistent) with 49 errors. Switch the archive index
generator to asterisk bullets, convert the existing entries, and align
the unit test assertions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause of step-08 deploy failures across all 4 test-development
distros in run 26572559126: `make compose-deploy mode=reinstall
apps=web-app-dashboard INFINITO_SERVICES_DISABLED=matomo` (step 07)
made make export the cmdline var into the recipe's shell env, which
reached `runner.py` → `make setup` → `make dotenv`. The dotenv
generator's `setdefault` then baked the runtime-only value into the
on-disk `.env`. Step 08 sourced that .env, filtered `web-app-matomo`
out of its primary apps, and exited with "All primary apps disabled".
Fix:
- Drop the 7 routing knobs (apps, mode, purge, bundles, disable,
full_cycle, variant) from `passthrough.STATIC_KEYS` and from
`default.env`. They are per-invocation; .env is the persistent SPOT
and must not capture them.
- Switch every consumer from `os.environ["KEY"]` strict access to
`os.environ.get("KEY", "")` so the missing-key path stops at "no
filter" instead of a KeyError.
- Rename the env-var surface to match the make-alias surface 1:1
(lowercase): INFINITO_APPS → apps, INFINITO_SERVICES_DISABLED →
disable, INFINITO_BUNDLES → bundles, INFINITO_FULL_CYCLE →
full_cycle, INFINITO_VARIANT → variant, INFINITO_DEPLOY_MODE → mode,
INFINITO_PURGE_ENTITIES → purge. The make alias was the user-facing
name in every doc; routing-knob env vars now use the same name so
the cmdline form, the recipe env-prefix, and the consumer reads
all line up.
- Collapse the Makefile `compose-deploy` recipe to just the two
knobs whose alias differs from the env name (`type` →
INFINITO_DEPLOY_TYPE, `debug` → INFINITO_DEBUG). GNU Make
auto-exports cmdline variables into the recipe's shell env, so the
identical-name `$(if $(apps),apps="$(apps)")` lines were pure
noise.
- Drop `disable: ${disable}` from compose.yml. The dev CLI builds
`extra_env` per `docker exec` call and forwards `disable` via
`-e`, so the container env never needs the value frozen at
compose-up time.
- Update the 3 actual leak call sites
(07/08/09_deploy_*.sh) to the make-alias form
(`disable=matomo`, `full_cycle=true`, `variant=1`).
- Add `SIM112` to ruff's ignore set: the lowercase env names are
the deliberate make/env parity, not a typo.
Repro before fix: `INFINITO_SERVICES_DISABLED=matomo make dotenv`
wrote `INFINITO_SERVICES_DISABLED=matomo` to `.env`. After fix:
`disable=matomo make dotenv` leaves `.env` untouched.
Tests: `make test` and `make autoformat` green (2711 tests, 8 lint
sub-targets).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The fully-checked-Markdown archive helper used to live under
cli/contributing/requirements/archive/ in this repo. It is a generic
spec-folder-management primitive (numbered NNN-topic.md files,
task-list completeness check, README ## Archive section append) and
now ships in pkgmgr.actions.archive so every kpmx-managed repository
can rely on the same convention.
Changes here:
* tests/lint/repository/documentation/test_requirements_completeness.py
imports the archive primitives from pkgmgr.actions.archive instead
of the vendored cli.contributing.requirements.archive, and points
contributors at `pkgmgr archive docs/requirements` in the failure
message.
* cli/contributing/requirements/README.md becomes a stub that
redirects readers to the kpmx archive subcommand.
* The vendored cli/contributing/requirements/archive/ package and its
unit tests under tests/unit/ are deleted.
* The pkgmgr role pin bumps to v1.15.0 to pick up the new archive
subcommand on deployed hosts.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* TC003: move `pathlib.Path` into `TYPE_CHECKING` blocks across the
new `requirements/archive` package and its completeness lint test.
* TC001: same treatment for `VaultHandler` in
`credentials/emit.py` (annotation-only).
* SIM105/S110: replace `try/except/pass` in
`current_play_redirect_domains.py` with
`contextlib.suppress(Exception)`.
* `archive/__main__.py`: mark `README_PATH.read_text(...)` with
`# nocheck: cache-read` — CLI rewrites the README in the same flow,
mirroring the `changelog/archive` pattern.
* Drive-by ruff-format condensations in the archive unit tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* `python -m cli.contributing.requirements.archive` moves the H1 of
every fully-checked `docs/requirements/NNN-*.md` into the
`## Archive` block of the directory README and deletes the file.
Files with any remaining `- [ ]` are skipped. Split SRP-wise into
discovery / inspect / readme / __main__.
* New lint `test_requirements_completeness` warns per unchecked
acceptance criterion (GitHub annotation) and fails when any
requirement is ready for archival, pointing at the CLI. Keeps the
requirements directory small so AI agents process less context.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The inventory validator at cli/administration/inventory/validate
walked `recursive_keys(host_vars) ⊆ recursive_keys(application_defaults)`
and flagged any extra key with "Missing default for <app>: <key>".
After commit 10cb02461 ("bake only variants.yml overrides into
matrix-deploy --vars") host_vars carry every key the active
variants.yml overlay declares. Since 29377e1bd
(`_build_application_defaults` reads only `meta/<topic>.yml`, not
variants.yml — to keep the variant-0 service-key leak closed), the
defaults' allow-set is artificially narrower than the role contract:
roles/web-opt-rdr-www declares `services.dashboard.{enabled,shared}`
exclusively in `meta/variants.yml`, so the validator rejects the
baked host_vars with three "Missing default" entries before
ansible-playbook even starts (job 77797115291 on run 26428083128,
distro=debian; distro-agnostic per the matrix).
Widen the validator's legal-key set per app to
recursive_keys(defaults[app]) ∪ ⋃ recursive_keys(variants[app][i])
variants.yml IS part of the role contract; the validator's previous
"defaults only" view was incomplete. Runtime
`_build_application_defaults` stays narrow (so 29377e1bd's
variant-0-leak fix stays in place) — only the validator widens. Typos
not declared by any variant remain caught: a new
`test_key_typo_in_variants_still_rejected` unit test pins that.
Package split (KISS/DRY/SPOT), mirroring the credentials/ refactor:
`validate/__main__.py` carried six orthogonal concerns plus
orchestration in 175 LoC. Split:
- `validate/keys.py`: `recursive_keys` (pure helper).
- `validate/loaders.py`: `load_yaml_file` (with vault-block stripping)
and `load_inventory_files` (directory scan).
- `validate/applications.py`: `compare_application_keys` (the widened
variants-aware logic).
- `validate/users.py`: `compare_user_keys`.
- `validate/hosts.py`: `validate_host_keys`.
- `validate/__main__.py`: thin orchestrator — argparse + load defaults/
variants/users + invoke compare_* + exit code. 75 LoC.
`validate/__init__.py` re-exports the public API
(`main`, `compare_application_keys`, `compare_user_keys`,
`load_inventory_files`, `load_yaml_file`, `recursive_keys`,
`validate_host_keys`, `PROJECT_ROOT`). External callers
(`tests/unit/cli/administration/inventory/validate/test_main.py`,
`tests/unit/cli/administration/inventory/validate/test_hosts.py`)
keep working unchanged via this re-export.
`__main__.py` uses ABSOLUTE imports (`from cli.administration.inventory.
validate.applications import ...`) instead of relative because it's
invoked as a subprocess by tests and by the deploy entrypoint; the
sys.path bootstrap at the top of the file makes `cli.…` resolvable
in that mode.
Test layout follows the split — one test file per module:
- `test_keys.py` (6 tests: empty/non-dict/flat/nested/deep/prefix).
- `test_loaders.py` (8 tests: vault stripping, missing file, broken
YAML, top-level + recursive directory scan, empty-block skip).
- `test_applications.py` (9 tests: all-keys-match, unknown app, extra
key, credentials skip, variants legal, variants union, no-variants
fallback, source path in error, typo still rejected).
- `test_users.py` (5 tests: match, unknown user warn, missing default,
password skip, credentials skip).
- `test_hosts.py` (renamed from test_inventory_hosts.py — unit tests
for `validate_host_keys`).
- `test_main.py` (renamed from test_inventory.py — end-to-end
subprocess tests for `main()`, including the
`test_variants_only_key_is_accepted` regression guard for this fix).
Coverage: 36 tests, all green. `make test` green across all 5 targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The credentials baker at cli/administration/inventory/credentials/__main__.py
walked `schema_creds.items()` one level deep and called `str(value)` on
every leaf. For nested credential groups like
``credentials.recaptcha = {key, secret}`` (declared in
roles/web-app-espocrm/meta/schema.yml and
roles/web-app-listmonk/meta/schema.yml) this collapsed the inner dict
into its Python repr, vault-encrypted that blob, and wrote it as a
single scalar at ``credentials.recaptcha``. The runtime ``config``
lookup then choked on the descent into ``recaptcha.key``:
The lookup plugin 'config' failed: Expected dict for 'key', got
_AnsibleTaggedStr at 'key'
Full path so far: applications['web-app-espocrm'].credentials.recaptcha.key
Current object: "{'key': '{{ CAPTCHA.RECAPTCHA.KEY }}', 'secret':
'{{ CAPTCHA.RECAPTCHA.SECRET }}'}"
(jobs 77797371397 espocrm and 77797371442 listmonk on run 26428080957.)
Introduce ``emit_credentials`` (new module: ``credentials/emit.py``)
that walks the schema recursively. Scalar leaves still vault-encrypt
via ``to_vault_block``; dict nodes recurse into a nested CommentedMap
via ``ensure_map``. The walker also tracks dotted full-key paths so
the existing ``--force``-mode override-apply logic sees the same dotted
keys it always saw.
Auto-healing for previously-corrupted host_vars: ``ensure_map``
overwrites non-CommentedMap placeholders, so the next bake replaces a
stringified ``credentials.recaptcha`` blob with the correct nested
structure. Operators with corrupted entries simply re-run the
credentials generator; no manual cleanup needed.
Package refactor (KISS/DRY/SPOT):
The 450+ LoC ``__main__.py`` was carrying four orthogonal concerns
(vault, --set parsing, recursive emission, prompts) plus orchestration.
Split by concern:
- ``credentials/vault.py``: ``is_vault_encrypted``,
``is_ruamel_vault``, ``to_vault_block`` (+ private helpers
``_vault_body``, ``_make_vault_scalar_from_text``).
- ``credentials/overrides.py``: ``parse_overrides``, ``override_for``.
- ``credentials/emit.py``: ``ensure_map``, ``emit_credentials``.
- ``credentials/prompts.py``: ``ask_for_confirmation``.
- ``credentials/__main__.py``: argparse + ``main`` + two thin mode
handlers (``_run_snippet_mode``, ``_run_default_mode``) + the
force-override helper.
``credentials/__init__.py`` re-exports the public API
(``main``, ``ask_for_confirmation``, ``parse_overrides``, etc.) so
external callers (notably ``tests/unit/cli/administration/inventory/
test_credentials.py``) keep working unchanged.
Test layout follows the split:
- ``tests/unit/cli/administration/inventory/credentials/test_vault.py``
(20 tests: vault detection, body extraction, scalar construction,
to_vault_block paths, plus the VaultHandler subprocess coverage).
- ``tests/unit/cli/administration/inventory/credentials/test_overrides.py``
(13 tests: parse_overrides + override_for incl. dotted nested keys).
- ``tests/unit/cli/administration/inventory/credentials/test_emit.py``
(10 tests: ensure_map placeholder replacement, recursive emission,
none → empty, skip_existing semantics, override precedence,
dotted-path tracking, the on-disk corruption recovery case).
- ``tests/unit/cli/administration/inventory/credentials/test_prompts.py``
(5 tests: ask_for_confirmation yes/no/garbage/case/whitespace).
``tests/unit/cli/administration/inventory/test_credentials.py`` keeps
only the end-to-end ``main()`` integration coverage (including
``test_nested_credentials_dict_recurses_instead_of_stringifying`` —
the explicit regression guard for the original failure mode).
Coverage: 47 new per-module unit tests + the 5 existing main()
integration tests, all green. ``make test`` green across all five
targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Since 576fad84f (2026-05-10) moved role images from
`defaults/main.yml` into `meta/services.yml`, matrix-deploy's variant
baker has been writing the FULL effective config (services.yml +
variants.yml deep-merged via `get_variants`) into host_vars through
the `--vars` JSON consumed by `apply_vars_overrides`. That populates
`applications.{role}.services.{svc}.image` (and `.version`, .cpus,
etc.) with the upstream value, which then silently disables
`apply_mirror_overrides` for every role with an explicit image
declaration: the default `if_missing` policy short-circuits when the
field is already non-blank, so the rendered compose / Dockerfile FROM
keeps pulling from the upstream registry rather than the GHCR mirror.
The regression went unnoticed because Docker Hub usually serves the
pull, but it eventually surfaced as a BuildKit 429 on
`docker.io/library/php:8.2-apache` during the suitecrm Dockerfile
build (job 77602110531 on run 26362676803). Nextcloud, moodle,
mariadb consumers, postgres consumers, simpleicons, phpmyadmin,
phpldapadmin, jenkins, baserow, fider, ... all share the same latent
defect — only suitecrm happened to hit the rate limit first.
Fix the root cause instead of papering over per-role with
`mirror_policy: force`: introduce `get_variant_overrides_only` next
to `get_variants` and switch payload.py to it. The new function
returns the raw `meta/variants.yml` entries per role — no merge with
`meta/services.yml`. matrix-deploy now bakes only the variant-
specific overrides into host_vars; the role's services.yml fields
flow through the normal Ansible lookup chain at deploy time, leaving
`image`/`version` blank in host_vars so `apply_mirror_overrides`
populates them from `mirrors.yml` under its default `if_missing`
policy.
Behavior preserved for the other `get_variants` consumers:
* utils/manager/inventory.py:104 (schema-driven credentials) keeps
the full merged variant payload.
* planner.py:58 (variant-count only) is unaffected.
* legacy_resolver.py:42 (CombinedResolver runtime services_overrides)
keeps the full merged view.
Coverage:
* 3 new tests in test_applications.py for the new helper (single
empty variant when no meta/variants.yml, raw override without base
merge, cache invariance).
* test_payload.py + test_builder.py mocks repointed from `get_variants`
to `get_variant_overrides_only`.
* `make test` green across all 5 targets (lint, unit, integration,
external, test-lint) — 188 tests total.
Side-effects assessed:
* Repo-wide grep confirms no template / task accesses
`applications.X.services.Y.image` directly via hostvars — every
consumer goes through `lookup('config', …)` which still deep-merges.
* Roles previously pulling upstream now resolve via the GHCR mirror
whenever the image is already synced (the `images-mirror-missing.yml`
workflow keeps the destination populated).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
InventoryManager.resolve_schema_includes_recursive previously read the
root role's base meta/services.yml when discovering shared providers,
regardless of which matrix variant the planner had pinned for the
round. For a role whose variants.yml enables additional services per
variant (e.g. svc-bkp-container-2-local variant 2 enabling
services.ldap), the base config had no `enabled+shared` flag for those
services, so the recursive resolver never visited the variant's
shared providers — their credentials were never generated, and the
deploy crashed in the provider's env.j2 with "credentials is defined
in schema at roles/<provider>/meta/schema.yml but not set in
application '<provider>'".
Wires the active variant through the existing inventory pipeline:
- utils/manager/inventory.py: InventoryManager(variant=None|int). When
set, load_role_config_by_path overlays variants.yml[N] onto the root
role's meta config (only for the root — transitive providers keep
their own base lookup). variant=None preserves the pre-refactor base
behavior for every standalone caller.
- cli/administration/inventory/credentials: new --variant N flag,
forwarded to InventoryManager.
- cli/administration/inventory/provision: new --app-variants <JSON>
flag ({app_id: variant_index}), forwarded per-app to the credentials
subprocess.
- cli/administration/deploy/development/inventory/builder.py: emits
--app-variants from spec.variant_selectors() when the round pins any
variant. Apps without variants.yml collapse to a single empty
overlay in get_variants so this is a no-op for them.
10 unit tests cover: variant overlay for root role, variant=None
fallback to base, out-of-range fallback, no overlay on non-root
paths, --variant CLI forwarding (set + default), --app-variants
generator forwarding (set + omit), builder --app-variants emission
(set + omit).
Caveat: bkp-c2l CI variant 1 still loads svc-db-openldap transitively
at runtime (outside the planner's variant closure) — a separate
sys-service-loader / role-meta bug, not addressed here.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements docs/requirements/021-sso-flavor-migration.md atomically:
* Schema: every role's meta/services.yml replaces the parallel
`oauth2:` / `oidc:` top-level blocks with a single `sso:` block
carrying `enabled`, `shared`, and a `flavor` discriminator
(`oidc` | `oauth2` | `saml`; default `oidc`). Flavor-specific
sub-keys live under `sso.<flavor>.*` (`sso.oauth2.{origin,acl,
allowed_groups}`, `sso.oidc.plugin`).
* Provider role: roles/web-app-oauth2-proxy/ is deleted in its
entirety; its 5 sidecar templates move under
roles/web-app-keycloak/templates/sso_proxy/, and the per-consumer
rendering task lives at roles/web-app-keycloak/tasks/sso_proxy.yml.
web-app-keycloak now declares `provides: sso`.
* Runtime: new resolver utils/roles/applications/services/sso.py
centralises the SSO predicates (`is_enabled`, `is_proxy_gated`,
`is_oidc_native`) and the oauth2 sub-values
(`oauth2_origin_{host,port}`, `oauth2_acl`, `oauth2_allowed_groups`).
Three lookup plugins consume the resolver: `sso` (single-app
predicate access), `sso_proxy_consumers` (enumerates oauth2-flavored
consumers), `sso_oidc_plugin` (renamed from `oidc_flavor`, Nextcloud
plugin selector). sys-stk-backend, sys-svc-compose, sys-svc-proxy
and plugins/filter/compose_volumes.py all gate on
`lookup('sso', ..., 'is_proxy_gated')` instead of manually combining
`enabled` and `flavor`.
* Identifier renames repo-wide: `services.<entity>.ports.local.oauth2`
→ `ports.local.sso`; `PORT_BANDS.local.oauth2` → `PORT_BANDS.local.sso`;
`OAUTH2_PROXY_*` env/Jinja vars → `SSO_PROXY_*`;
`oauth2_proxy_cookie_secret` → `sso_proxy_cookie_secret`;
`OAUTH2_SERVICE_ENABLED` + `OIDC_SERVICE_ENABLED` collapse into
`SSO_SERVICE_ENABLED`; Playwright gating helpers
(isServiceEnabled/skipUnlessServiceEnabled/requireService/safeIsEnabled/
safeSkipUnlessEnabled) move from `oidc`/`oauth2` to `sso`.
* Dual-block resolution: per req's Background-Table, web-app-bookwyrm
is `flavor: oauth2`; web-app-gitea and web-app-friendica keep
`flavor: oauth2` with their existing `oauth2.acl.blacklist`
(/user/login) and `oauth2.acl.whitelist` (federation + discovery
endpoints) — preserving the operative oauth2-proxy gating.
* Tests: the two mutual-exclusion guards
(tests/integration/iam/oauth2_oidc/test_mutual_exclusive.py and
test_acl_mutual_exclusive.py) are deleted per Decision #10;
test_oauth2_contract.py is renamed and rewritten to
test_sso_contract.py, asserting that `flavor: oauth2` roles carry
`oauth2.origin.{host,port}` plus `ports.local.sso`; test_sso.py
(web-app integration) now requires the unified `sso:` block with an
explicit flavor or a `# nocheck: sso` opt-out; new
tests/lint/repository/no_legacy_sso_paths/ guards the source tree
against re-introducing the legacy strings.
* Docs: req 21 itself has every AC checked off, and reqs 006, 008,
009, 013, 015, 016, 019, 020 plus 10+ role READMEs and
docs/contributing/actions/testing/suppression.md are updated to the
unified SSO naming.
Empirically validated end-to-end against the validation app set
(web-app-keycloak, web-app-nextcloud, web-app-bookwyrm, web-app-gitea,
web-app-friendica, web-app-dashboard) in matrix variants 0 and 1; v2
also passes the 4 target roles' Playwright after a documented
pre-existing KC_HOSTNAME-drift workaround (separate concern, not
caused by this refactor).
BREAKING CHANGE: every consumer of `services.oauth2.*`,
`services.oidc.*`, `OAUTH2_PROXY_*`, `OAUTH2_SERVICE_ENABLED`,
`OIDC_SERVICE_ENABLED`, `oauth2_proxy_cookie_secret`,
`oauth2_proxy_application_id`, `ports.local.oauth2`, the
`web-app-oauth2-proxy` role, or the legacy gating helper service names
(`oidc`/`oauth2`) must migrate to the unified `sso` shape documented
in docs/requirements/021-sso-flavor-migration.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`categories_of()` now emits the sub-level key with the same hyphen
the role-id already uses (`web-app`, `svc-bkp`, `sys-ctl`) instead of
the dot-form (`web.app`, `svc.bkp`, `sys.ctl`). Reads cleanly against
the role naming convention and removes the only place in the
matching layer where role-id segments got rewritten.
Updates the matching unit tests and the only consumer annotation
that referenced the dot form (`sys-ctl-cln-bkps`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Move role_is_invokable / role_has_required_by / role_has_nocheck out
of plugins/filter (where they were flagged as test-only filters) into
utils/roles/required_by_coverage.py. All three use load_yaml_any,
read_text, and ROLE_FILE_META_SERVICES from the canonical SPOTs.
- Refactor the verifier (cli.meta.roles.services.called) the same way:
iterate roles/*/ directly, look up meta/services.yml via the constant,
parse with load_yaml_any, and tolerate a missing roles_dir (unit tests
point at a fake /repo).
- Make the mailu-dependency integration test yaml-aware for meta/
services.yml: walk the parsed tree and skip values inside any
`required_by` subtree. `required_by` is inverse coverage metadata for
the deploy verifier, not a "consumer of mailu" reference, so previous
regex-on-text matching produced false positives for cert-deploy.
- sys-ctl-mtn-cert-renew is gated by `lookup('tls', application_id,
'mode') == 'letsencrypt'`, which the verifier can't express via
role-id/category match — mark nocheck with the reason inline.
- Add README.md for cli/meta/roles/services (CLI-category convention).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Asserts at deploy time that every role declaring `required_by` in its
meta/services.yml actually executed (at least one TASK block ending with
a non-skip status of `ok` / `changed` / `fatal` / `included`). Catches
silent coverage regressions where a once-flag swallows an include before
the role's body can run.
CLI: `python -m cli.meta.roles.services.called --logfile X --apps a,b,c`
Library: `cli.meta.roles.services.called.verify(...)` supports per-pass
slicing via `log_byte_offset` and reads either host-side log files or
in-container logs when `container=` is given.
Hooked into `cli.administration.deploy.development.deploy._run_deploy`
so matrix-deploy runs (CI + local make compose-deploy mode=initialize/
reinstall) snapshot the log offset before each compose.exec and verify
the resulting slice after rc=0. Defaults ANSIBLE_LOG_PATH to
/tmp/infinito-deploy.log so the check also runs locally without env-var
setup.
Includes 35 self-contained unit tests covering category derivation,
TASK-block status parsing (incl. ANSI escape stripping + substring
role-id guard + multi-skip edge case), required_role_ids YAML scan,
host log slicing with byte offsets, end-to-end verify, and CLI argparse.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reworks the REPL into a stateful shell that tracks a category path: typing a bare category name cd's into it, `/` returns to root, `..` walks one level up (chained as `../..` etc.), path tokens with slashes or a leading `..` navigate, `infinito [path]` always jumps absolute, and `ls` renders the current folder with a 🗂️/⚙️ emoji column, name, and a 30-char-truncated description (folders sorted before commands). Executable leaves (sub-packages with `__main__.py`) run instead of cd-ing into them. The prompt prints `🖥️ infinito <space-separated path>> `; the screen clears (incl. scrollback via `\033[3J`) before each command run, and the run echoes the resolved invocation in the fresh frame so the user sees what is about to execute.
Splits `repl.py` (~250 lines) into focused modules: `constants.py` (sentinels), `navigation.py` (path/category resolve + prompt + normalize), `ls.py` (folder rendering + truncate), `runner.py` (subprocess dispatch + clear_screen), `banner.py` (startup banner). `repl.py` is now a thin `main()` loop with backward-compat re-exports so external consumers keep the `repl.<symbol>` surface. Mirrors the split on the test side into `test_navigation.py`, `test_runner.py`, `test_ls.py`, and a `test_repl.py` that owns only main-loop integration with a shared `_MainLoopHarness` mixin.
Adds `scripts/tests/environment/11_console.sh` as a CI smoke that feeds a navigation sequence through the REPL and asserts every prompt shape plus the `ls` emoji output, so a regression in the prompt format, the path resolver, or the folder rendering is caught by the environment-test orchestrator.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four follow-ups uncovered by \`make test\` after 3c9db6142:
* tests/lint/ansible/variables/test_role_and_group_vars_used:
TLS_ENABLED is declared in group_vars/all/02_tls.yml but never
read via \`{{ TLS_ENABLED }}\`; it is consumed by
plugins/lookup/{tls,email,cert}.py, which the YAML/Jinja-scoped
lint cannot see. Tag the declaration with the project's standard
\`# nocheck: unused-var\` marker and document the consumer.
* tests/lint/filesystem/python/test_no_raw_project_walk: the new
test_no_ansible_env_lookups.py used \`Path.rglob("*")\` for its
walk, which is the exact shape the no-raw-walks lint forbids in
test code. Switch to \`iter_project_files(extensions=_SCAN_EXTS)\`
with a relative-path prefix filter; same coverage, shared cache.
* tests/unit/cli/administration/deploy/development/test_inventory_vars_file_spot:
the new lint test legitimately names the dev inventory path
(\`inventories/development/default.yml\`) in its failure message.
Add it to the SPOT-drift-guard allowlist with a justification.
* tests/integration/ansible/jinja/test_variable_definitions: the
debug task added in 0884eb4e2 prints \`ansible_env\` and \`vars\`,
both Ansible magic variables. Add them to the test's reserved-name
list so the static analyzer stops flagging them as undefined.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wrap `python -m cli` in a host-side REPL launched via `make console`. The
loop normalizes input (strips leading `infinito`/`cli`, treats bare `help`
as `--help`), ignores Ctrl+C so accidental input cancels don't kill the
session, and exits only on `exit`/`quit`/`:q` or Ctrl+D. Includes a banner
with help/web/docs/license/author hints. Drops the now-redundant
`infinito --help` footer from `make help`. Adds 22 unit tests covering
normalization, subprocess dispatch, and main-loop control flow.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root cause of the DinD DNS-check failure on test-deploy-server
(networks.internet.ip4 falling through to 127.0.0.1 despite
INFINITO_IP4=172.30.0.10 reaching every link of the env pipeline):
cli/administration/inventory/provision/cli.py declared --ip4 with a
hardcoded default of "127.0.0.1" and ensure_host_vars_file wrote that
value unconditionally into host_vars/<host>.yml. group_vars'
defaults_networks.internet.ip4 resolves "172.30.0.10" via
\`lookup('env', 'INFINITO_IP4')\`, but tasks/stages/01_constructor.yml
merges \`defaults_networks | combine(networks | default({}, true))\` so
host_vars wins. INFINITO_IP4 looked broken end-to-end; the breakage
actually lived in the provision-CLI host_vars overlay.
INFINITO_DNS_IP and INFINITO_DOMAIN propagated correctly because
the provision CLI had no --dns and the --primary-domain write was
None-gated and never invoked.
While here, --ssl-disabled and --primary-domain are removed too:
neither had any effect in the dev flow. provision/cli.py applies
\`--vars-file inventories/development/default.yml\` AFTER
ensure_host_vars_file via apply_vars_overrides_from_file, whose
_deep_update_commented_map unconditionally overwrites
(\`target[key] = value\`). The dev vars-file always set
\`TLS_ENABLED: true\`, so \`--ssl-disabled\` (which forced
\`TLS_ENABLED: false\` in host_vars) was systematically reverted to
true. Dev/CI ran with TLS enabled the whole time. Inventory stays at
\`TLS_ENABLED: true\` to preserve that effective behaviour.
Removed from cli/administration/inventory/provision/cli.py:
- --ip4, --ip6: defaulted to 127.0.0.1 / ::1, shadowed group_vars
- --primary-domain: never set by any caller, None-gated write
- --ssl-disabled: always re-overwritten by --vars-file later in the
same provision pass
Removed from cli/administration/inventory/provision/host_vars.py the
matching writes in ensure_host_vars_file; the function now only sets
\`ansible_connection: local\` for localhost-class hosts. Defaults for
DOMAIN_PRIMARY / TLS_ENABLED / networks.internet.{ip4,ip6} live in
group_vars (env-driven) and the inventory vars-file (deployment-wide
flags); host_vars stops duplicating them.
Callers cleaned up:
- cli/administration/deploy/development/inventory/builder.py
- cli/administration/deploy/container/command.py
- scripts/tests/deploy/local/utils/fresh-kept-all-init-and-deploy.sh
- scripts/tests/deploy/local/reset/inventory.sh
Tests + docs follow:
- tests/unit/cli/administration/inventory/provision/test_host_vars.py
now asserts host_vars no longer carries baked DOMAIN_PRIMARY /
TLS_ENABLED / networks defaults
- docs/administration/deploy.md drops --ssl-disabled from the
provision example
- docs/contributing/design/inventory.md re-scopes the host_vars
column: the DOMAIN_PRIMARY / TLS_ENABLED / networks.internet.{ip4,ip6}
values arrive via --vars / --vars-file overlays, otherwise group_vars
provides the env-driven default
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
compose.yml no longer hides services behind \`profiles: ["ci"]\`. The
profile existed solely as a no-op safety net against a bare
\`docker compose up\` in the repo root, but the top-level compose.yml is
already explicitly scoped to local dev / CI testing (production goes
through the Ansible roles under roles/<role>/templates/), so the safety
net is redundant with the documented scope.
Every \`--profile ci\` callsite collapses:
- compose.yml: \`profiles: ["ci"]\` removed from coredns + infinito
- Profile.args() deleted; Profile keeps is_ci() and
registry_cache_active() for the cache-override gating
- compose.py / down.py: profile.args() splat removed from the
docker-compose command builder; down.py drops its now-unused
Profile import
- 5 shell wrappers (install, lint, tests/code, meta/resolve/apps,
meta/resolve/diff/affected_roles): plain \`docker compose\` instead
of \`docker compose --profile ci\`
- tests/unit/.../test_profile.py: TestProfileArgs class removed
- tests/unit/.../test_compose.py: two cmd-shape asserts updated to
reflect the missing --profile pair
Docs:
- docs/contributing/artefact/files/compose.yml.md: the
\"every service MUST be behind a profiles: entry\" rule is gone
- docs/contributing/actions/debugging/ci.md: the
\`docker compose ... --profile ci config\` debug snippet drops the
flag; while touching the file the two em-dash clause separators in
the reproduction-profiles shell block are rewritten to colons
(documentation.md em-dash rule), and one list-item in
compose.yml.md is collapsed to one sentence per line for
consistency with its siblings
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
With env.development and env/ci.env both gone, env/default.env was the
last inhabitant of the directory. Hoisting it to the repo root removes
the empty-shell indirection: the SPOT is now [default.env](default.env),
sitting next to .env (its generated companion) and other repo-root
configuration artefacts like compose.yml, Dockerfile, and Makefile.
Path updates land in:
- generator: cli/meta/env/__main__.py (REPO_ROOT / "default.env")
- workflow: .github/workflows/test-install-make.yml (awk reads from
default.env, error message updated in lockstep)
- lint tests: test_used_keys_declared.py and test_infinito_prefix.py
switched from globbing env/*.env to targeting default.env directly;
test_default_env_keys_handled.py points at the new path
- drift test: test_inventory_vars_file_spot.py whitelist + STATIC_ENV
- smoke test: tests/integration/meta/env/test_dotenv_generator.py
STATIC_ENV + every docstring/error message
Doc-string and prose references in utils/env/*, scripts/*,
cli/administration/deploy/development/{common,coredns}.py,
Makefile, Dockerfile, and every doc page that mentioned env/default.env
are rewritten in the same pass so future readers see one path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The four ci.env keys (INFINITO_DNS_IP, INFINITO_DOMAIN, INFINITO_DOMAIN_RE,
INFINITO_IP4) were only honored by docker compose --env-file, so processes
that sourced .env via load.sh saw INFINITO_IP4 as unset. As a result
networks.internet.ip4 in the Ansible playbook fell through to the
127.0.0.1 default and the DinD DNS-check handler failed on the deploy
server matrix.
Moving the defaults into env/default.env makes them flow through the same
.env / load.sh / os.environ chain as every other INFINITO_* key, so
\${INFINITO_IP4} in compose.yml, lookup('env', 'INFINITO_IP4') in
group_vars, and envsubst over the CoreDNS Corefile template all see the
same value deterministically.
Side fix: utils/env/parser.py now undoes the writer's \\\\/\\" escaping for
quoted values, closing a round-trip gap that surfaced once
INFINITO_DOMAIN_RE (the only value containing a literal backslash) started
flowing through the generator.
All --env-file env/ci.env call sites (compose.py, down.py, install/lint/
test wrappers, resolve scripts, test-dns workflow + script) are removed.
coredns.py now reads .env via parse_static_env, docs and unit/integration
tests follow.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Relocates `cli/meta/{ports,networks/suggest}` to
`cli/contributing/network/{ports,address/suggest}` so port and subnet
allocators live alongside the rest of the contributor-side tooling
(mirror, update, changelog) instead of `meta`. Mirrors the tests
into `tests/unit/cli/contributing/network/` and renames the contract
docs to `docs/contributing/tools/network/{port,address}.md`. Prog
names update from `cli meta ports suggest` / `cli meta networks
suggest` to `cli contributing network ports suggest` / `cli
contributing network address suggest`; the corresponding lint
references and `utils.meta.scan` docstring follow.
Adds a third subcommand: `cli contributing network diagnose`
(stdlib-only DNS / TCP / TLS / PMTU probe against the standard
infrastructure hosts plus interface MTU, `/etc/resolv.conf`,
`/etc/hosts`, proxy env, CA-bundle summaries). Split across `cli.py`
(orchestration), `config.py` (SPOT for hosts / per-distro install
commands), `format.py` (output + subprocess helpers), `tools.py`
(distro-aware auto-install of `iputils-ping` + `iproute2`),
`report.py` (system-info dumps + IPv6-default-route detection), and
`probes.py` (per-host probes). Exposed as `make debug-network`.
Companion `docs/contributing/actions/debugging/network.md` absorbs
the diagnose-tool walkthrough (incl. how to read MTU-induced TLS
handshake timeouts) so the merged page covers DNS / MTU / IPv6 /
connectivity end-to-end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Promote the key from GHA-only (gha_passthrough) to always-emitted (passthrough)
so the deploy CLI can read it strictly via os.environ[...]. The previous
INFINITO_NO_BUILD layout left the key unset locally, which crashed
cli/.../up.py and cli/.../compose.py with KeyError after the env-pipeline
refactor that introduced gha_passthrough.
New semantics:
INFINITO_BUILD=1 build the local image (default; local mode)
INFINITO_BUILD=0 skip build, pull image (CI overrides via -e)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pulls VENV/PYTHON/PIP/PYTHONPATH out of the dotenv handler pipeline
into scripts/meta/python/detect.sh. load.sh now sources detect.sh and
prepends ${VENV}/bin to PATH above the INFINITO_ENV_LOADED guard, so
every BASH_ENV re-entry re-resolves PYTHON the moment a venv
materialises mid-chain — the same fix as b79ed2869 but at the SPOT,
which retires the reload-trick from ansible.sh, python.sh and
pre-commit.sh.
Also renames INVENTORY_VARS_FILE → INFINITO_INVENTORY_VARS_FILE under
the new env-prefix lint, drops the 5 dead INFINITO_ENV_*_LOADED unsets
left over from the multi-loader era, and adds a lint guarding the
INFINITO_ prefix invariant on env/*.env keys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three new lint tests close gaps in the INFINITO_* SPOT contract:
* tests/lint/filesystem/env/test_default_env_keys_handled.py
Every default.env key must be wired through a handler.
* tests/lint/filesystem/env/test_used_keys_declared.py
Every INFINITO_* in .sh/.py must live in env/{default,ci}.env or
a handler; per-line `# nocheck:` suppresses runtime-only refs.
* tests/lint/filesystem/python/test_no_infinito_get_defaults.py
Forbid ANY default in `.get("INFINITO_*", ...)` -- including the
empty-string fallback (which silently masks an unset key). The
nocheck check walks every physical line the call spans so an
autoformat-split call still honours the marker.
Cache namespace flattens to a single prefix:
INFINITO_PACKAGE_CACHE_* -> INFINITO_CACHE_PACKAGE_*
INFINITO_REGISTRY_CACHE_* -> INFINITO_CACHE_REGISTRY_*
The five previously driver-injected paths (APT_LIST, NPMRC,
PIP_CONF, REGISTRY_PROXY_CONF, FRONTEND_CA_FILE) move into
env/default.env and passthrough.STATIC_KEYS; the
`cache_env_overrides()` helper and its two call sites are gone
along with the test-fixture override for FRONTEND_CA_DIR.
SPOT-strict rewrites:
- 19 sites: `something.get("INFINITO_X", "")` -> subscript
`something["INFINITO_X"]` (loud KeyError when env loader did
not run; all keys live in env/default.env).
- 2 handler-internal bootstrap reads suppressed with
`# nocheck: handler-bootstrap-read`.
- 30 runtime-only / driver-injected / test-fixture references
annotated with `# nocheck: <reason>` per-line.
- Unit tests that previously removed SPOT-guaranteed keys via
`{k: v ... if k \!= KEY}` or `clear=True` switch to setting
the key to "" explicitly, matching the new contract.
Side fixes:
- INFINITO_DNS_DOMAIN + INFINITO_DNS_HOSTS_FILE move from
shell-side defaults in common.sh to env/default.env and
passthrough.STATIC_KEYS.
- dns_systemd_is_operational() drops the dead
INFINITO_DNS_FORCE_HOSTS_FALLBACK override branch (the
variable was never set anywhere).
- INFINITO_WORKER_FETCH switches from `os.environ.get(KEY, "1")`
to `os.environ[KEY]`.
- scripts/tests/code/wrapper.sh exec args move into a bash
array so `# nocheck:` on INFINITO_TEST_TYPE can live on its
own element line (SC2215 fix).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rename env/static.env → env/default.env across the entire repo
(string label, doc links, Python file-path constants) — the
`parse_static_env*` function names stay as a stable API surface.
Sort all `*.env` / `*.env.j2` entries alphabetically (52 in
default.env, 4 in ci.env, 46 of 49 j2 templates; 3 templates
with Jinja2 `{% %}` control flow are skipped because conditional
blocks would tear apart under a naive sort). Sort all 83
Makefile target blocks alphabetically; .PHONY group headers
stay in place.
Collapse the two remaining multi-line Makefile recipe comments
(dotenv-force, test-signed) to single-line WHY comments.
Add four lint tests under tests/lint/filesystem/ to keep the
invariants enforced going forward:
- env/test_alphabetical_sort.py
- env/test_single_line_comments.py (# nocheck: markers exempt)
- makefile/test_alphabetical_targets.py
- makefile/test_single_line_comments.py (# nocheck: markers exempt)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace 8 bash env modules with a single Python generator
(cli.meta.env) that materialises env/static.env + runtime context
into a gitignored .env. Each variable lives in its own handler
under utils/env/handlers/; defaults carry inline one-line `# ...`
comments.
Rename to the INFINITO_* schema:
- workflow inputs: APPS, SERVICES_DISABLED, CMD, DEBUG, WHITELIST,
BUNDLES, FULL_CYCLE, VARIANT, RUN_FLAGS, LIMIT_HOST, PARALLEL
- inventory: INVENTORY_DIR / INVENTORY_FILE / HOST_VARS_FILE,
PW_FILE -> INVENTORY_PASSWORD_FILE
- venv: VENV_BASE / VENV_FALLBACK / VENV_NAME
- new SPOT defaults: COMPILE_SILENCE, CONTAINER_NO_CA,
WAIT_HEALTH_TIMEOUT_S, PACKAGE_CACHE_FRONTEND_INIT_IMAGE,
promoted IMAGE_TAG from GHA-only to general passthrough
compose.yml hardening:
- inline \${VAR:-default} removed; strict \${VAR:?...} everywhere
- new integration lint blocks regressions
Tooling:
- scripts/install/all.sh owns the stamp-file install logic
(Makefile target delegates)
- scripts/meta/env/load.sh is the single shell entry point;
8 bash modules deleted
- new shell lint: no INFINITO_* defaults outside env/static.env
Tests:
- 93 new unit tests for utils/env/{builder,parser,runtime,writer}
- restructure tests/lint into repository/ (file-content rules) vs
filesystem/{shell,python,env}/ (filesystem-shape rules)
- drift + smoke integration tests against the generator
Docs:
- variables.md is now a pointer; per-variable docs live next to
the key in env/static.env / env/ci.env
- compose.yml.md drops the Default column
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Matrix-deploy orchestrator now pushes `VARIANT_INDEX` (0-based round
id) alongside the existing `ASYNC_ENABLED` extra var for both PASS 1
(sync) and PASS 2 (async) runs. Test-e2e-playwright's archive step
consumes both to derive a per-(variant, pass) destination so a
subsequent round cannot overwrite the previous one's reports.
- New `roles/test-e2e-playwright/files/archive_playwright_reports.sh`
moves the playwright-junit.xml / playwright-report/ / test-results/
outputs into `variant-<N>/<sync|async>/` under the role's reports
dir, idempotently (re-running the same tuple replaces rather than
fails). `run_one.yml` invokes the script via `ansible.builtin.script`
so the YAML stays a single 7-line task.
- `VARIANT_INDEX: 0` lives in `defaults/main.yml` to keep the variable
defined for dev / non-matrix invocations. Role defaults are not
considered SPOT, so the `| default(0)` belt-and-suspenders inside
the script invocation stays valid against the spot-default lint.
- Summary script now renders an 8-column table: Time | App | Variant |
Pass | Test | Status | Duration | Message. Per-test ISO timestamps
are derived from the JUnit `<testsuite timestamp>` attribute plus the
cumulative duration of preceding cases; file mtime is the fallback
when the attribute is absent. App / Variant / Pass are extracted
from the artefact path layout produced by the archive script.
- Matrix-deploy unit tests refreshed to expect the new
`extra_ansible_vars` shape; summary unit tests gain coverage for
`_to_iso_z`, `_parse_iso_timestamp`, the four supported path
layouts, and the file-mtime fallback (55 tests total).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Widens the cache-read lint guard from ``tests/lint/`` to every ``.py``
under ``tests/``. Integration tests show the same compounding cost
pattern (one role's compose template / nginx conf / scrape fragment
read across many shape tests) that motivated the original lint for
lint/.
Converts 91 violations across 28 test files:
- **Cache (utils.cache.files.read_text)** for project-tracked reads
and single-read tempdir fixtures, e.g. test_prometheus.py (32x),
test_keycloak_import_flow.py (11x), test_dashboard_oidc.py (3x),
test_changelog_md.py (8x), test_package_mirror.py (8x), plus
test_logging.py / test_summarize.py / test_version_updater.py /
test_get_url_retry.py / test_pull_specific_host.py.
- **Cache + UnicodeDecodeError handler** for project-walk scanners
that previously used ``errors="replace"`` / ``errors="ignore"`` to
defend against non-UTF-8 files (compose volumes, password quoting,
template image, mailu dependency, URL reachability). Wraps each
read in ``try/except (OSError, UnicodeDecodeError): continue``,
which has the same observable effect (skip undecodable files).
- **Refactor away the binary heuristic** in test_no_raw_docker.py:
``read_bytes() + b"\x00" not in raw`` becomes
``read_text() / except UnicodeDecodeError``. Same files skipped,
with cache benefit.
- **Variable extraction** in test_changelog_md.py / test_package_mirror.py
for cases where ``self.assertEqual(path.read_text(...), expected)``
exceeded ruff's 88-char limit. Ruff was wrapping the marker onto a
line outside the AST span of the read_text call.
- **Compute-from-setup** for two test methods that previously read
``self.changelog`` twice (before/after) to verify a no-op:
``before`` is now derived from the known input, dropping the first
read.
Five ``# nocheck: cache-read`` markers remain, each annotated inline
with the reason: idempotency tests where the second read must hit
disk to catch non-idempotent regressions, and tests with intermediate
``write_text`` whose subsequent read would return stale cache data.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`3c1d5d821` solved the cross-round backbone-abriss problem by making
`include` equal the union of every variant's closure for every round.
Side-effect: every matrix round then ALSO deployed the full union, so
`08_deploy_performance.sh` saw `dashboard.infinito.example` still
answering HTTP 200 after a matomo-only `deploy-fresh-purged-apps` —
the deploy step was no longer matomo-only at all.
Split the two concerns instead:
- `include` (PlanEntry[3]) goes back to the per-round variant-closure.
Both `init`/`builder` (`--include` for the inventory bake) and
`deploy` (`--id` for ansible) consume it. Each round therefore
deploys ONLY its own variant-closure again — the pre-`3c1d5d821`
semantics.
- `purge_set` (PlanEntry[4], new) carries the union across all rounds.
`deploy.py` uses it for the inter-round wipe instead of the previous
round's include. Result: between rounds the WHOLE matrix footprint
is purged, not just the previous round's variant slice — a variant
pinning `services.<X>.shared: false` cannot leave its bundled
provider on the host for a sibling round that wires the same service
shared. The backbone-abriss `3c1d5d821` was hunting stays fixed:
whichever round needs the backbone re-deploys it immediately after
the wipe (purge runs at the start of each round, not the end).
`previous_round_include` tracking is gone. The first iteration skips
the purge (nothing to wipe yet), which also covers `total_rounds == 1`
and `--variant N` pinned single-folder modes cleanly.
Unrelated, but staged together because it is part of the same
investigation:
- `roles/dev-nix/templates/nix.conf.j2`: enable `keep-derivations` /
`keep-outputs`, so `nix-collect-garbage -d` (run by the aggressive
prune step) does not drop the build closure of pkgmgr's flake; this
was making every deploy re-fetch the build inputs from
cache.nixos.org and overshoot pkgmgr's update timeout.
- `roles/pkgmgr/tasks/01_core.yml`: raise the pkgmgr update timeout
from 600s to 1800s and lower the retry count from 6 to 3, so a slow
fetch on a single try no longer pushes total wallclock past three
hours.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Commit 16f323319 added a strict variant-only include rule + conflict
detector in `inventory/variants.py`, gated behind the presence of any
primary's `meta/variants.yml`. In practice the rule never paid off:
* The CI failure that motivated it (matomo's matrix transition leaving
a stale dashboard nginx vhost -> 502) is already addressed by the
nginx-purge primitive shipped in commit bfd04e4a1.
* The existing `CombinedResolver` path already implements the "any
role says true or `in group_names` -> pull" semantics via
`utils.roles.applications.services.registry.is_explicit_truth`, so
every needed service still gets pulled and services that all roles
mark False stay out of the include.
* The strict literal-equality conflict detector was a false-positive
factory: 79/80 variant-equipped primaries failed it against the
current repo state because roles legitimately disagree on `enabled`
flags for shared services.
This commit removes the entire variant-only branch and restores the
single-path planner. Round-index clamping for non-primary deps is left
exactly as before 16f323319 -- only the additions are reverted.
The file-split refactor from 16f323319 is intentionally kept; only the
`variants.py` submodule, the corresponding `test_variants.py`, and the
variant-only gate test class in `test_planner.py` are removed. The
`__init__.py` re-exports + `__all__` are pruned to match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two changes ship together because the variant-only matrix-deploy logic
was written into the legacy 697-line inventory.py and then re-split:
* Source package (max 240 lines per submodule, well under the 300 budget):
- spec.py DevInventorySpec + PlanEntry data types
- payload.py _resolve_variant_payloads + _bake_overrides
- legacy_resolver.py CombinedResolver fallback (default-mode path)
- variants.py strict variant-only include resolution
- planner.py plan_dev_inventory_matrix + filter_plan_to_variant
- builder.py build_dev_inventory + build_dev_inventory_matrix
- __init__.py public-API re-exports + back-compat shims
* Test package mirrors the source 1:1 with a shared _fixtures.make_spec
to avoid duplicating the DevInventorySpec defaults across files.
* New variant-only matrix mode (gated on any primary shipping
meta/variants.yml): only services whose merged variant payload pins
both enabled and shared to literal True become inventory groups;
Jinja flags never qualify (group_names does not exist yet at build
time); run_after only orders the include, never expands it;
cross-primary disagreement on a (service-key, round) raises ValueError.
Default mode (no variants.yml on any primary) keeps the legacy
CombinedResolver path untouched.
* Docs in design/{inventory,variants}.md updated to point at the new
per-submodule paths; test_nginx_vhosts.py's Test classes now also
inherit directly from unittest.TestCase so the AST-level lint in
test_all_test_files_have_tests.py picks them up (transitive base via
NginxVhostsTestBase alone is not enough).
Coverage: 458 cli unit tests, 349 utils unit tests, 99 dev-deploy tests,
all green; ruff check clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Consolidates image declarations under a single schema: every role's
`<svc>.{image,version}` lives in `meta/services.yml` alongside compose
service entries. The `image` lookup now reads from disk per role,
`iter_role_images` collapses to a single source, and the mirror chain
(resolver + apply_mirror_overrides) drops the dead `images:` branch.
`test-e2e-playwright` resolves `TEST_E2E_PLAYWRIGHT_IMAGE` directly via
`lookup('image', 'playwright', 'ref')` in vars/main.yml; the runtime
set_fact + override-empty-string dance is gone.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four additive changes to `infinito meta roles applications complexity`:
* Render two new columns for the one-hop view — `embeds_direct` /
`consumers_direct` — alongside the existing transitive `embeds` /
`consumers`. The direct fields are independent of `--level`: a
reader can always see what the role embeds / is-embedded-by
without its indirect tail.
* Render a `total` column that sums the four numeric columns. It
serves as a coarse "overall touch points" score; `--sort total`
surfaces the most-coupled roles.
* `--order {asc,desc}` (default `asc`) controls sort direction.
Applies to every `--sort` choice; `--sort total --order desc`
puts the hubs at the top of the table.
* `--filter SUBSTRING` (case-insensitive) keeps only rows whose
role name contains SUBSTRING. The complexity scores are computed
against the full role tree first, then filtering applies to the
rendered rows only — a role's transitive counts still reflect the
whole codebase.
The tuple shape grew by appending the new fields, so existing
index 0..4 assertions in the unit tests stay valid; new
assertions cover the direct-vs-transitive distinction and the
total computation.
Bonus: trailing-newline fix on `roles/web-svc-file/meta/services.yml`
(the autoformat surfaced a missing EOL after the recent dashboard
cleanup).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surface the reverse direction of the shared-service dependency graph
in `infinito meta roles applications complexity`. Each row now also
counts and lists the application roles that transitively embed *this*
role (the "consumers" / "consumed-by" closure), with the same
`--level` depth cap as the forward direction.
* Forward closure (column `embeds`, list `services`) and reverse
closure (column `consumers`, list `consumed_by`) share a single
BFS over (forward, reverse) adjacency maps built once per
invocation.
* `--sort` accepts `embeds` (default), `consumers`, or `name`. The
former `points` label is renamed to `embeds` because the column
header reads as a sentence with the role name (e.g. `web-app-foo
embeds 12, consumers 78`), which is sharper than a generic
"points" score.
* `--format cli` (default) renders three numeric columns — name,
embeds, consumers — for a compact terminal view.
`--format json` emits the full payload with the resolved role
lists for each direction.
* Tests cover the chain case in both directions and verify that
`--level` caps the reverse-direction BFS the same way.
Strip every `req <NNN>`, `req-<NNN>`, `Requirement <NNN>`,
`requirement-<NNN>`, and `docs/requirements/<NNN>-...md` mention from
non-Markdown source files. Comments anchored to requirement numbers
rot when requirements are renumbered or rewritten and conflate
present-state behaviour (which the comment should describe) with the
audit trail of how the code got there (which belongs in commit
messages, PR descriptions, and the requirement file itself).
* Document the rule in `docs/contributing/documentation.md` under
"Comments". Code MUST describe the *why* in present-state language;
contract enforcement lives in tests, not folklore.
* Clean 254 source files across `roles/`, `tests/`, `plugins/`,
`cli/`, `scripts/`, `utils/`, `group_vars/`, and the repo-root
`eslint.config.js`. Pure comment / docstring / log-message edits;
no behavioural change.
* Add `tests/lint/repository/test_no_req_references_in_code.py` to
enforce the rule going forward across `.js`, `.py`, `.yml`,
`.yaml`, `.j2`, and `.sh` files. The legacy
`test_no_stale_requirement_refs.py` is exempt as the only file
whose purpose is to detect these very tokens.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Removes 67 live references to docs/requirements/<NNN>-*.md files across
43 contributor docs, role READMEs, code comments, task files, and tests.
Requirement docs are initial-creation-only artefacts; live references
freeze a snapshot that rots once the per-feature SPOT (a contributor
doc, a code module, a lint test) takes over the live behaviour.
In the same pass, fixes documentation.md violations on the touched
files: drops em-dashes used as thought breaks, removes "SPOT for ..."
self-labels, strips historical phrasing ("the legacy", "previously",
"used to", "is no longer used", "matching the historical"), and
reflows soft-wrapped sentences so each sentence stays on one logical
line.
The `test_inventory_vars_file_spot.py` whitelist for the literal
inventory path drops the docs/requirements/003 entry and instead
skips docs/requirements/ entirely; future requirement docs that
mention the literal path are then ignored automatically.
Verified by tests/lint/repository/test_no_stale_requirement_refs.py
(green tree-wide).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Caps the BFS over each role's shared-service deps:
- `--level 1` returns direct deps only.
- `--level 2` adds their direct deps.
- Omitting the flag keeps the unbounded full-closure behaviour.
Argparse rejects `--level < 1` with rc=2.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
For each application role, recursively resolve its shared-service
dependency closure and report (name, points, services-csv).
- BFS over each role's meta/services.yml; provider role per service-key
comes from utils.roles.applications.services.registry.
- Default truth predicate is is_explicit_truth so the
'<role>' in group_names Jinja form counts as a dep; --no-group-names
restricts to literal `true` flags only.
- Sort: --sort points (default, asc, name as tiebreaker) | --sort name (asc).
- Tests cover chain resolution, group-names toggle, self-not-counted,
non-application roles skipped.
Plus: roles/desk-gnome/meta/services.yml -- yaml autoformat drift.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Package renames (dynamic discovery picks up new names automatically):
- cli/build/role_include/ -> cli/build/include/
- cli/build/roles_list/ -> cli/build/list/
- cli/meta/roles/lifecycle_filter/ -> cli/meta/roles/lifecycle/
Domain logic moved out of cli/ (no CLI surface, just a helper class):
- cli/meta/roles/applications/resolution/services/{resolver,errors}.py
-> utils/roles/applications/services/{resolver,errors}.py
- drop dead services/yaml_utils.py + its only test
Bug fixes:
- cli/meta/{callorder -> roles/callorder} dropped its hard-coded
parents[3] PROJECT_ROOT (broke after the directory move and resolved
to <repo>/cli/); it now reuses utils.PROJECT_ROOT.
Help renderer (cli/core/help.py + colors.py):
- DIM color on the description column to separate it from the command name
- Blank line between the long command name and its wrapped description
- Italic invocation under the category heading + leading blank line
- New Style.ITALIC
Doc updates: path references in docs/requirements/00{8,10}*.md plus
soft-wrap fixes in the touched regions.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Module moves (with imports, paths, argv literals, scripts, docs, and
parents[N] formulas updated in lockstep):
- cli/deploy -> cli/administration/deploy
- cli/validate/inventory -> cli/administration/inventory/validate
- cli/vault -> cli/administration/secrets/vault
- cli/administration/inventory/encrypt -> cli/administration/secrets/seal
- cli/mirror -> cli/contributing/mirror
The new cli/administration/secrets cluster collocates the whole-file
ansible-vault wrapper next to the inventory-aware per-field sealer.
The user-facing CLI subcommand paths follow the package layout, e.g.
"infinito administration deploy dedicated <inv>" and
"infinito administration inventory validate <inv>".
Help system redesign in cli/core/{discovery,help,app}.py:
- Discovery now distinguishes runnable commands (directories with
__main__.py) from category folders (directories that only group
further commands). Empty support packages are filtered out.
- Each category folder MAY carry a README.md; its H1 heading is used
as the title and its first paragraph as the one-line description in
the help output.
- "infinito" / "infinito --help" prints only the top-level entries
(one level deep). "infinito <category>" navigates into a folder and
shows its sub-folders and commands one level deep, with the README
title as the page heading and the invocation as a dim subtitle.
- New "--tree [N]" flag walks the tree under the (optional) path
prefix recursively. N caps the depth (1-based); omit N for an
unbounded walk. The output uses Unicode tree connectors, an emoji
per entry (folder for categories, gear for commands), a globally
aligned description column, and multi-line wrapped descriptions
with tree-continuity bars on continuation lines.
- Category and command listings use the full invocation
("infinito meta callorder") rather than the bare leaf name, so the
output is copy-pasteable.
Convention doc + 17 CLI READMEs:
- New docs/contributing/artefact/files/cli/readme_md.md describes the
H1+paragraph layout the help system reads, and how it differs from
role/docs READMEs.
- README.md added for cli/{administration, build, contributing, meta}
and their immediate sub-folders, plus cli/meta/roles/applications
and its resolution sub-folder. Each follows the "title with trailing
emoji + single descriptive paragraph" convention.
Mirror resolver/sync description strings (the help system reads them
from argparse --help output) added so they no longer surface as
"options:" in the global index. cli/mirror/cleanup already had a
description and is unchanged.
Fixes uncovered by the move:
- import unittest.mock added to four dedicated-deploy tests so
unittest.mock.patch resolves under stdlib unittest discovery
(Python 3.13 no longer surfaces unittest.mock through "import
unittest" alone).
- parents[N] index in the new
cli/administration/inventory/validate/__init__.py corrected from
3 to 4 to match the deeper layout.
- test_paths.INVENTORY_VALIDATOR_PATH assertion and test_runner
substring assertions updated to match the new validator path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
TC (flake8-type-checking): typing-only imports moved into
``if TYPE_CHECKING:`` blocks across 8 source files via auto-fix
(117 hits). Reduces runtime import surface and rules out circular
imports for type-only deps. Project-wide ``from __future__ import
annotations`` (FA, already enabled) keeps annotations as strings,
so the moved imports never need to load at runtime.
N (pep8-naming): 32 hits handled.
Renames (semantic + lint):
- ``isSmallerThenMaximumBackupSize`` -> ``is_larger_than_maximum_backup_size``
(snake_case + fixes both the typo "Then -> Than" and the
semantic mismatch: the function returns ``current > maximum``)
- ``isDirectoryDeletable`` / ``deleteVersion`` /
``getAmountOfIteration`` / ``deleteIteration`` -> snake_case
- 8 test functions with ALL-CAPS env-var fragments
(``test_..._DOMAIN_PRIMARY_...`` etc.) -> snake_case
- Mock alias ``VH`` -> ``vh`` in test_host_vars.py
Exception class renames (``N818`` Error suffix):
- ``TimeLimitExceededException`` -> ``TimeLimitExceededError``
- ``AttemptException`` -> ``AttemptError``
- ``PathNotFound`` -> ``PathNotFoundError`` (propagated across
4 callers under tests/integration/lookups/config/)
- ``_ExitJson`` / ``_FailJson`` -> ``*_Error``
Refactor:
- ``METADATA = {...}`` local constant in two functions of
``plugins/filter/invokable_paths.py`` lifted to module-level
``_METADATA_KEYS``.
- ``from typing import TYPE_CHECKING`` in
``tasks/utils/migrate_meta_layout/__main__.py`` moved above
the ``sys.path.insert`` hack to satisfy E402 (collateral from
the TC auto-fix that landed it after the path mutation).
Per-file ignores in pyproject.toml for the legitimate cases the
rule cannot accept:
- ``pull-specific-host.py`` -> ``N999`` (Ansible-invoked role-level
script, hyphenated filename pinned by the surrounding stack).
- 4 matrix-deploy files -> ``N806`` for the ``include_R`` /
``raw_include_R`` round-index notation; the uppercase ``R``
here mirrors the formula in the docstring rather than naming
a constant.
- ``plugins/filter/value_generator.py`` -> ``N806`` for the
lazy-loaded class binding ``ValueGenerator =
_value_generator_cls()``, which retains the imported class
identifier per PEP 8.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cleanup logic between rounds is now full-include rather than
variant-changed-only:
- handler() tracks previous_round_include instead of
previous_round_variants. Between rounds the wrapper purges every
app from the previous round's include set, so round R boots from
an empty host.
- Every round deploys its full round_include (round 0 baseline AND
R>0). The "skip variant-unchanged apps" path is gone, because
variants that drop deps left cross-round residue under the
previous selective-purge model.
- _purge_app_entities docstring updated; the matrix-deploy comment
in handler() reflects the new semantics.
test_matrix_deploy.py:
- test_round_one_only_deploys_apps_with_real_variant_one renamed
test_every_round_redeploys_full_include_after_purge.
- test_three_round_plan_only_re_deploys_apps_with_real_variant
renamed test_three_round_plan_redeploys_full_include_each_round.
- two_round + full_cycle purge assertions updated to expect the
full previous-round include.
- New test_round_transition_purges_previous_include_even_when_dep_falls_out
pins the fall-out case (round 0 = nextcloud + coturn, round 1 =
nextcloud only -> purge MUST include coturn).
variants.md updated to match (points 4 + 5 rewritten, reference
table aligned). documentation.md compliance pass: dropped em-dash
clause separator and the historical optimisation reference on
point 4, lifted reference-table entry into the new wording.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop four one-shot or now-redundant tools from cli/fix/:
* vars_main_files — bulk-created vars/main.yml with application_id
per role-prefix during the SPOT migration. The schema is now
enforced declaratively by tests/integration/roles/applications/id/
test_matches_role_name.py and the role-mapping SPOT lint.
* move_unnecessary_dependencies — moved meta dependencies to guarded
include_role on first sweep. Permanent enforcement lives in
tests/integration/roles/meta/dependencies/test_unnecessary.py.
* tabs — replaced literal tab characters with two-space indentation
in YAML files. Autoformat covers new occurrences.
* ini_py — generic helper that filled missing __init__.py files; new
packages now ship them from the start.
Move the remaining (still-actively-used) docker version updater out
from under cli/fix/ and into cli/contributing/, where the recurring
maintenance tooling lives:
* cli/fix/docker_image_versions → cli/contributing/docker/update_versions
Update the corresponding wiring:
* scripts/github/update_docker_image_versions.sh now calls
python -m cli.contributing.docker.update_versions
(the entry point invoked daily by .github/workflows/update.yml).
* docs/requirements/008-role-meta-layout.md references the new path.
* PROJECT_ROOT in __init__.py drops one parents[] level (the package
sits one folder deeper now).
cli/fix/ is fully removed; new contributor-facing tooling belongs
under cli/contributing/.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
