Commit Graph
18 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 4.7 10cb02461e fix(inventory): bake only variants.yml overrides into matrix-deploy --vars
Since 576fad84f (2026-05-10) moved role images from
`defaults/main.yml` into `meta/services.yml`, matrix-deploy's variant
baker has been writing the FULL effective config (services.yml +
variants.yml deep-merged via `get_variants`) into host_vars through
the `--vars` JSON consumed by `apply_vars_overrides`. That populates
`applications.{role}.services.{svc}.image` (and `.version`, .cpus,
etc.) with the upstream value, which then silently disables
`apply_mirror_overrides` for every role with an explicit image
declaration: the default `if_missing` policy short-circuits when the
field is already non-blank, so the rendered compose / Dockerfile FROM
keeps pulling from the upstream registry rather than the GHCR mirror.

The regression went unnoticed because Docker Hub usually serves the
pull, but it eventually surfaced as a BuildKit 429 on
`docker.io/library/php:8.2-apache` during the suitecrm Dockerfile
build (job 77602110531 on run 26362676803). Nextcloud, moodle,
mariadb consumers, postgres consumers, simpleicons, phpmyadmin,
phpldapadmin, jenkins, baserow, fider, ... all share the same latent
defect — only suitecrm happened to hit the rate limit first.

Fix the root cause instead of papering over per-role with
`mirror_policy: force`: introduce `get_variant_overrides_only` next
to `get_variants` and switch payload.py to it. The new function
returns the raw `meta/variants.yml` entries per role — no merge with
`meta/services.yml`. matrix-deploy now bakes only the variant-
specific overrides into host_vars; the role's services.yml fields
flow through the normal Ansible lookup chain at deploy time, leaving
`image`/`version` blank in host_vars so `apply_mirror_overrides`
populates them from `mirrors.yml` under its default `if_missing`
policy.

Behavior preserved for the other `get_variants` consumers:
* utils/manager/inventory.py:104 (schema-driven credentials) keeps
  the full merged variant payload.
* planner.py:58 (variant-count only) is unaffected.
* legacy_resolver.py:42 (CombinedResolver runtime services_overrides)
  keeps the full merged view.

Coverage:
* 3 new tests in test_applications.py for the new helper (single
  empty variant when no meta/variants.yml, raw override without base
  merge, cache invariance).
* test_payload.py + test_builder.py mocks repointed from `get_variants`
  to `get_variant_overrides_only`.
* `make test` green across all 5 targets (lint, unit, integration,
  external, test-lint) — 188 tests total.

Side-effects assessed:
* Repo-wide grep confirms no template / task accesses
  `applications.X.services.Y.image` directly via hostvars — every
  consumer goes through `lookup('config', …)` which still deep-merges.
* Roles previously pulling upstream now resolve via the GHCR mirror
  whenever the image is already synced (the `images-mirror-missing.yml`
  workflow keeps the destination populated).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 17:33:33 +02:00
kevinveenbirkenbachandClaude Opus 4.7 a10555cd08 feat(inventory): variant-aware credentials generation
InventoryManager.resolve_schema_includes_recursive previously read the
root role's base meta/services.yml when discovering shared providers,
regardless of which matrix variant the planner had pinned for the
round. For a role whose variants.yml enables additional services per
variant (e.g. svc-bkp-container-2-local variant 2 enabling
services.ldap), the base config had no `enabled+shared` flag for those
services, so the recursive resolver never visited the variant's
shared providers — their credentials were never generated, and the
deploy crashed in the provider's env.j2 with "credentials is defined
in schema at roles/<provider>/meta/schema.yml but not set in
application '<provider>'".

Wires the active variant through the existing inventory pipeline:

- utils/manager/inventory.py: InventoryManager(variant=None|int). When
  set, load_role_config_by_path overlays variants.yml[N] onto the root
  role's meta config (only for the root — transitive providers keep
  their own base lookup). variant=None preserves the pre-refactor base
  behavior for every standalone caller.
- cli/administration/inventory/credentials: new --variant N flag,
  forwarded to InventoryManager.
- cli/administration/inventory/provision: new --app-variants <JSON>
  flag ({app_id: variant_index}), forwarded per-app to the credentials
  subprocess.
- cli/administration/deploy/development/inventory/builder.py: emits
  --app-variants from spec.variant_selectors() when the round pins any
  variant. Apps without variants.yml collapse to a single empty
  overlay in get_variants so this is a no-op for them.

10 unit tests cover: variant overlay for root role, variant=None
fallback to base, out-of-range fallback, no overlay on non-root
paths, --variant CLI forwarding (set + default), --app-variants
generator forwarding (set + omit), builder --app-variants emission
(set + omit).

Caveat: bkp-c2l CI variant 1 still loads svc-db-openldap transitively
at runtime (outside the planner's variant closure) — a separate
sys-service-loader / role-meta bug, not addressed here.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 13:04:20 +02:00
kevinveenbirkenbachandClaude Opus 4.7 9d7854bd2e fix(lint): wire new env-lookup guard into the existing lint contracts
Four follow-ups uncovered by \`make test\` after 3c9db6142:

* tests/lint/ansible/variables/test_role_and_group_vars_used:
  TLS_ENABLED is declared in group_vars/all/02_tls.yml but never
  read via \`{{ TLS_ENABLED }}\`; it is consumed by
  plugins/lookup/{tls,email,cert}.py, which the YAML/Jinja-scoped
  lint cannot see. Tag the declaration with the project's standard
  \`# nocheck: unused-var\` marker and document the consumer.

* tests/lint/filesystem/python/test_no_raw_project_walk: the new
  test_no_ansible_env_lookups.py used \`Path.rglob("*")\` for its
  walk, which is the exact shape the no-raw-walks lint forbids in
  test code. Switch to \`iter_project_files(extensions=_SCAN_EXTS)\`
  with a relative-path prefix filter; same coverage, shared cache.

* tests/unit/cli/administration/deploy/development/test_inventory_vars_file_spot:
  the new lint test legitimately names the dev inventory path
  (\`inventories/development/default.yml\`) in its failure message.
  Add it to the SPOT-drift-guard allowlist with a justification.

* tests/integration/ansible/jinja/test_variable_definitions: the
  debug task added in 0884eb4e2 prints \`ansible_env\` and \`vars\`,
  both Ansible magic variables. Add them to the test's reserved-name
  list so the static analyzer stops flagging them as undefined.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 17:57:14 +02:00
kevinveenbirkenbachandClaude Opus 4.7 13c93ad6d5 refactor(compose): drop the ci profile and its --profile ci arg everywhere
compose.yml no longer hides services behind \`profiles: ["ci"]\`. The
profile existed solely as a no-op safety net against a bare
\`docker compose up\` in the repo root, but the top-level compose.yml is
already explicitly scoped to local dev / CI testing (production goes
through the Ansible roles under roles/<role>/templates/), so the safety
net is redundant with the documented scope.

Every \`--profile ci\` callsite collapses:
- compose.yml: \`profiles: ["ci"]\` removed from coredns + infinito
- Profile.args() deleted; Profile keeps is_ci() and
  registry_cache_active() for the cache-override gating
- compose.py / down.py: profile.args() splat removed from the
  docker-compose command builder; down.py drops its now-unused
  Profile import
- 5 shell wrappers (install, lint, tests/code, meta/resolve/apps,
  meta/resolve/diff/affected_roles): plain \`docker compose\` instead
  of \`docker compose --profile ci\`
- tests/unit/.../test_profile.py: TestProfileArgs class removed
- tests/unit/.../test_compose.py: two cmd-shape asserts updated to
  reflect the missing --profile pair

Docs:
- docs/contributing/artefact/files/compose.yml.md: the
  \"every service MUST be behind a profiles: entry\" rule is gone
- docs/contributing/actions/debugging/ci.md: the
  \`docker compose ... --profile ci config\` debug snippet drops the
  flag; while touching the file the two em-dash clause separators in
  the reproduction-profiles shell block are rewritten to colons
  (documentation.md em-dash rule), and one list-item in
  compose.yml.md is collapsed to one sentence per line for
  consistency with its siblings

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 16:22:33 +02:00
kevinveenbirkenbachandClaude Opus 4.7 12a56e449d refactor(env): move env/default.env to repo-root default.env and drop env/
With env.development and env/ci.env both gone, env/default.env was the
last inhabitant of the directory. Hoisting it to the repo root removes
the empty-shell indirection: the SPOT is now [default.env](default.env),
sitting next to .env (its generated companion) and other repo-root
configuration artefacts like compose.yml, Dockerfile, and Makefile.

Path updates land in:
- generator: cli/meta/env/__main__.py (REPO_ROOT / "default.env")
- workflow: .github/workflows/test-install-make.yml (awk reads from
  default.env, error message updated in lockstep)
- lint tests: test_used_keys_declared.py and test_infinito_prefix.py
  switched from globbing env/*.env to targeting default.env directly;
  test_default_env_keys_handled.py points at the new path
- drift test: test_inventory_vars_file_spot.py whitelist + STATIC_ENV
- smoke test: tests/integration/meta/env/test_dotenv_generator.py
  STATIC_ENV + every docstring/error message

Doc-string and prose references in utils/env/*, scripts/*,
cli/administration/deploy/development/{common,coredns}.py,
Makefile, Dockerfile, and every doc page that mentioned env/default.env
are rewritten in the same pass so future readers see one path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 16:22:33 +02:00
kevinveenbirkenbachandClaude Opus 4.7 07a9440ac7 refactor(env): fold ci.env into default.env so INFINITO_IP4/DNS/DOMAIN reach Ansible
The four ci.env keys (INFINITO_DNS_IP, INFINITO_DOMAIN, INFINITO_DOMAIN_RE,
INFINITO_IP4) were only honored by docker compose --env-file, so processes
that sourced .env via load.sh saw INFINITO_IP4 as unset. As a result
networks.internet.ip4 in the Ansible playbook fell through to the
127.0.0.1 default and the DinD DNS-check handler failed on the deploy
server matrix.

Moving the defaults into env/default.env makes them flow through the same
.env / load.sh / os.environ chain as every other INFINITO_* key, so
\${INFINITO_IP4} in compose.yml, lookup('env', 'INFINITO_IP4') in
group_vars, and envsubst over the CoreDNS Corefile template all see the
same value deterministically.

Side fix: utils/env/parser.py now undoes the writer's \\\\/\\" escaping for
quoted values, closing a round-trip gap that surfaced once
INFINITO_DOMAIN_RE (the only value containing a literal backslash) started
flowing through the generator.

All --env-file env/ci.env call sites (compose.py, down.py, install/lint/
test wrappers, resolve scripts, test-dns workflow + script) are removed.
coredns.py now reads .env via parse_static_env, docs and unit/integration
tests follow.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 14:55:10 +02:00
kevinveenbirkenbachandClaude Opus 4.7 81b57f6f50 refactor(env): rename INFINITO_NO_BUILD -> INFINITO_BUILD with inverted truthiness
Promote the key from GHA-only (gha_passthrough) to always-emitted (passthrough)
so the deploy CLI can read it strictly via os.environ[...]. The previous
INFINITO_NO_BUILD layout left the key unset locally, which crashed
cli/.../up.py and cli/.../compose.py with KeyError after the env-pipeline
refactor that introduced gha_passthrough.

New semantics:
  INFINITO_BUILD=1  build the local image  (default; local mode)
  INFINITO_BUILD=0  skip build, pull image (CI overrides via -e)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 02:30:33 +02:00
kevinveenbirkenbachandClaude Opus 4.7 80c396b839 refactor(env+install/spot): collapse python/venv resolution into shell SPOT
Pulls VENV/PYTHON/PIP/PYTHONPATH out of the dotenv handler pipeline
into scripts/meta/python/detect.sh. load.sh now sources detect.sh and
prepends ${VENV}/bin to PATH above the INFINITO_ENV_LOADED guard, so
every BASH_ENV re-entry re-resolves PYTHON the moment a venv
materialises mid-chain — the same fix as b79ed2869 but at the SPOT,
which retires the reload-trick from ansible.sh, python.sh and
pre-commit.sh.

Also renames INVENTORY_VARS_FILE → INFINITO_INVENTORY_VARS_FILE under
the new env-prefix lint, drops the 5 dead INFINITO_ENV_*_LOADED unsets
left over from the multi-loader era, and adds a lint guarding the
INFINITO_ prefix invariant on env/*.env keys.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 00:39:52 +02:00
kevinveenbirkenbachandClaude Opus 4.7 eb4bcd23a8 refactor(env/spot): forbid .get() defaults + cache namespace rename
Three new lint tests close gaps in the INFINITO_* SPOT contract:
* tests/lint/filesystem/env/test_default_env_keys_handled.py
  Every default.env key must be wired through a handler.
* tests/lint/filesystem/env/test_used_keys_declared.py
  Every INFINITO_* in .sh/.py must live in env/{default,ci}.env or
  a handler; per-line `# nocheck:` suppresses runtime-only refs.
* tests/lint/filesystem/python/test_no_infinito_get_defaults.py
  Forbid ANY default in `.get("INFINITO_*", ...)` -- including the
  empty-string fallback (which silently masks an unset key). The
  nocheck check walks every physical line the call spans so an
  autoformat-split call still honours the marker.

Cache namespace flattens to a single prefix:
  INFINITO_PACKAGE_CACHE_*  -> INFINITO_CACHE_PACKAGE_*
  INFINITO_REGISTRY_CACHE_* -> INFINITO_CACHE_REGISTRY_*
The five previously driver-injected paths (APT_LIST, NPMRC,
PIP_CONF, REGISTRY_PROXY_CONF, FRONTEND_CA_FILE) move into
env/default.env and passthrough.STATIC_KEYS; the
`cache_env_overrides()` helper and its two call sites are gone
along with the test-fixture override for FRONTEND_CA_DIR.

SPOT-strict rewrites:
  - 19 sites: `something.get("INFINITO_X", "")` -> subscript
    `something["INFINITO_X"]` (loud KeyError when env loader did
    not run; all keys live in env/default.env).
  - 2 handler-internal bootstrap reads suppressed with
    `# nocheck: handler-bootstrap-read`.
  - 30 runtime-only / driver-injected / test-fixture references
    annotated with `# nocheck: <reason>` per-line.
  - Unit tests that previously removed SPOT-guaranteed keys via
    `{k: v ... if k \!= KEY}` or `clear=True` switch to setting
    the key to "" explicitly, matching the new contract.

Side fixes:
  - INFINITO_DNS_DOMAIN + INFINITO_DNS_HOSTS_FILE move from
    shell-side defaults in common.sh to env/default.env and
    passthrough.STATIC_KEYS.
  - dns_systemd_is_operational() drops the dead
    INFINITO_DNS_FORCE_HOSTS_FALLBACK override branch (the
    variable was never set anywhere).
  - INFINITO_WORKER_FETCH switches from `os.environ.get(KEY, "1")`
    to `os.environ[KEY]`.
  - scripts/tests/code/wrapper.sh exec args move into a bash
    array so `# nocheck:` on INFINITO_TEST_TYPE can live on its
    own element line (SC2215 fix).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 22:26:16 +02:00
kevinveenbirkenbachandClaude Opus 4.7 22cf6f38f1 refactor(env+make): rename static.env→default.env + sort + lint gates
Rename env/static.env → env/default.env across the entire repo
(string label, doc links, Python file-path constants) — the
`parse_static_env*` function names stay as a stable API surface.

Sort all `*.env` / `*.env.j2` entries alphabetically (52 in
default.env, 4 in ci.env, 46 of 49 j2 templates; 3 templates
with Jinja2 `{% %}` control flow are skipped because conditional
blocks would tear apart under a naive sort). Sort all 83
Makefile target blocks alphabetically; .PHONY group headers
stay in place.

Collapse the two remaining multi-line Makefile recipe comments
(dotenv-force, test-signed) to single-line WHY comments.

Add four lint tests under tests/lint/filesystem/ to keep the
invariants enforced going forward:
- env/test_alphabetical_sort.py
- env/test_single_line_comments.py        (# nocheck: markers exempt)
- makefile/test_alphabetical_targets.py
- makefile/test_single_line_comments.py   (# nocheck: markers exempt)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 22:26:16 +02:00
kevinveenbirkenbachandClaude Opus 4.7 9a578952c3 feat(env): SPOT env-pipeline + INFINITO_* namespace
Replace 8 bash env modules with a single Python generator
(cli.meta.env) that materialises env/static.env + runtime context
into a gitignored .env. Each variable lives in its own handler
under utils/env/handlers/; defaults carry inline one-line `# ...`
comments.

Rename to the INFINITO_* schema:
- workflow inputs: APPS, SERVICES_DISABLED, CMD, DEBUG, WHITELIST,
  BUNDLES, FULL_CYCLE, VARIANT, RUN_FLAGS, LIMIT_HOST, PARALLEL
- inventory: INVENTORY_DIR / INVENTORY_FILE / HOST_VARS_FILE,
  PW_FILE -> INVENTORY_PASSWORD_FILE
- venv: VENV_BASE / VENV_FALLBACK / VENV_NAME
- new SPOT defaults: COMPILE_SILENCE, CONTAINER_NO_CA,
  WAIT_HEALTH_TIMEOUT_S, PACKAGE_CACHE_FRONTEND_INIT_IMAGE,
  promoted IMAGE_TAG from GHA-only to general passthrough

compose.yml hardening:
- inline \${VAR:-default} removed; strict \${VAR:?...} everywhere
- new integration lint blocks regressions

Tooling:
- scripts/install/all.sh owns the stamp-file install logic
  (Makefile target delegates)
- scripts/meta/env/load.sh is the single shell entry point;
  8 bash modules deleted
- new shell lint: no INFINITO_* defaults outside env/static.env

Tests:
- 93 new unit tests for utils/env/{builder,parser,runtime,writer}
- restructure tests/lint into repository/ (file-content rules) vs
  filesystem/{shell,python,env}/ (filesystem-shape rules)
- drift + smoke integration tests against the generator

Docs:
- variables.md is now a pointer; per-variable docs live next to
  the key in env/static.env / env/ci.env
- compose.yml.md drops the Default column

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 20:40:34 +02:00
kevinveenbirkenbachandClaude Opus 4.7 0aa34b903e feat(playwright): persist artifacts per variant+pass, add Time column
- Matrix-deploy orchestrator now pushes `VARIANT_INDEX` (0-based round
  id) alongside the existing `ASYNC_ENABLED` extra var for both PASS 1
  (sync) and PASS 2 (async) runs. Test-e2e-playwright's archive step
  consumes both to derive a per-(variant, pass) destination so a
  subsequent round cannot overwrite the previous one's reports.
- New `roles/test-e2e-playwright/files/archive_playwright_reports.sh`
  moves the playwright-junit.xml / playwright-report/ / test-results/
  outputs into `variant-<N>/<sync|async>/` under the role's reports
  dir, idempotently (re-running the same tuple replaces rather than
  fails). `run_one.yml` invokes the script via `ansible.builtin.script`
  so the YAML stays a single 7-line task.
- `VARIANT_INDEX: 0` lives in `defaults/main.yml` to keep the variable
  defined for dev / non-matrix invocations. Role defaults are not
  considered SPOT, so the `| default(0)` belt-and-suspenders inside
  the script invocation stays valid against the spot-default lint.
- Summary script now renders an 8-column table: Time | App | Variant |
  Pass | Test | Status | Duration | Message. Per-test ISO timestamps
  are derived from the JUnit `<testsuite timestamp>` attribute plus the
  cumulative duration of preceding cases; file mtime is the fallback
  when the attribute is absent. App / Variant / Pass are extracted
  from the artefact path layout produced by the archive script.
- Matrix-deploy unit tests refreshed to expect the new
  `extra_ansible_vars` shape; summary unit tests gain coverage for
  `_to_iso_z`, `_parse_iso_timestamp`, the four supported path
  layouts, and the file-mtime fallback (55 tests total).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-17 15:30:06 +02:00
kevinveenbirkenbachandClaude Opus 4.7 bd1a3a69ad fix(deploy/development/inventory/planner): split per-round include from union purge_set
`3c1d5d821` solved the cross-round backbone-abriss problem by making
`include` equal the union of every variant's closure for every round.
Side-effect: every matrix round then ALSO deployed the full union, so
`08_deploy_performance.sh` saw `dashboard.infinito.example` still
answering HTTP 200 after a matomo-only `deploy-fresh-purged-apps` —
the deploy step was no longer matomo-only at all.

Split the two concerns instead:

- `include` (PlanEntry[3]) goes back to the per-round variant-closure.
  Both `init`/`builder` (`--include` for the inventory bake) and
  `deploy` (`--id` for ansible) consume it. Each round therefore
  deploys ONLY its own variant-closure again — the pre-`3c1d5d821`
  semantics.
- `purge_set` (PlanEntry[4], new) carries the union across all rounds.
  `deploy.py` uses it for the inter-round wipe instead of the previous
  round's include. Result: between rounds the WHOLE matrix footprint
  is purged, not just the previous round's variant slice — a variant
  pinning `services.<X>.shared: false` cannot leave its bundled
  provider on the host for a sibling round that wires the same service
  shared. The backbone-abriss `3c1d5d821` was hunting stays fixed:
  whichever round needs the backbone re-deploys it immediately after
  the wipe (purge runs at the start of each round, not the end).

`previous_round_include` tracking is gone. The first iteration skips
the purge (nothing to wipe yet), which also covers `total_rounds == 1`
and `--variant N` pinned single-folder modes cleanly.

Unrelated, but staged together because it is part of the same
investigation:

- `roles/dev-nix/templates/nix.conf.j2`: enable `keep-derivations` /
  `keep-outputs`, so `nix-collect-garbage -d` (run by the aggressive
  prune step) does not drop the build closure of pkgmgr's flake; this
  was making every deploy re-fetch the build inputs from
  cache.nixos.org and overshoot pkgmgr's update timeout.
- `roles/pkgmgr/tasks/01_core.yml`: raise the pkgmgr update timeout
  from 600s to 1800s and lower the retry count from 6 to 3, so a slow
  fetch on a single try no longer pushes total wallclock past three
  hours.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 00:44:47 +02:00
kevinveenbirkenbachandClaude Opus 4.7 a2b5269244 refactor(inventory): drop unused variant-only matrix mode
Commit 16f323319 added a strict variant-only include rule + conflict
detector in `inventory/variants.py`, gated behind the presence of any
primary's `meta/variants.yml`. In practice the rule never paid off:

* The CI failure that motivated it (matomo's matrix transition leaving
  a stale dashboard nginx vhost -> 502) is already addressed by the
  nginx-purge primitive shipped in commit bfd04e4a1.
* The existing `CombinedResolver` path already implements the "any
  role says true or `in group_names` -> pull" semantics via
  `utils.roles.applications.services.registry.is_explicit_truth`, so
  every needed service still gets pulled and services that all roles
  mark False stay out of the include.
* The strict literal-equality conflict detector was a false-positive
  factory: 79/80 variant-equipped primaries failed it against the
  current repo state because roles legitimately disagree on `enabled`
  flags for shared services.

This commit removes the entire variant-only branch and restores the
single-path planner. Round-index clamping for non-primary deps is left
exactly as before 16f323319 -- only the additions are reverted.

The file-split refactor from 16f323319 is intentionally kept; only the
`variants.py` submodule, the corresponding `test_variants.py`, and the
variant-only gate test class in `test_planner.py` are removed. The
`__init__.py` re-exports + `__all__` are pruned to match.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 04:25:07 +02:00
kevinveenbirkenbachandClaude Opus 4.7 f102c229f0 refactor(inventory): split monolith into SRP submodules + variant-only mode
Two changes ship together because the variant-only matrix-deploy logic
was written into the legacy 697-line inventory.py and then re-split:

* Source package (max 240 lines per submodule, well under the 300 budget):
  - spec.py            DevInventorySpec + PlanEntry data types
  - payload.py         _resolve_variant_payloads + _bake_overrides
  - legacy_resolver.py CombinedResolver fallback (default-mode path)
  - variants.py        strict variant-only include resolution
  - planner.py         plan_dev_inventory_matrix + filter_plan_to_variant
  - builder.py         build_dev_inventory + build_dev_inventory_matrix
  - __init__.py        public-API re-exports + back-compat shims

* Test package mirrors the source 1:1 with a shared _fixtures.make_spec
  to avoid duplicating the DevInventorySpec defaults across files.

* New variant-only matrix mode (gated on any primary shipping
  meta/variants.yml): only services whose merged variant payload pins
  both enabled and shared to literal True become inventory groups;
  Jinja flags never qualify (group_names does not exist yet at build
  time); run_after only orders the include, never expands it;
  cross-primary disagreement on a (service-key, round) raises ValueError.
  Default mode (no variants.yml on any primary) keeps the legacy
  CombinedResolver path untouched.

* Docs in design/{inventory,variants}.md updated to point at the new
  per-submodule paths; test_nginx_vhosts.py's Test classes now also
  inherit directly from unittest.TestCase so the AST-level lint in
  test_all_test_files_have_tests.py picks them up (transitive base via
  NginxVhostsTestBase alone is not enough).

Coverage: 458 cli unit tests, 349 utils unit tests, 99 dev-deploy tests,
all green; ruff check clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 02:17:26 +02:00
kevinveenbirkenbachandClaude Opus 4.7 c55ad69c41 refactor: drop requirement-file references from code, add doc rule + lint
Strip every `req <NNN>`, `req-<NNN>`, `Requirement <NNN>`,
`requirement-<NNN>`, and `docs/requirements/<NNN>-...md` mention from
non-Markdown source files. Comments anchored to requirement numbers
rot when requirements are renumbered or rewritten and conflate
present-state behaviour (which the comment should describe) with the
audit trail of how the code got there (which belongs in commit
messages, PR descriptions, and the requirement file itself).

* Document the rule in `docs/contributing/documentation.md` under
  "Comments". Code MUST describe the *why* in present-state language;
  contract enforcement lives in tests, not folklore.
* Clean 254 source files across `roles/`, `tests/`, `plugins/`,
  `cli/`, `scripts/`, `utils/`, `group_vars/`, and the repo-root
  `eslint.config.js`. Pure comment / docstring / log-message edits;
  no behavioural change.
* Add `tests/lint/repository/test_no_req_references_in_code.py` to
  enforce the rule going forward across `.js`, `.py`, `.yml`,
  `.yaml`, `.j2`, and `.sh` files. The legacy
  `test_no_stale_requirement_refs.py` is exempt as the only file
  whose purpose is to detect these very tokens.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 16:25:22 +02:00
kevinveenbirkenbachandClaude Opus 4.7 256e027525 docs: drop stale requirement refs + reflow soft-wraps to one sentence per line
Removes 67 live references to docs/requirements/<NNN>-*.md files across
43 contributor docs, role READMEs, code comments, task files, and tests.
Requirement docs are initial-creation-only artefacts; live references
freeze a snapshot that rots once the per-feature SPOT (a contributor
doc, a code module, a lint test) takes over the live behaviour.

In the same pass, fixes documentation.md violations on the touched
files: drops em-dashes used as thought breaks, removes "SPOT for ..."
self-labels, strips historical phrasing ("the legacy", "previously",
"used to", "is no longer used", "matching the historical"), and
reflows soft-wrapped sentences so each sentence stays on one logical
line.

The `test_inventory_vars_file_spot.py` whitelist for the literal
inventory path drops the docs/requirements/003 entry and instead
skips docs/requirements/ entirely; future requirement docs that
mention the literal path are then ignored automatically.

Verified by tests/lint/repository/test_no_stale_requirement_refs.py
(green tree-wide).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-09 05:00:46 +02:00
kevinveenbirkenbachandClaude Opus 4.7 a26bfc4e94 refactor(cli): reorg admin/contributing tree + per-folder help system
Module moves (with imports, paths, argv literals, scripts, docs, and
parents[N] formulas updated in lockstep):

- cli/deploy             -> cli/administration/deploy
- cli/validate/inventory -> cli/administration/inventory/validate
- cli/vault              -> cli/administration/secrets/vault
- cli/administration/inventory/encrypt -> cli/administration/secrets/seal
- cli/mirror             -> cli/contributing/mirror

The new cli/administration/secrets cluster collocates the whole-file
ansible-vault wrapper next to the inventory-aware per-field sealer.
The user-facing CLI subcommand paths follow the package layout, e.g.
"infinito administration deploy dedicated <inv>" and
"infinito administration inventory validate <inv>".

Help system redesign in cli/core/{discovery,help,app}.py:

- Discovery now distinguishes runnable commands (directories with
  __main__.py) from category folders (directories that only group
  further commands). Empty support packages are filtered out.
- Each category folder MAY carry a README.md; its H1 heading is used
  as the title and its first paragraph as the one-line description in
  the help output.
- "infinito" / "infinito --help" prints only the top-level entries
  (one level deep). "infinito <category>" navigates into a folder and
  shows its sub-folders and commands one level deep, with the README
  title as the page heading and the invocation as a dim subtitle.
- New "--tree [N]" flag walks the tree under the (optional) path
  prefix recursively. N caps the depth (1-based); omit N for an
  unbounded walk. The output uses Unicode tree connectors, an emoji
  per entry (folder for categories, gear for commands), a globally
  aligned description column, and multi-line wrapped descriptions
  with tree-continuity bars on continuation lines.
- Category and command listings use the full invocation
  ("infinito meta callorder") rather than the bare leaf name, so the
  output is copy-pasteable.

Convention doc + 17 CLI READMEs:

- New docs/contributing/artefact/files/cli/readme_md.md describes the
  H1+paragraph layout the help system reads, and how it differs from
  role/docs READMEs.
- README.md added for cli/{administration, build, contributing, meta}
  and their immediate sub-folders, plus cli/meta/roles/applications
  and its resolution sub-folder. Each follows the "title with trailing
  emoji + single descriptive paragraph" convention.

Mirror resolver/sync description strings (the help system reads them
from argparse --help output) added so they no longer surface as
"options:" in the global index. cli/mirror/cleanup already had a
description and is unchanged.

Fixes uncovered by the move:

- import unittest.mock added to four dedicated-deploy tests so
  unittest.mock.patch resolves under stdlib unittest discovery
  (Python 3.13 no longer surfaces unittest.mock through "import
  unittest" alone).
- parents[N] index in the new
  cli/administration/inventory/validate/__init__.py corrected from
  3 to 4 to match the deeper layout.
- test_paths.INVENTORY_VALIDATOR_PATH assertion and test_runner
  substring assertions updated to match the new validator path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 20:17:09 +02:00