Commit Graph
19 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 5 5b8465260f feat(ci): deploy only the variants a change reaches
An image bump in a provider redeployed every variant of every consumer,
including the variants that switch that provider off and therefore never
deploy the bumped image at all. The diff-derived whitelist now narrows each
affected role to the variants whose own round closure still reaches a seed,
emitted as `role#0,2` selection tokens. That closure is walked on the same
variant-merged services maps the matrix planner builds its rounds from, so the
narrowing follows a parent's variant choice across every generation of the
chain. It covers every auto-derived whitelist: the weekly update PRs,
Dependabot, and any other diff-driven run.

The narrowing never shrinks the matrix on partial information. A seed role
keeps every variant, because its own files changed. A role reached only
through a run_after edge keeps every variant too, because the round closure
does not follow that edge and a role whose hits are empty is therefore
unnarrowed rather than dropped.

A pinned token naming a role the discovery query does not return now warns
instead of aborting the run. The diff-derived whitelist legitimately names
roles the lifecycle envelope filters out, and once it pins variants such a
token would abort every chunk; role names are still checked against roles/
before the run starts, so a typo is caught one layer earlier.

The round's variant merge moves out of the deploy inventory package into
utils/roles/applications/variants.py. The CI selection has to build the same
maps, and importing the deploy package for them pulls in a module that reads
INFINITO_SRC_DIR at import time, which the runner host does not set.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-21 22:51:03 +02:00
kevinveenbirkenbachandClaude Opus 5 05fcec52e7 fix(planner): give every variant-carrying app the round's index, not only the include
round_variants was built by walking round_include. When a primary's variant switches its couplings off the planner resolves no prerequisites, round_include collapses to the primary, and no dependency gets an index at all. derive_includes then evaluates them at base config, where an unrendered Jinja coupling counts as enabled, so the closure inflates to everything reachable instead of what the variant declares.

Measured: svc-ai-litellm round 1 deploys 10 apps instead of 32, web-app-bigbluebutton round 1 and web-app-peertube round 1 drop from 34 to 10 and 11. Every other round of every role measured is unchanged, including all fourteen nextcloud rounds. The apps that leave are not dependencies of the round: svc-ai-litellm declares litellm, lmstudio, ollama and postgres and no sso at all, while bigbluebutton and peertube declare sso but switch it off in variant 1 together with dashboard, matomo, css, logout and prometheus.

complexity/cli.py:300 already documents the rule this makes true: the planner hands each pulled-in dependency round_index if round_index is below its variant count else 0, and pulls_at_zero implements exactly that. The coverage model was describing behaviour the planner only delivered for apps inside the include.

init.py filters the printed variant map to the apps actually shown, because round_variants now carries 266 entries and round 1 would otherwise print 112 of them on one line while ten apps deploy.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 13:29:43 +02:00
kevinveenbirkenbachandClaude Opus 5 f1635e6a59 feat(variants): let a variant dictate the config of the providers it pulls in
A round deploys a provider once, so the role that pulls it in is the only place that can say how lean it has to be. A variant entry may now carry any application config topic under a service it depends on: a mapping deep-merges, so naming one key changes that key alone, and null empties the topic. Empty was the case that needed a marker of its own, because merging an empty mapping is a no-op and would leave exactly the services the round was trying to shed.

The topic vocabulary and the merge semantics live in utils.roles.applications.topics, which is also where resources.py now gets them rather than spelling the same null-or-merge out a second time. Collecting who demands what stays in the inventory tree, refuses two roles demanding different values for one provider in the same round, and skips a role's own entity entry so the existing services.<own>.users declarations are not read as overrides.

The CI set-cover learns the same fact. Its greedy cover retires a role's variant-0 row when an earlier row of another role embeds it, on the stated premise that embedding pulls the provider's variant 0. A variant that dictates that provider's config breaks the premise from the other side: whatever ran, it was not the provider as the provider declares it. Such a pair now covers nothing, so the retired row keeps its job, its artefact and its verdict.

Measured on the current tree: 20 override pairs, 41 covered rows before and after, so nothing is revoked today. The rule bites the moment an override lands in a row that covers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-01 21:08:08 +02:00
kevinveenbirkenbachandClaude Opus 5 48845db979 refactor(profile): split runtime capabilities from runtime identity
One boolean answered five unrelated questions: whether the pull-through cache stack runs, whether image pulls take the GHCR mirror, whether the docker data root may be wiped, plus runtime naming and inventory resolution. The workspace is the environment where those answers diverge — it simulates a developer machine on a real runner — so getting one of them meant misreporting where the suite runs and silently paying for the other four.

is_ci() and runs_on_github() stay facts about the machine. cache_stack_enabled, image_mirror_enabled and docker_root_ephemeral become decisions that default off those facts and can be declared explicitly, so an environment states what it wants. Every default reproduces the previous condition exactly, including the distinction the old mirror and wipe gates drew between a real runner and act.

73ae40585 got the cache stack by flipping the identity flag and lost the docker-root wipe with it; the workspace now declares all three capabilities instead. Pairs with 4c068d1cf, which makes the resulting stray bind sources harmless rather than fatal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-23 19:52:03 +02:00
kevinveenbirkenbachandClaude Opus 5 5137465c6f style: delete every comment the comments-valid policy rejects
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.

Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-18 03:22:02 +02:00
kevinveenbirkenbachandClaude Opus 4.8 f4fa8adf16 feat(deploy): prune orphaned transitive deps on disable
disable= now drops providers left with no surviving parent, not only
the named provider roles. Compose (init) and swarm (derive_includes)
share one reachability cut: BFS from the surviving primaries over
dependency + service edges with disabled roles as cut-nodes. No-disable
deploys stay byte-identical.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 20:05:59 +02:00
kevinveenbirkenbachandClaude Opus 4.8 d9a53a0545 chore(lint): pin ruff to 0.16.0 and resolve the rules that version enforces
The ruff toolchain was unpinned, so host venvs, the lint container and CI
installed different releases over time. That drift is why lint-python only
failed on CI: locally ruff 0.15.x preview-gated ISC004/FURB192 and (falsely)
fired S310 on urllib.request.Request, while CI's 0.16.0 has ISC004/FURB192
stable and drops the S310-on-Request false positive.

Pin ruff==0.16.0 in pyproject.toml's dev extras (Dependabot keeps it current)
and have utils.install.lint.ruff read that pin and reinstall whenever the
found version drifts, so every environment resolves the same ruff. Then fix
what 0.16.0 flags: parenthesise the implicit string concatenations (ISC004),
use min() over sorted()[0] (FURB192), drop the now-unused S310 noqa on the
Request() builders (RUF100), and delete the narration comments that the
comment lint surfaces in the touched files.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 23:08:26 +02:00
kevinveenbirkenbachandClaude Opus 4.8 2f9a54cea7 fix(deploy): honor variants in include resolution
Inventory include sets ignored the active variant: swarm
derive_includes walked the variant-free base config (dynamic service
flags count as enabled) and the compose CombinedResolver followed
run_after as an include edge. Providers a variant pins to false
(e.g. mobilizon variant 1 sso) were still deployed.

- derive_includes: swap every app listed in INFINITO_APP_VARIANTS to
  its variant config, index 0 included (nextcloud variant 0 disables
  11 partner services)
- CombinedResolver: follow_run_after flag; include resolution passes
  False so run_after stays a pure ordering hint
- drop dead deps.py chain (resolve_deploy_ids_for_*) and unused
  resolve_with_edges; rewiring them would resurrect variant-blind
  run_after-following includes
- inventory.md: document the real include pipeline and
  is_explicit_truth semantics

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-04 03:21:56 +02:00
kevinveenbirkenbachandClaude Opus 4.8 35c2ff5f26 feat(deploy): runner-split variant bundles + unified variant selector
Thread a single `variant` selector (CSV) end-to-end so CI can split a
role's matrix deploy across runners.

- variant_select.py (SPOT, shared by init + deploy): one --variant taking
  a CSV (2 pins one round, 0,1,2 is a runner bundle, empty = full matrix);
  replaces the old separate --variant/--variants.
- planner.filter_plan_to_variants filters the plan to a round subset.
- CI discovery (utils/github/variant_bundles.py) splits roles with more
  than INFINITO_VARIANT_BUNDLE_SIZE (default 3) variants into bundles of
  consecutive indices, one matrix runner each (5 -> 0,1,2 + 3,4), threaded
  via the variant env. test-deploy-{local,universal,workstation,server}.yml
  switch to matrix.include with a comma-free variant_slug for artifact/job
  names (GitHub expressions have no replace()).
- runtime_budget honours INFINITO_VARIANT_TIME_OVERFLOW (cut|fail) when a
  runner cannot finish its slice within INFINITO_MAX_RUNTIME.

Unit tests + docs (variants.md, deploy.md) included.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-21 14:52:50 +02:00
kevinveenbirkenbachandClaude Opus 4.7 8e293bd921 fix(compose-deploy): stop persisting runtime routing knobs into .env
Root cause of step-08 deploy failures across all 4 test-development
distros in run 26572559126: `make compose-deploy mode=reinstall
apps=web-app-dashboard INFINITO_SERVICES_DISABLED=matomo` (step 07)
made make export the cmdline var into the recipe's shell env, which
reached `runner.py` → `make setup` → `make dotenv`. The dotenv
generator's `setdefault` then baked the runtime-only value into the
on-disk `.env`. Step 08 sourced that .env, filtered `web-app-matomo`
out of its primary apps, and exited with "All primary apps disabled".

Fix:

- Drop the 7 routing knobs (apps, mode, purge, bundles, disable,
  full_cycle, variant) from `passthrough.STATIC_KEYS` and from
  `default.env`. They are per-invocation; .env is the persistent SPOT
  and must not capture them.
- Switch every consumer from `os.environ["KEY"]` strict access to
  `os.environ.get("KEY", "")` so the missing-key path stops at "no
  filter" instead of a KeyError.
- Rename the env-var surface to match the make-alias surface 1:1
  (lowercase): INFINITO_APPS → apps, INFINITO_SERVICES_DISABLED →
  disable, INFINITO_BUNDLES → bundles, INFINITO_FULL_CYCLE →
  full_cycle, INFINITO_VARIANT → variant, INFINITO_DEPLOY_MODE → mode,
  INFINITO_PURGE_ENTITIES → purge. The make alias was the user-facing
  name in every doc; routing-knob env vars now use the same name so
  the cmdline form, the recipe env-prefix, and the consumer reads
  all line up.
- Collapse the Makefile `compose-deploy` recipe to just the two
  knobs whose alias differs from the env name (`type` →
  INFINITO_DEPLOY_TYPE, `debug` → INFINITO_DEBUG). GNU Make
  auto-exports cmdline variables into the recipe's shell env, so the
  identical-name `$(if $(apps),apps="$(apps)")` lines were pure
  noise.
- Drop `disable: ${disable}` from compose.yml. The dev CLI builds
  `extra_env` per `docker exec` call and forwards `disable` via
  `-e`, so the container env never needs the value frozen at
  compose-up time.
- Update the 3 actual leak call sites
  (07/08/09_deploy_*.sh) to the make-alias form
  (`disable=matomo`, `full_cycle=true`, `variant=1`).
- Add `SIM112` to ruff's ignore set: the lowercase env names are
  the deliberate make/env parity, not a typo.

Repro before fix: `INFINITO_SERVICES_DISABLED=matomo make dotenv`
wrote `INFINITO_SERVICES_DISABLED=matomo` to `.env`. After fix:
`disable=matomo make dotenv` leaves `.env` untouched.

Tests: `make test` and `make autoformat` green (2711 tests, 8 lint
sub-targets).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-28 17:47:03 +02:00
kevinveenbirkenbachandClaude Opus 4.7 10cb02461e fix(inventory): bake only variants.yml overrides into matrix-deploy --vars
Since 576fad84f (2026-05-10) moved role images from
`defaults/main.yml` into `meta/services.yml`, matrix-deploy's variant
baker has been writing the FULL effective config (services.yml +
variants.yml deep-merged via `get_variants`) into host_vars through
the `--vars` JSON consumed by `apply_vars_overrides`. That populates
`applications.{role}.services.{svc}.image` (and `.version`, .cpus,
etc.) with the upstream value, which then silently disables
`apply_mirror_overrides` for every role with an explicit image
declaration: the default `if_missing` policy short-circuits when the
field is already non-blank, so the rendered compose / Dockerfile FROM
keeps pulling from the upstream registry rather than the GHCR mirror.

The regression went unnoticed because Docker Hub usually serves the
pull, but it eventually surfaced as a BuildKit 429 on
`docker.io/library/php:8.2-apache` during the suitecrm Dockerfile
build (job 77602110531 on run 26362676803). Nextcloud, moodle,
mariadb consumers, postgres consumers, simpleicons, phpmyadmin,
phpldapadmin, jenkins, baserow, fider, ... all share the same latent
defect — only suitecrm happened to hit the rate limit first.

Fix the root cause instead of papering over per-role with
`mirror_policy: force`: introduce `get_variant_overrides_only` next
to `get_variants` and switch payload.py to it. The new function
returns the raw `meta/variants.yml` entries per role — no merge with
`meta/services.yml`. matrix-deploy now bakes only the variant-
specific overrides into host_vars; the role's services.yml fields
flow through the normal Ansible lookup chain at deploy time, leaving
`image`/`version` blank in host_vars so `apply_mirror_overrides`
populates them from `mirrors.yml` under its default `if_missing`
policy.

Behavior preserved for the other `get_variants` consumers:
* utils/manager/inventory.py:104 (schema-driven credentials) keeps
  the full merged variant payload.
* planner.py:58 (variant-count only) is unaffected.
* legacy_resolver.py:42 (CombinedResolver runtime services_overrides)
  keeps the full merged view.

Coverage:
* 3 new tests in test_applications.py for the new helper (single
  empty variant when no meta/variants.yml, raw override without base
  merge, cache invariance).
* test_payload.py + test_builder.py mocks repointed from `get_variants`
  to `get_variant_overrides_only`.
* `make test` green across all 5 targets (lint, unit, integration,
  external, test-lint) — 188 tests total.

Side-effects assessed:
* Repo-wide grep confirms no template / task accesses
  `applications.X.services.Y.image` directly via hostvars — every
  consumer goes through `lookup('config', …)` which still deep-merges.
* Roles previously pulling upstream now resolve via the GHCR mirror
  whenever the image is already synced (the `images-mirror-missing.yml`
  workflow keeps the destination populated).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 17:33:33 +02:00
kevinveenbirkenbachandClaude Opus 4.7 a10555cd08 feat(inventory): variant-aware credentials generation
InventoryManager.resolve_schema_includes_recursive previously read the
root role's base meta/services.yml when discovering shared providers,
regardless of which matrix variant the planner had pinned for the
round. For a role whose variants.yml enables additional services per
variant (e.g. svc-bkp-container-2-local variant 2 enabling
services.ldap), the base config had no `enabled+shared` flag for those
services, so the recursive resolver never visited the variant's
shared providers — their credentials were never generated, and the
deploy crashed in the provider's env.j2 with "credentials is defined
in schema at roles/<provider>/meta/schema.yml but not set in
application '<provider>'".

Wires the active variant through the existing inventory pipeline:

- utils/manager/inventory.py: InventoryManager(variant=None|int). When
  set, load_role_config_by_path overlays variants.yml[N] onto the root
  role's meta config (only for the root — transitive providers keep
  their own base lookup). variant=None preserves the pre-refactor base
  behavior for every standalone caller.
- cli/administration/inventory/credentials: new --variant N flag,
  forwarded to InventoryManager.
- cli/administration/inventory/provision: new --app-variants <JSON>
  flag ({app_id: variant_index}), forwarded per-app to the credentials
  subprocess.
- cli/administration/deploy/development/inventory/builder.py: emits
  --app-variants from spec.variant_selectors() when the round pins any
  variant. Apps without variants.yml collapse to a single empty
  overlay in get_variants so this is a no-op for them.

10 unit tests cover: variant overlay for root role, variant=None
fallback to base, out-of-range fallback, no overlay on non-root
paths, --variant CLI forwarding (set + default), --app-variants
generator forwarding (set + omit), builder --app-variants emission
(set + omit).

Caveat: bkp-c2l CI variant 1 still loads svc-db-openldap transitively
at runtime (outside the planner's variant closure) — a separate
sys-service-loader / role-meta bug, not addressed here.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-24 13:04:20 +02:00
kevinveenbirkenbachandClaude Opus 4.7 9a0620f7bb fix(inventory): drop provision-CLI args that shadowed group_vars defaults
Root cause of the DinD DNS-check failure on test-deploy-server
(networks.internet.ip4 falling through to 127.0.0.1 despite
INFINITO_IP4=172.30.0.10 reaching every link of the env pipeline):

  cli/administration/inventory/provision/cli.py declared --ip4 with a
  hardcoded default of "127.0.0.1" and ensure_host_vars_file wrote that
  value unconditionally into host_vars/<host>.yml. group_vars'
  defaults_networks.internet.ip4 resolves "172.30.0.10" via
  \`lookup('env', 'INFINITO_IP4')\`, but tasks/stages/01_constructor.yml
  merges \`defaults_networks | combine(networks | default({}, true))\` so
  host_vars wins. INFINITO_IP4 looked broken end-to-end; the breakage
  actually lived in the provision-CLI host_vars overlay.

  INFINITO_DNS_IP and INFINITO_DOMAIN propagated correctly because
  the provision CLI had no --dns and the --primary-domain write was
  None-gated and never invoked.

While here, --ssl-disabled and --primary-domain are removed too:
neither had any effect in the dev flow. provision/cli.py applies
\`--vars-file inventories/development/default.yml\` AFTER
ensure_host_vars_file via apply_vars_overrides_from_file, whose
_deep_update_commented_map unconditionally overwrites
(\`target[key] = value\`). The dev vars-file always set
\`TLS_ENABLED: true\`, so \`--ssl-disabled\` (which forced
\`TLS_ENABLED: false\` in host_vars) was systematically reverted to
true. Dev/CI ran with TLS enabled the whole time. Inventory stays at
\`TLS_ENABLED: true\` to preserve that effective behaviour.

Removed from cli/administration/inventory/provision/cli.py:
- --ip4, --ip6: defaulted to 127.0.0.1 / ::1, shadowed group_vars
- --primary-domain: never set by any caller, None-gated write
- --ssl-disabled: always re-overwritten by --vars-file later in the
  same provision pass

Removed from cli/administration/inventory/provision/host_vars.py the
matching writes in ensure_host_vars_file; the function now only sets
\`ansible_connection: local\` for localhost-class hosts. Defaults for
DOMAIN_PRIMARY / TLS_ENABLED / networks.internet.{ip4,ip6} live in
group_vars (env-driven) and the inventory vars-file (deployment-wide
flags); host_vars stops duplicating them.

Callers cleaned up:
- cli/administration/deploy/development/inventory/builder.py
- cli/administration/deploy/container/command.py
- scripts/tests/deploy/local/utils/fresh-kept-all-init-and-deploy.sh
- scripts/tests/deploy/local/reset/inventory.sh

Tests + docs follow:
- tests/unit/cli/administration/inventory/provision/test_host_vars.py
  now asserts host_vars no longer carries baked DOMAIN_PRIMARY /
  TLS_ENABLED / networks defaults
- docs/administration/deploy.md drops --ssl-disabled from the
  provision example
- docs/contributing/design/inventory.md re-scopes the host_vars
  column: the DOMAIN_PRIMARY / TLS_ENABLED / networks.internet.{ip4,ip6}
  values arrive via --vars / --vars-file overlays, otherwise group_vars
  provides the env-driven default

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 17:05:41 +02:00
kevinveenbirkenbachandClaude Opus 4.7 a77a570409 refactor(env): consolidate INFINITO_* namespace and centralize path/image SPOTs
Closes a stack of env-pipeline regressions left over from the recent
INFINITO_* namespace refactor by:

* DOMAIN_PRIMARY now reads INFINITO_DOMAIN strictly (was: legacy DOMAIN
  env var → defaulted to infinito.localhost, which never resolved
  because CoreDNS serves the INFINITO_DOMAIN value, infinito.example).
* group_vars/all/08_networks.yml IPs (ip4/ip6/dns) now read
  INFINITO_IP4 / INFINITO_IP6 / INFINITO_DNS_IP.
* PLAYWRIGHT_KEEP_ALL renamed to INFINITO_PLAYWRIGHT_KEEP end-to-end
  (env-builder, role default, run_one.yml docker exec, playwright.config.js,
  lint allowlist, docs anchor, CI workflow vars).
* PKGMGR_IMAGE renamed to INFINITO_PARENT_IMAGE; Dockerfile ARG drops
  the inline default so env/default.env is the only SPOT.
* INFINITO_SRC_DIR centralized: Dockerfile ARG (no default) + ENV;
  compose forwards it as build-arg, volume mount path, and container env;
  every Python CLI + shell helper + CI workflow now reads it strictly
  (os.environ[...] / "${INFINITO_SRC_DIR:?...}") instead of the literal
  /opt/src/infinito.
* New entries in env/default.env: INFINITO_IP6=::1, INFINITO_PARENT_IMAGE,
  INFINITO_PLAYWRIGHT_KEEP=false. All three propagated through
  utils/env/handlers/passthrough.STATIC_KEYS and (where consumed in the
  container) through compose.yml's environment block.

Validated end-to-end with `make test` (2360 tests across lint, lint-*,
test-lint, test-external, test-unit, test-integration).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-21 02:30:34 +02:00
kevinveenbirkenbachandClaude Opus 4.7 9a578952c3 feat(env): SPOT env-pipeline + INFINITO_* namespace
Replace 8 bash env modules with a single Python generator
(cli.meta.env) that materialises env/static.env + runtime context
into a gitignored .env. Each variable lives in its own handler
under utils/env/handlers/; defaults carry inline one-line `# ...`
comments.

Rename to the INFINITO_* schema:
- workflow inputs: APPS, SERVICES_DISABLED, CMD, DEBUG, WHITELIST,
  BUNDLES, FULL_CYCLE, VARIANT, RUN_FLAGS, LIMIT_HOST, PARALLEL
- inventory: INVENTORY_DIR / INVENTORY_FILE / HOST_VARS_FILE,
  PW_FILE -> INVENTORY_PASSWORD_FILE
- venv: VENV_BASE / VENV_FALLBACK / VENV_NAME
- new SPOT defaults: COMPILE_SILENCE, CONTAINER_NO_CA,
  WAIT_HEALTH_TIMEOUT_S, PACKAGE_CACHE_FRONTEND_INIT_IMAGE,
  promoted IMAGE_TAG from GHA-only to general passthrough

compose.yml hardening:
- inline \${VAR:-default} removed; strict \${VAR:?...} everywhere
- new integration lint blocks regressions

Tooling:
- scripts/install/all.sh owns the stamp-file install logic
  (Makefile target delegates)
- scripts/meta/env/load.sh is the single shell entry point;
  8 bash modules deleted
- new shell lint: no INFINITO_* defaults outside env/static.env

Tests:
- 93 new unit tests for utils/env/{builder,parser,runtime,writer}
- restructure tests/lint into repository/ (file-content rules) vs
  filesystem/{shell,python,env}/ (filesystem-shape rules)
- drift + smoke integration tests against the generator

Docs:
- variables.md is now a pointer; per-variable docs live next to
  the key in env/static.env / env/ci.env
- compose.yml.md drops the Default column

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 20:40:34 +02:00
kevinveenbirkenbachandClaude Opus 4.7 bd1a3a69ad fix(deploy/development/inventory/planner): split per-round include from union purge_set
`3c1d5d821` solved the cross-round backbone-abriss problem by making
`include` equal the union of every variant's closure for every round.
Side-effect: every matrix round then ALSO deployed the full union, so
`08_deploy_performance.sh` saw `dashboard.infinito.example` still
answering HTTP 200 after a matomo-only `deploy-fresh-purged-apps` —
the deploy step was no longer matomo-only at all.

Split the two concerns instead:

- `include` (PlanEntry[3]) goes back to the per-round variant-closure.
  Both `init`/`builder` (`--include` for the inventory bake) and
  `deploy` (`--id` for ansible) consume it. Each round therefore
  deploys ONLY its own variant-closure again — the pre-`3c1d5d821`
  semantics.
- `purge_set` (PlanEntry[4], new) carries the union across all rounds.
  `deploy.py` uses it for the inter-round wipe instead of the previous
  round's include. Result: between rounds the WHOLE matrix footprint
  is purged, not just the previous round's variant slice — a variant
  pinning `services.<X>.shared: false` cannot leave its bundled
  provider on the host for a sibling round that wires the same service
  shared. The backbone-abriss `3c1d5d821` was hunting stays fixed:
  whichever round needs the backbone re-deploys it immediately after
  the wipe (purge runs at the start of each round, not the end).

`previous_round_include` tracking is gone. The first iteration skips
the purge (nothing to wipe yet), which also covers `total_rounds == 1`
and `--variant N` pinned single-folder modes cleanly.

Unrelated, but staged together because it is part of the same
investigation:

- `roles/dev-nix/templates/nix.conf.j2`: enable `keep-derivations` /
  `keep-outputs`, so `nix-collect-garbage -d` (run by the aggressive
  prune step) does not drop the build closure of pkgmgr's flake; this
  was making every deploy re-fetch the build inputs from
  cache.nixos.org and overshoot pkgmgr's update timeout.
- `roles/pkgmgr/tasks/01_core.yml`: raise the pkgmgr update timeout
  from 600s to 1800s and lower the retry count from 6 to 3, so a slow
  fetch on a single try no longer pushes total wallclock past three
  hours.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-15 00:44:47 +02:00
kevinveenbirkenbach 3c1d5d8214 fix(deploy/development/inventory/planner): include union of all variant closures per round
A primary app's variant MAY pin `services.<X>.shared: false` to test
the bundled-provider path. That removes <X>'s shared provider from
that variant's closure; the inter-round purge then tears down a
backbone the next variant still expects, leaving consumers in the
following round on an empty dashboard (Index of /), a missing
postgres (5432 Connection refused), or a torn-down mailu.

Pre-compute every variant's closure once, then union them and use the
union as the include for every round. Variants keep changing HOW each
consumer is wired (baked host_vars per round); the deployed app set
stays the superset so purge → redeploy stays round-independent.
2026-05-14 17:12:56 +02:00
kevinveenbirkenbachandClaude Opus 4.7 a2b5269244 refactor(inventory): drop unused variant-only matrix mode
Commit 16f323319 added a strict variant-only include rule + conflict
detector in `inventory/variants.py`, gated behind the presence of any
primary's `meta/variants.yml`. In practice the rule never paid off:

* The CI failure that motivated it (matomo's matrix transition leaving
  a stale dashboard nginx vhost -> 502) is already addressed by the
  nginx-purge primitive shipped in commit bfd04e4a1.
* The existing `CombinedResolver` path already implements the "any
  role says true or `in group_names` -> pull" semantics via
  `utils.roles.applications.services.registry.is_explicit_truth`, so
  every needed service still gets pulled and services that all roles
  mark False stay out of the include.
* The strict literal-equality conflict detector was a false-positive
  factory: 79/80 variant-equipped primaries failed it against the
  current repo state because roles legitimately disagree on `enabled`
  flags for shared services.

This commit removes the entire variant-only branch and restores the
single-path planner. Round-index clamping for non-primary deps is left
exactly as before 16f323319 -- only the additions are reverted.

The file-split refactor from 16f323319 is intentionally kept; only the
`variants.py` submodule, the corresponding `test_variants.py`, and the
variant-only gate test class in `test_planner.py` are removed. The
`__init__.py` re-exports + `__all__` are pruned to match.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 04:25:07 +02:00
kevinveenbirkenbachandClaude Opus 4.7 f102c229f0 refactor(inventory): split monolith into SRP submodules + variant-only mode
Two changes ship together because the variant-only matrix-deploy logic
was written into the legacy 697-line inventory.py and then re-split:

* Source package (max 240 lines per submodule, well under the 300 budget):
  - spec.py            DevInventorySpec + PlanEntry data types
  - payload.py         _resolve_variant_payloads + _bake_overrides
  - legacy_resolver.py CombinedResolver fallback (default-mode path)
  - variants.py        strict variant-only include resolution
  - planner.py         plan_dev_inventory_matrix + filter_plan_to_variant
  - builder.py         build_dev_inventory + build_dev_inventory_matrix
  - __init__.py        public-API re-exports + back-compat shims

* Test package mirrors the source 1:1 with a shared _fixtures.make_spec
  to avoid duplicating the DevInventorySpec defaults across files.

* New variant-only matrix mode (gated on any primary shipping
  meta/variants.yml): only services whose merged variant payload pins
  both enabled and shared to literal True become inventory groups;
  Jinja flags never qualify (group_names does not exist yet at build
  time); run_after only orders the include, never expands it;
  cross-primary disagreement on a (service-key, round) raises ValueError.
  Default mode (no variants.yml on any primary) keeps the legacy
  CombinedResolver path untouched.

* Docs in design/{inventory,variants}.md updated to point at the new
  per-submodule paths; test_nginx_vhosts.py's Test classes now also
  inherit directly from unittest.TestCase so the AST-level lint in
  test_all_test_files_have_tests.py picks them up (transitive base via
  NginxVhostsTestBase alone is not enough).

Coverage: 458 cli unit tests, 349 utils unit tests, 99 dev-deploy tests,
all green; ruff check clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 02:17:26 +02:00