kevinveenbirkenbach/computer-playbook
1
0 Fork 0
mirror of https://github.com/kevinveenbirkenbach/computer-playbook.git synced 2026-11-06 02:54:26 +00:00
Code Issues Packages Projects Releases Wiki Activity
14,682 Commits 9 Branches 78 Tags
fix/dev-python/python-upper-bound
Commit Graph
74 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 4.8 ae7e45dddb feat(deploy): runtime-budget guard stops the variant matrix before INFINITO_MAX_RUNTIME 
Add INFINITO_MAX_RUNTIME (env handler: 6h on real GitHub Actions, 48h otherwise)
and a RuntimeBudget guard in the dev-deploy variant loop. Before each next
variant round the guard projects elapsed + longest round so far + 30m buffer;
if it would exceed the budget it stops and emits a CI warning annotation
(never fails — the rounds that ran stay deployed).

- utils/env/handlers/infinito_max_runtime.py: new handler (canonical SPOT for
  the var; uses ctx.on_gha, materialised into .env by the generator).
- cli/administration/deploy/development/runtime_budget.py: RuntimeBudget +
  duration parser (6h/48h/90m/3600/3600s), extracted to keep deploy.py lean.
- deploy.py: loop uses budget.exhausted()/start_round()/end_round().
- tests for both the handler and RuntimeBudget.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-21 06:03:18 +02:00
kevinveenbirkenbachandClaude Opus 4.7 8e293bd921 fix(compose-deploy): stop persisting runtime routing knobs into .env 
Root cause of step-08 deploy failures across all 4 test-development
distros in run 26572559126: `make compose-deploy mode=reinstall
apps=web-app-dashboard INFINITO_SERVICES_DISABLED=matomo` (step 07)
made make export the cmdline var into the recipe's shell env, which
reached `runner.py` → `make setup` → `make dotenv`. The dotenv
generator's `setdefault` then baked the runtime-only value into the
on-disk `.env`. Step 08 sourced that .env, filtered `web-app-matomo`
out of its primary apps, and exited with "All primary apps disabled".

Fix:

- Drop the 7 routing knobs (apps, mode, purge, bundles, disable,
  full_cycle, variant) from `passthrough.STATIC_KEYS` and from
  `default.env`. They are per-invocation; .env is the persistent SPOT
  and must not capture them.
- Switch every consumer from `os.environ["KEY"]` strict access to
  `os.environ.get("KEY", "")` so the missing-key path stops at "no
  filter" instead of a KeyError.
- Rename the env-var surface to match the make-alias surface 1:1
  (lowercase): INFINITO_APPS → apps, INFINITO_SERVICES_DISABLED →
  disable, INFINITO_BUNDLES → bundles, INFINITO_FULL_CYCLE →
  full_cycle, INFINITO_VARIANT → variant, INFINITO_DEPLOY_MODE → mode,
  INFINITO_PURGE_ENTITIES → purge. The make alias was the user-facing
  name in every doc; routing-knob env vars now use the same name so
  the cmdline form, the recipe env-prefix, and the consumer reads
  all line up.
- Collapse the Makefile `compose-deploy` recipe to just the two
  knobs whose alias differs from the env name (`type` →
  INFINITO_DEPLOY_TYPE, `debug` → INFINITO_DEBUG). GNU Make
  auto-exports cmdline variables into the recipe's shell env, so the
  identical-name `$(if $(apps),apps="$(apps)")` lines were pure
  noise.
- Drop `disable: ${disable}` from compose.yml. The dev CLI builds
  `extra_env` per `docker exec` call and forwards `disable` via
  `-e`, so the container env never needs the value frozen at
  compose-up time.
- Update the 3 actual leak call sites
  (07/08/09_deploy_*.sh) to the make-alias form
  (`disable=matomo`, `full_cycle=true`, `variant=1`).
- Add `SIM112` to ruff's ignore set: the lowercase env names are
  the deliberate make/env parity, not a typo.

Repro before fix: `INFINITO_SERVICES_DISABLED=matomo make dotenv`
wrote `INFINITO_SERVICES_DISABLED=matomo` to `.env`. After fix:
`disable=matomo make dotenv` leaves `.env` untouched.

Tests: `make test` and `make autoformat` green (2711 tests, 8 lint
sub-targets).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-28 17:47:03 +02:00
kevinveenbirkenbachandClaude Opus 4.7 10cb02461e fix(inventory): bake only variants.yml overrides into matrix-deploy --vars 
Since 576fad84f (2026-05-10) moved role images from
`defaults/main.yml` into `meta/services.yml`, matrix-deploy's variant
baker has been writing the FULL effective config (services.yml +
variants.yml deep-merged via `get_variants`) into host_vars through
the `--vars` JSON consumed by `apply_vars_overrides`. That populates
`applications.{role}.services.{svc}.image` (and `.version`, .cpus,
etc.) with the upstream value, which then silently disables
`apply_mirror_overrides` for every role with an explicit image
declaration: the default `if_missing` policy short-circuits when the
field is already non-blank, so the rendered compose / Dockerfile FROM
keeps pulling from the upstream registry rather than the GHCR mirror.

The regression went unnoticed because Docker Hub usually serves the
pull, but it eventually surfaced as a BuildKit 429 on
`docker.io/library/php:8.2-apache` during the suitecrm Dockerfile
build (job 77602110531 on run 26362676803). Nextcloud, moodle,
mariadb consumers, postgres consumers, simpleicons, phpmyadmin,
phpldapadmin, jenkins, baserow, fider, ... all share the same latent
defect — only suitecrm happened to hit the rate limit first.

Fix the root cause instead of papering over per-role with
`mirror_policy: force`: introduce `get_variant_overrides_only` next
to `get_variants` and switch payload.py to it. The new function
returns the raw `meta/variants.yml` entries per role — no merge with
`meta/services.yml`. matrix-deploy now bakes only the variant-
specific overrides into host_vars; the role's services.yml fields
flow through the normal Ansible lookup chain at deploy time, leaving
`image`/`version` blank in host_vars so `apply_mirror_overrides`
populates them from `mirrors.yml` under its default `if_missing`
policy.

Behavior preserved for the other `get_variants` consumers:
* utils/manager/inventory.py:104 (schema-driven credentials) keeps
  the full merged variant payload.
* planner.py:58 (variant-count only) is unaffected.
* legacy_resolver.py:42 (CombinedResolver runtime services_overrides)
  keeps the full merged view.

Coverage:
* 3 new tests in test_applications.py for the new helper (single
  empty variant when no meta/variants.yml, raw override without base
  merge, cache invariance).
* test_payload.py + test_builder.py mocks repointed from `get_variants`
  to `get_variant_overrides_only`.
* `make test` green across all 5 targets (lint, unit, integration,
  external, test-lint) — 188 tests total.

Side-effects assessed:
* Repo-wide grep confirms no template / task accesses
  `applications.X.services.Y.image` directly via hostvars — every
  consumer goes through `lookup('config', …)` which still deep-merges.
* Roles previously pulling upstream now resolve via the GHCR mirror
  whenever the image is already synced (the `images-mirror-missing.yml`
  workflow keeps the destination populated).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-25 17:33:33 +02:00
kevinveenbirkenbachandClaude Opus 4.7 a10555cd08 feat(inventory): variant-aware credentials generation 
InventoryManager.resolve_schema_includes_recursive previously read the
root role's base meta/services.yml when discovering shared providers,
regardless of which matrix variant the planner had pinned for the
round. For a role whose variants.yml enables additional services per
variant (e.g. svc-bkp-container-2-local variant 2 enabling
services.ldap), the base config had no `enabled+shared` flag for those
services, so the recursive resolver never visited the variant's
shared providers — their credentials were never generated, and the
deploy crashed in the provider's env.j2 with "credentials is defined
in schema at roles/<provider>/meta/schema.yml but not set in
application '<provider>'".

Wires the active variant through the existing inventory pipeline:

- utils/manager/inventory.py: InventoryManager(variant=None|int). When
  set, load_role_config_by_path overlays variants.yml[N] onto the root
  role's meta config (only for the root — transitive providers keep
  their own base lookup). variant=None preserves the pre-refactor base
  behavior for every standalone caller.
- cli/administration/inventory/credentials: new --variant N flag,
  forwarded to InventoryManager.
- cli/administration/inventory/provision: new --app-variants <JSON>
  flag ({app_id: variant_index}), forwarded per-app to the credentials
  subprocess.
- cli/administration/deploy/development/inventory/builder.py: emits
  --app-variants from spec.variant_selectors() when the round pins any
  variant. Apps without variants.yml collapse to a single empty
  overlay in get_variants so this is a no-op for them.

10 unit tests cover: variant overlay for root role, variant=None
fallback to base, out-of-range fallback, no overlay on non-root
paths, --variant CLI forwarding (set + default), --app-variants
generator forwarding (set + omit), builder --app-variants emission
(set + omit).

Caveat: bkp-c2l CI variant 1 still loads svc-db-openldap transitively
at runtime (outside the planner's variant closure) — a separate
sys-service-loader / role-meta bug, not addressed here.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-24 13:04:20 +02:00
kevinveenbirkenbachandClaude Opus 4.7 8235907ee9 refactor(cli/deploy): move required_by verifier into dedicated runner 
Hooks the verifier into `cli.administration.deploy.dedicated.runner.
run_ansible_playbook` so every deploy path that reaches the dedicated
runner is checked — including `compose-deploy mode=update`, bundle
redeploys, and direct production calls, which previously bypassed the
dev-wrapper's verifier.

Runner pins ANSIBLE_LOG_PATH, captures pre-run log size, then verifies
the post-run slice host-locally (no more `docker exec tail` indirection
from the dev wrapper). Dev wrapper drops its redundant verifier call.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-23 04:23:45 +02:00
kevinveenbirkenbachandClaude Opus 4.7 8bf4a5f08e feat(cli/meta): add cli.meta.roles.services.called verifier + dev-deploy hook 
Asserts at deploy time that every role declaring `required_by` in its
meta/services.yml actually executed (at least one TASK block ending with
a non-skip status of `ok` / `changed` / `fatal` / `included`). Catches
silent coverage regressions where a once-flag swallows an include before
the role's body can run.

CLI: `python -m cli.meta.roles.services.called --logfile X --apps a,b,c`
Library: `cli.meta.roles.services.called.verify(...)` supports per-pass
slicing via `log_byte_offset` and reads either host-side log files or
in-container logs when `container=` is given.

Hooked into `cli.administration.deploy.development.deploy._run_deploy`
so matrix-deploy runs (CI + local make compose-deploy mode=initialize/
reinstall) snapshot the log offset before each compose.exec and verify
the resulting slice after rc=0. Defaults ANSIBLE_LOG_PATH to
/tmp/infinito-deploy.log so the check also runs locally without env-var
setup.

Includes 35 self-contained unit tests covering category derivation,
TASK-block status parsing (incl. ANSI escape stripping + substring
role-id guard + multi-skip edge case), required_role_ids YAML scan,
host log slicing with byte offsets, end-to-end verify, and CLI argparse.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-23 04:23:45 +02:00
kevinveenbirkenbachandClaude Opus 4.7 b5c3f33b56 fix(ci): surface in-container ansible log on the runner for role-runtime summary 
The CI `📊 Top role runtimes` step parses `$ANSIBLE_LOG_PATH` on the
runner, but ansible-playbook actually runs inside the dev container via
`docker compose exec` — and the env var was never forwarded there, so
the log file never materialised. Even when set, the log would have
lived inside the container, invisible to the runner-side parser.

Forwards `ANSIBLE_LOG_PATH` through `compose.exec`'s `extra_env` so
ansible-playbook writes the log at the configured path inside the
container, then `docker cp`s it back to the runner from the cleanup
trap before the stack is torn down.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 02:53:05 +02:00
kevinveenbirkenbachandClaude Opus 4.7 9a0620f7bb fix(inventory): drop provision-CLI args that shadowed group_vars defaults 
Root cause of the DinD DNS-check failure on test-deploy-server
(networks.internet.ip4 falling through to 127.0.0.1 despite
INFINITO_IP4=172.30.0.10 reaching every link of the env pipeline):

  cli/administration/inventory/provision/cli.py declared --ip4 with a
  hardcoded default of "127.0.0.1" and ensure_host_vars_file wrote that
  value unconditionally into host_vars/<host>.yml. group_vars'
  defaults_networks.internet.ip4 resolves "172.30.0.10" via
  \`lookup('env', 'INFINITO_IP4')\`, but tasks/stages/01_constructor.yml
  merges \`defaults_networks | combine(networks | default({}, true))\` so
  host_vars wins. INFINITO_IP4 looked broken end-to-end; the breakage
  actually lived in the provision-CLI host_vars overlay.

  INFINITO_DNS_IP and INFINITO_DOMAIN propagated correctly because
  the provision CLI had no --dns and the --primary-domain write was
  None-gated and never invoked.

While here, --ssl-disabled and --primary-domain are removed too:
neither had any effect in the dev flow. provision/cli.py applies
\`--vars-file inventories/development/default.yml\` AFTER
ensure_host_vars_file via apply_vars_overrides_from_file, whose
_deep_update_commented_map unconditionally overwrites
(\`target[key] = value\`). The dev vars-file always set
\`TLS_ENABLED: true\`, so \`--ssl-disabled\` (which forced
\`TLS_ENABLED: false\` in host_vars) was systematically reverted to
true. Dev/CI ran with TLS enabled the whole time. Inventory stays at
\`TLS_ENABLED: true\` to preserve that effective behaviour.

Removed from cli/administration/inventory/provision/cli.py:
- --ip4, --ip6: defaulted to 127.0.0.1 / ::1, shadowed group_vars
- --primary-domain: never set by any caller, None-gated write
- --ssl-disabled: always re-overwritten by --vars-file later in the
  same provision pass

Removed from cli/administration/inventory/provision/host_vars.py the
matching writes in ensure_host_vars_file; the function now only sets
\`ansible_connection: local\` for localhost-class hosts. Defaults for
DOMAIN_PRIMARY / TLS_ENABLED / networks.internet.{ip4,ip6} live in
group_vars (env-driven) and the inventory vars-file (deployment-wide
flags); host_vars stops duplicating them.

Callers cleaned up:
- cli/administration/deploy/development/inventory/builder.py
- cli/administration/deploy/container/command.py
- scripts/tests/deploy/local/utils/fresh-kept-all-init-and-deploy.sh
- scripts/tests/deploy/local/reset/inventory.sh

Tests + docs follow:
- tests/unit/cli/administration/inventory/provision/test_host_vars.py
  now asserts host_vars no longer carries baked DOMAIN_PRIMARY /
  TLS_ENABLED / networks defaults
- docs/administration/deploy.md drops --ssl-disabled from the
  provision example
- docs/contributing/design/inventory.md re-scopes the host_vars
  column: the DOMAIN_PRIMARY / TLS_ENABLED / networks.internet.{ip4,ip6}
  values arrive via --vars / --vars-file overlays, otherwise group_vars
  provides the env-driven default

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 17:05:41 +02:00
kevinveenbirkenbachandClaude Opus 4.7 13c93ad6d5 refactor(compose): drop the ci profile and its --profile ci arg everywhere 
compose.yml no longer hides services behind \`profiles: ["ci"]\`. The
profile existed solely as a no-op safety net against a bare
\`docker compose up\` in the repo root, but the top-level compose.yml is
already explicitly scoped to local dev / CI testing (production goes
through the Ansible roles under roles/<role>/templates/), so the safety
net is redundant with the documented scope.

Every \`--profile ci\` callsite collapses:
- compose.yml: \`profiles: ["ci"]\` removed from coredns + infinito
- Profile.args() deleted; Profile keeps is_ci() and
  registry_cache_active() for the cache-override gating
- compose.py / down.py: profile.args() splat removed from the
  docker-compose command builder; down.py drops its now-unused
  Profile import
- 5 shell wrappers (install, lint, tests/code, meta/resolve/apps,
  meta/resolve/diff/affected_roles): plain \`docker compose\` instead
  of \`docker compose --profile ci\`
- tests/unit/.../test_profile.py: TestProfileArgs class removed
- tests/unit/.../test_compose.py: two cmd-shape asserts updated to
  reflect the missing --profile pair

Docs:
- docs/contributing/artefact/files/compose.yml.md: the
  \"every service MUST be behind a profiles: entry\" rule is gone
- docs/contributing/actions/debugging/ci.md: the
  \`docker compose ... --profile ci config\` debug snippet drops the
  flag; while touching the file the two em-dash clause separators in
  the reproduction-profiles shell block are rewritten to colons
  (documentation.md em-dash rule), and one list-item in
  compose.yml.md is collapsed to one sentence per line for
  consistency with its siblings

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 16:22:33 +02:00
kevinveenbirkenbachandClaude Opus 4.7 12a56e449d refactor(env): move env/default.env to repo-root default.env and drop env/ 
With env.development and env/ci.env both gone, env/default.env was the
last inhabitant of the directory. Hoisting it to the repo root removes
the empty-shell indirection: the SPOT is now [default.env](default.env),
sitting next to .env (its generated companion) and other repo-root
configuration artefacts like compose.yml, Dockerfile, and Makefile.

Path updates land in:
- generator: cli/meta/env/__main__.py (REPO_ROOT / "default.env")
- workflow: .github/workflows/test-install-make.yml (awk reads from
  default.env, error message updated in lockstep)
- lint tests: test_used_keys_declared.py and test_infinito_prefix.py
  switched from globbing env/*.env to targeting default.env directly;
  test_default_env_keys_handled.py points at the new path
- drift test: test_inventory_vars_file_spot.py whitelist + STATIC_ENV
- smoke test: tests/integration/meta/env/test_dotenv_generator.py
  STATIC_ENV + every docstring/error message

Doc-string and prose references in utils/env/*, scripts/*,
cli/administration/deploy/development/{common,coredns}.py,
Makefile, Dockerfile, and every doc page that mentioned env/default.env
are rewritten in the same pass so future readers see one path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 16:22:33 +02:00
kevinveenbirkenbachandClaude Opus 4.7 f20278b4d5 refactor(env): drop env.development marker and its compose layering 
The env.development marker file existed for two purposes that no longer
justify the wiring cost: a per-machine compose --env-file override layer
appended after auto-loaded .env, and a fallback signal for
detect_runtime() to return "dev" when the marker exists. With every
INFINITO_* default now collapsed into env/default.env -> .env,
persistent per-machine overrides are best expressed as
\`export INFINITO_FOO=…\` before invoking make (load.sh honors
caller-set values via setdefault). The marker fallback in
detect_runtime was redundant with the explicit RUNTIME=dev that every
dev-flow callsite already passes (fresh-*.sh, reset/inventory.sh).

Removed:
- Makefile mark-development target + .PHONY entry
- env.development from .gitignore / .dockerignore
- env_local conditional in compose.up()
- env_development conditional in down._compose_run()
- env.development conditional in scripts/meta/resolve/apps.sh and
  scripts/meta/resolve/diff/affected_roles.sh
- env.development marker branch + the now-unused project_root parameter
  from cli/meta/runtime/detect_runtime; __main__.py caller follows
- "Optional Development Marker" section in docs/contributing/environment/
  setup.md and the marker reference in docs/administration/installation.md

Also drops two RUNTIME=='dev' guards in roles/web-app-bigbluebutton that
relied on the auto-detect path:
- the compose build etherpad task in tasks/02_compose.yml
- the freeswitch iptables-REJECT entrypoint override in
  templates/compose.override.yml.j2

With auto-detection of "dev" gone the guards would otherwise only fire
on explicit RUNTIME=dev runs, silently regressing the local-dev path
operators rely on; both behaviors now apply unconditionally.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 16:22:33 +02:00
kevinveenbirkenbachandClaude Opus 4.7 07a9440ac7 refactor(env): fold ci.env into default.env so INFINITO_IP4/DNS/DOMAIN reach Ansible 
The four ci.env keys (INFINITO_DNS_IP, INFINITO_DOMAIN, INFINITO_DOMAIN_RE,
INFINITO_IP4) were only honored by docker compose --env-file, so processes
that sourced .env via load.sh saw INFINITO_IP4 as unset. As a result
networks.internet.ip4 in the Ansible playbook fell through to the
127.0.0.1 default and the DinD DNS-check handler failed on the deploy
server matrix.

Moving the defaults into env/default.env makes them flow through the same
.env / load.sh / os.environ chain as every other INFINITO_* key, so
\${INFINITO_IP4} in compose.yml, lookup('env', 'INFINITO_IP4') in
group_vars, and envsubst over the CoreDNS Corefile template all see the
same value deterministically.

Side fix: utils/env/parser.py now undoes the writer's \\\\/\\" escaping for
quoted values, closing a round-trip gap that surfaced once
INFINITO_DOMAIN_RE (the only value containing a literal backslash) started
flowing through the generator.

All --env-file env/ci.env call sites (compose.py, down.py, install/lint/
test wrappers, resolve scripts, test-dns workflow + script) are removed.
coredns.py now reads .env via parse_static_env, docs and unit/integration
tests follow.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 14:55:10 +02:00
kevinveenbirkenbachandClaude Opus 4.7 a77a570409 refactor(env): consolidate INFINITO_* namespace and centralize path/image SPOTs 
Closes a stack of env-pipeline regressions left over from the recent
INFINITO_* namespace refactor by:

* DOMAIN_PRIMARY now reads INFINITO_DOMAIN strictly (was: legacy DOMAIN
  env var → defaulted to infinito.localhost, which never resolved
  because CoreDNS serves the INFINITO_DOMAIN value, infinito.example).
* group_vars/all/08_networks.yml IPs (ip4/ip6/dns) now read
  INFINITO_IP4 / INFINITO_IP6 / INFINITO_DNS_IP.
* PLAYWRIGHT_KEEP_ALL renamed to INFINITO_PLAYWRIGHT_KEEP end-to-end
  (env-builder, role default, run_one.yml docker exec, playwright.config.js,
  lint allowlist, docs anchor, CI workflow vars).
* PKGMGR_IMAGE renamed to INFINITO_PARENT_IMAGE; Dockerfile ARG drops
  the inline default so env/default.env is the only SPOT.
* INFINITO_SRC_DIR centralized: Dockerfile ARG (no default) + ENV;
  compose forwards it as build-arg, volume mount path, and container env;
  every Python CLI + shell helper + CI workflow now reads it strictly
  (os.environ[...] / "${INFINITO_SRC_DIR:?...}") instead of the literal
  /opt/src/infinito.
* New entries in env/default.env: INFINITO_IP6=::1, INFINITO_PARENT_IMAGE,
  INFINITO_PLAYWRIGHT_KEEP=false. All three propagated through
  utils/env/handlers/passthrough.STATIC_KEYS and (where consumed in the
  container) through compose.yml's environment block.

Validated end-to-end with `make test` (2360 tests across lint, lint-*,
test-lint, test-external, test-unit, test-integration).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 02:30:34 +02:00
kevinveenbirkenbachandClaude Opus 4.7 81b57f6f50 refactor(env): rename INFINITO_NO_BUILD -> INFINITO_BUILD with inverted truthiness 
Promote the key from GHA-only (gha_passthrough) to always-emitted (passthrough)
so the deploy CLI can read it strictly via os.environ[...]. The previous
INFINITO_NO_BUILD layout left the key unset locally, which crashed
cli/.../up.py and cli/.../compose.py with KeyError after the env-pipeline
refactor that introduced gha_passthrough.

New semantics:
  INFINITO_BUILD=1  build the local image  (default; local mode)
  INFINITO_BUILD=0  skip build, pull image (CI overrides via -e)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 02:30:33 +02:00
kevinveenbirkenbachandClaude Opus 4.7 80c396b839 refactor(env+install/spot): collapse python/venv resolution into shell SPOT 
Pulls VENV/PYTHON/PIP/PYTHONPATH out of the dotenv handler pipeline
into scripts/meta/python/detect.sh. load.sh now sources detect.sh and
prepends ${VENV}/bin to PATH above the INFINITO_ENV_LOADED guard, so
every BASH_ENV re-entry re-resolves PYTHON the moment a venv
materialises mid-chain — the same fix as b79ed2869 but at the SPOT,
which retires the reload-trick from ansible.sh, python.sh and
pre-commit.sh.

Also renames INVENTORY_VARS_FILE → INFINITO_INVENTORY_VARS_FILE under
the new env-prefix lint, drops the 5 dead INFINITO_ENV_*_LOADED unsets
left over from the multi-loader era, and adds a lint guarding the
INFINITO_ prefix invariant on env/*.env keys.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-21 00:39:52 +02:00
kevinveenbirkenbachandClaude Opus 4.7 eb4bcd23a8 refactor(env/spot): forbid .get() defaults + cache namespace rename 
Three new lint tests close gaps in the INFINITO_* SPOT contract:
* tests/lint/filesystem/env/test_default_env_keys_handled.py
  Every default.env key must be wired through a handler.
* tests/lint/filesystem/env/test_used_keys_declared.py
  Every INFINITO_* in .sh/.py must live in env/{default,ci}.env or
  a handler; per-line `# nocheck:` suppresses runtime-only refs.
* tests/lint/filesystem/python/test_no_infinito_get_defaults.py
  Forbid ANY default in `.get("INFINITO_*", ...)` -- including the
  empty-string fallback (which silently masks an unset key). The
  nocheck check walks every physical line the call spans so an
  autoformat-split call still honours the marker.

Cache namespace flattens to a single prefix:
  INFINITO_PACKAGE_CACHE_*  -> INFINITO_CACHE_PACKAGE_*
  INFINITO_REGISTRY_CACHE_* -> INFINITO_CACHE_REGISTRY_*
The five previously driver-injected paths (APT_LIST, NPMRC,
PIP_CONF, REGISTRY_PROXY_CONF, FRONTEND_CA_FILE) move into
env/default.env and passthrough.STATIC_KEYS; the
`cache_env_overrides()` helper and its two call sites are gone
along with the test-fixture override for FRONTEND_CA_DIR.

SPOT-strict rewrites:
  - 19 sites: `something.get("INFINITO_X", "")` -> subscript
    `something["INFINITO_X"]` (loud KeyError when env loader did
    not run; all keys live in env/default.env).
  - 2 handler-internal bootstrap reads suppressed with
    `# nocheck: handler-bootstrap-read`.
  - 30 runtime-only / driver-injected / test-fixture references
    annotated with `# nocheck: <reason>` per-line.
  - Unit tests that previously removed SPOT-guaranteed keys via
    `{k: v ... if k \!= KEY}` or `clear=True` switch to setting
    the key to "" explicitly, matching the new contract.

Side fixes:
  - INFINITO_DNS_DOMAIN + INFINITO_DNS_HOSTS_FILE move from
    shell-side defaults in common.sh to env/default.env and
    passthrough.STATIC_KEYS.
  - dns_systemd_is_operational() drops the dead
    INFINITO_DNS_FORCE_HOSTS_FALLBACK override branch (the
    variable was never set anywhere).
  - INFINITO_WORKER_FETCH switches from `os.environ.get(KEY, "1")`
    to `os.environ[KEY]`.
  - scripts/tests/code/wrapper.sh exec args move into a bash
    array so `# nocheck:` on INFINITO_TEST_TYPE can live on its
    own element line (SC2215 fix).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 22:26:16 +02:00
kevinveenbirkenbachandClaude Opus 4.7 9a578952c3 feat(env): SPOT env-pipeline + INFINITO_* namespace 
Replace 8 bash env modules with a single Python generator
(cli.meta.env) that materialises env/static.env + runtime context
into a gitignored .env. Each variable lives in its own handler
under utils/env/handlers/; defaults carry inline one-line `# ...`
comments.

Rename to the INFINITO_* schema:
- workflow inputs: APPS, SERVICES_DISABLED, CMD, DEBUG, WHITELIST,
  BUNDLES, FULL_CYCLE, VARIANT, RUN_FLAGS, LIMIT_HOST, PARALLEL
- inventory: INVENTORY_DIR / INVENTORY_FILE / HOST_VARS_FILE,
  PW_FILE -> INVENTORY_PASSWORD_FILE
- venv: VENV_BASE / VENV_FALLBACK / VENV_NAME
- new SPOT defaults: COMPILE_SILENCE, CONTAINER_NO_CA,
  WAIT_HEALTH_TIMEOUT_S, PACKAGE_CACHE_FRONTEND_INIT_IMAGE,
  promoted IMAGE_TAG from GHA-only to general passthrough

compose.yml hardening:
- inline \${VAR:-default} removed; strict \${VAR:?...} everywhere
- new integration lint blocks regressions

Tooling:
- scripts/install/all.sh owns the stamp-file install logic
  (Makefile target delegates)
- scripts/meta/env/load.sh is the single shell entry point;
  8 bash modules deleted
- new shell lint: no INFINITO_* defaults outside env/static.env

Tests:
- 93 new unit tests for utils/env/{builder,parser,runtime,writer}
- restructure tests/lint into repository/ (file-content rules) vs
  filesystem/{shell,python,env}/ (filesystem-shape rules)
- drift + smoke integration tests against the generator

Docs:
- variables.md is now a pointer; per-variable docs live next to
  the key in env/static.env / env/ci.env
- compose.yml.md drops the Default column

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-20 20:40:34 +02:00
kevinveenbirkenbachandClaude Opus 4.7 0aa34b903e feat(playwright): persist artifacts per variant+pass, add Time column 
- Matrix-deploy orchestrator now pushes `VARIANT_INDEX` (0-based round
  id) alongside the existing `ASYNC_ENABLED` extra var for both PASS 1
  (sync) and PASS 2 (async) runs. Test-e2e-playwright's archive step
  consumes both to derive a per-(variant, pass) destination so a
  subsequent round cannot overwrite the previous one's reports.
- New `roles/test-e2e-playwright/files/archive_playwright_reports.sh`
  moves the playwright-junit.xml / playwright-report/ / test-results/
  outputs into `variant-<N>/<sync|async>/` under the role's reports
  dir, idempotently (re-running the same tuple replaces rather than
  fails). `run_one.yml` invokes the script via `ansible.builtin.script`
  so the YAML stays a single 7-line task.
- `VARIANT_INDEX: 0` lives in `defaults/main.yml` to keep the variable
  defined for dev / non-matrix invocations. Role defaults are not
  considered SPOT, so the `| default(0)` belt-and-suspenders inside
  the script invocation stays valid against the spot-default lint.
- Summary script now renders an 8-column table: Time | App | Variant |
  Pass | Test | Status | Duration | Message. Per-test ISO timestamps
  are derived from the JUnit `<testsuite timestamp>` attribute plus the
  cumulative duration of preceding cases; file mtime is the fallback
  when the attribute is absent. App / Variant / Pass are extracted
  from the artefact path layout produced by the archive script.
- Matrix-deploy unit tests refreshed to expect the new
  `extra_ansible_vars` shape; summary unit tests gain coverage for
  `_to_iso_z`, `_parse_iso_timestamp`, the four supported path
  layouts, and the file-mtime fallback (55 tests total).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-17 15:30:06 +02:00
kevinveenbirkenbachandClaude Opus 4.7 525699a9ab test(lint/cache-read): extend scope to utils, cli, plugins, library 
Modules imported by tests run inside the same pytest process as the
tests themselves. Raw ``path.read_text(...)`` calls in those modules
bypass the cache the tests just populated. ``roles/`` stays excluded
(deployed scripts have no ``utils/`` on PYTHONPATH); ``utils/cache/``
stays excluded (cache implementation itself).

Five conversions to ``utils.cache.files.read_text``:
- ``utils/handler/yaml.py`` (YAML loader, read-only)
- ``utils/annotations/summarize.py`` (CI log scanner, with
  ``UnicodeDecodeError`` fallback replacing ``errors="replace"``)
- ``cli/meta/roles/applications/resolution/combined/__main__.py``
  (single-shot CLI JSON input)
- ``cli/core/help.py`` (read_folder_title + read_folder_description;
  same README is read many times across CLI help iteration)

Eight ``# nocheck: cache-read`` markers, each annotated inline:
- ``utils/claude/settings_sort.py`` (read then rewrite sorted)
- ``utils/docker/version_updater.py`` (read then rewrite config)
- ``cli/administration/deploy/development/network.py`` (tests mock
  ``Path.read_text`` directly; caching would bypass the mock)
- ``cli/administration/deploy/development/log_utils.py`` (append flow)
- ``cli/contributing/changelog/archive/changelog_md.py``,
  ``__main__.py``, ``package_mirror.py`` (file mutated in same flow)

Two callers refactored to variable extraction so ruff's 88-char wrap
keeps the ``# nocheck`` marker on the ``read_text(...)`` AST span line
(``cli/.../network.py`` and ``package_mirror.py:105``).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-15 14:23:32 +02:00
kevinveenbirkenbachandClaude Opus 4.7 bd1a3a69ad fix(deploy/development/inventory/planner): split per-round include from union purge_set 
`3c1d5d821` solved the cross-round backbone-abriss problem by making
`include` equal the union of every variant's closure for every round.
Side-effect: every matrix round then ALSO deployed the full union, so
`08_deploy_performance.sh` saw `dashboard.infinito.example` still
answering HTTP 200 after a matomo-only `deploy-fresh-purged-apps` —
the deploy step was no longer matomo-only at all.

Split the two concerns instead:

- `include` (PlanEntry[3]) goes back to the per-round variant-closure.
  Both `init`/`builder` (`--include` for the inventory bake) and
  `deploy` (`--id` for ansible) consume it. Each round therefore
  deploys ONLY its own variant-closure again — the pre-`3c1d5d821`
  semantics.
- `purge_set` (PlanEntry[4], new) carries the union across all rounds.
  `deploy.py` uses it for the inter-round wipe instead of the previous
  round's include. Result: between rounds the WHOLE matrix footprint
  is purged, not just the previous round's variant slice — a variant
  pinning `services.<X>.shared: false` cannot leave its bundled
  provider on the host for a sibling round that wires the same service
  shared. The backbone-abriss `3c1d5d821` was hunting stays fixed:
  whichever round needs the backbone re-deploys it immediately after
  the wipe (purge runs at the start of each round, not the end).

`previous_round_include` tracking is gone. The first iteration skips
the purge (nothing to wipe yet), which also covers `total_rounds == 1`
and `--variant N` pinned single-folder modes cleanly.

Unrelated, but staged together because it is part of the same
investigation:

- `roles/dev-nix/templates/nix.conf.j2`: enable `keep-derivations` /
  `keep-outputs`, so `nix-collect-garbage -d` (run by the aggressive
  prune step) does not drop the build closure of pkgmgr's flake; this
  was making every deploy re-fetch the build inputs from
  cache.nixos.org and overshoot pkgmgr's update timeout.
- `roles/pkgmgr/tasks/01_core.yml`: raise the pkgmgr update timeout
  from 600s to 1800s and lower the retry count from 6 to 3, so a slow
  fetch on a single try no longer pushes total wallclock past three
  hours.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-15 00:44:47 +02:00
kevinveenbirkenbach 3c1d5d8214 fix(deploy/development/inventory/planner): include union of all variant closures per round 
A primary app's variant MAY pin `services.<X>.shared: false` to test
the bundled-provider path. That removes <X>'s shared provider from
that variant's closure; the inter-round purge then tears down a
backbone the next variant still expects, leaving consumers in the
following round on an empty dashboard (Index of /), a missing
postgres (5432 Connection refused), or a torn-down mailu.

Pre-compute every variant's closure once, then union them and use the
union as the include for every round. Variants keep changing HOW each
consumer is wired (baked host_vars per round); the deployed app set
stays the superset so purge → redeploy stays round-independent.
 2026-05-14 17:12:56 +02:00
kevinveenbirkenbachandClaude Opus 4.7 a2b5269244 refactor(inventory): drop unused variant-only matrix mode 
Commit 16f323319 added a strict variant-only include rule + conflict
detector in `inventory/variants.py`, gated behind the presence of any
primary's `meta/variants.yml`. In practice the rule never paid off:

* The CI failure that motivated it (matomo's matrix transition leaving
  a stale dashboard nginx vhost -> 502) is already addressed by the
  nginx-purge primitive shipped in commit bfd04e4a1.
* The existing `CombinedResolver` path already implements the "any
  role says true or `in group_names` -> pull" semantics via
  `utils.roles.applications.services.registry.is_explicit_truth`, so
  every needed service still gets pulled and services that all roles
  mark False stay out of the include.
* The strict literal-equality conflict detector was a false-positive
  factory: 79/80 variant-equipped primaries failed it against the
  current repo state because roles legitimately disagree on `enabled`
  flags for shared services.

This commit removes the entire variant-only branch and restores the
single-path planner. Round-index clamping for non-primary deps is left
exactly as before 16f323319 -- only the additions are reverted.

The file-split refactor from 16f323319 is intentionally kept; only the
`variants.py` submodule, the corresponding `test_variants.py`, and the
variant-only gate test class in `test_planner.py` are removed. The
`__init__.py` re-exports + `__all__` are pruned to match.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 04:25:07 +02:00
kevinveenbirkenbachandClaude Opus 4.7 f102c229f0 refactor(inventory): split monolith into SRP submodules + variant-only mode 
Two changes ship together because the variant-only matrix-deploy logic
was written into the legacy 697-line inventory.py and then re-split:

* Source package (max 240 lines per submodule, well under the 300 budget):
  - spec.py            DevInventorySpec + PlanEntry data types
  - payload.py         _resolve_variant_payloads + _bake_overrides
  - legacy_resolver.py CombinedResolver fallback (default-mode path)
  - variants.py        strict variant-only include resolution
  - planner.py         plan_dev_inventory_matrix + filter_plan_to_variant
  - builder.py         build_dev_inventory + build_dev_inventory_matrix
  - __init__.py        public-API re-exports + back-compat shims

* Test package mirrors the source 1:1 with a shared _fixtures.make_spec
  to avoid duplicating the DevInventorySpec defaults across files.

* New variant-only matrix mode (gated on any primary shipping
  meta/variants.yml): only services whose merged variant payload pins
  both enabled and shared to literal True become inventory groups;
  Jinja flags never qualify (group_names does not exist yet at build
  time); run_after only orders the include, never expands it;
  cross-primary disagreement on a (service-key, round) raises ValueError.
  Default mode (no variants.yml on any primary) keeps the legacy
  CombinedResolver path untouched.

* Docs in design/{inventory,variants}.md updated to point at the new
  per-submodule paths; test_nginx_vhosts.py's Test classes now also
  inherit directly from unittest.TestCase so the AST-level lint in
  test_all_test_files_have_tests.py picks them up (transitive base via
  NginxVhostsTestBase alone is not enough).

Coverage: 458 cli unit tests, 349 utils unit tests, 99 dev-deploy tests,
all green; ruff check clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 02:17:26 +02:00
kevinveenbirkenbachandClaude Opus 4.7 a26bfc4e94 refactor(cli): reorg admin/contributing tree + per-folder help system 
Module moves (with imports, paths, argv literals, scripts, docs, and
parents[N] formulas updated in lockstep):

- cli/deploy             -> cli/administration/deploy
- cli/validate/inventory -> cli/administration/inventory/validate
- cli/vault              -> cli/administration/secrets/vault
- cli/administration/inventory/encrypt -> cli/administration/secrets/seal
- cli/mirror             -> cli/contributing/mirror

The new cli/administration/secrets cluster collocates the whole-file
ansible-vault wrapper next to the inventory-aware per-field sealer.
The user-facing CLI subcommand paths follow the package layout, e.g.
"infinito administration deploy dedicated <inv>" and
"infinito administration inventory validate <inv>".

Help system redesign in cli/core/{discovery,help,app}.py:

- Discovery now distinguishes runnable commands (directories with
  __main__.py) from category folders (directories that only group
  further commands). Empty support packages are filtered out.
- Each category folder MAY carry a README.md; its H1 heading is used
  as the title and its first paragraph as the one-line description in
  the help output.
- "infinito" / "infinito --help" prints only the top-level entries
  (one level deep). "infinito <category>" navigates into a folder and
  shows its sub-folders and commands one level deep, with the README
  title as the page heading and the invocation as a dim subtitle.
- New "--tree [N]" flag walks the tree under the (optional) path
  prefix recursively. N caps the depth (1-based); omit N for an
  unbounded walk. The output uses Unicode tree connectors, an emoji
  per entry (folder for categories, gear for commands), a globally
  aligned description column, and multi-line wrapped descriptions
  with tree-continuity bars on continuation lines.
- Category and command listings use the full invocation
  ("infinito meta callorder") rather than the bare leaf name, so the
  output is copy-pasteable.

Convention doc + 17 CLI READMEs:

- New docs/contributing/artefact/files/cli/readme_md.md describes the
  H1+paragraph layout the help system reads, and how it differs from
  role/docs READMEs.
- README.md added for cli/{administration, build, contributing, meta}
  and their immediate sub-folders, plus cli/meta/roles/applications
  and its resolution sub-folder. Each follows the "title with trailing
  emoji + single descriptive paragraph" convention.

Mirror resolver/sync description strings (the help system reads them
from argparse --help output) added so they no longer surface as
"options:" in the global index. cli/mirror/cleanup already had a
description and is unchanged.

Fixes uncovered by the move:

- import unittest.mock added to four dedicated-deploy tests so
  unittest.mock.patch resolves under stdlib unittest discovery
  (Python 3.13 no longer surfaces unittest.mock through "import
  unittest" alone).
- parents[N] index in the new
  cli/administration/inventory/validate/__init__.py corrected from
  3 to 4 to match the deeper layout.
- test_paths.INVENTORY_VALIDATOR_PATH assertion and test_runner
  substring assertions updated to match the new validator path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-08 20:17:09 +02:00