failed_roles for the swarm/docker scope only flags a role whose job in that mode actually ran and was not green; total_state is green when every mode that ran is green. A role with no job in a mode (a host driver that never deploys in swarm) is no longer counted as failed for that mode, so 'ci trigger --failed swarm' / 'ci status --failed swarm' stop listing drv-* and other roles that simply do not run in swarm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The --failed swarm/compose/total filters keyed the deploy-job regex on the
literal words "Compose"/"Swarm", which no real CI job carries. Job names are
"<glyph> <app> <variant>" (e.g. "... / test-deploy-swarm / 🐝 web-app-x 0,1"),
so the regex matched nothing and the CLI reported "Nothing failed" for runs
full of failures. app_for_job in the role-runtime path had the identical bug
(returned the shard token instead of the app id).
Match on the 🐳/🐝 glyph instead, tolerate the orchestrator caller prefix and
the " <variant>" shard suffix, and route both consumers through one parser
(runs.app_of_job). Centralize the naming as SPOTs: MODE_GLYPHS in runs.py and
ORCHESTRATOR_PREFIX plus a workflow-sourced deploy_job_name() helper in tests,
so fixtures render real names from the workflow YAML and cannot drift back to
job names GitHub never emits.
Add an integration test that renders names from test-deploy-{compose,swarm}.yml
and asserts the parser extracts app+mode for both modes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
--run URL selects the run whose results determine the --failed apps;
without it the latest deploy run on the branch is used. Repo + run id are
parsed from the URL.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the unused shebangs (EXE001), mark the PASS emoji constant noqa
S105, annotate the test fixture as ClassVar, and use reserved example.com
hosts / a nocheck marker for the placeholder run URLs so the
url-reachability test stops flagging them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add 'infinito administration deploy ci status' and '... trigger' for
inspecting and re-running the compose/swarm deploy-test matrix of a
branch's CI run.
status: per-role docker (compose) and swarm result plus an aggregated
total column (green only when both modes are green), a trailing 'run'
column with the job URLs, --failed [total|swarm|compose] filtering, and
table/string output. Default reads the latest deploy run on the current
branch; --url reads a specific run.
trigger: dispatches entry-manual.yml for the current branch -- all roles
by default, only the last run's failures with --failed [scope], or an
explicit --apps list.
The target repo is resolved from the branch's tracking remote (a fork),
not gh's default repo. Covered by unit tests for run parsing, status
rendering, and trigger dispatch logic.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the RuntimeBudget guard and its INFINITO_MAX_RUNTIME /
INFINITO_VARIANT_TIME_OVERFLOW env handlers: the deploy round loop now runs
every variant unconditionally instead of skipping (cut) or aborting (fail)
when the projected runtime would exceed the budget. Removes runtime_budget.py,
both env handlers + their registration, the budget calls in deploy.py, the
obsolete unit tests + the TestHandlerBudget class, and the docs reference.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The handler COMMENT rendered a 188-char line in .env; the env-file-schema
lint caps non-nocheck comment lines at 140. Trim it. Also collapses two
ruff-format lines in the variant-split unit tests. Follow-up to the
runner-split variant feature; no behaviour change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Thread a single `variant` selector (CSV) end-to-end so CI can split a
role's matrix deploy across runners.
- variant_select.py (SPOT, shared by init + deploy): one --variant taking
a CSV (2 pins one round, 0,1,2 is a runner bundle, empty = full matrix);
replaces the old separate --variant/--variants.
- planner.filter_plan_to_variants filters the plan to a round subset.
- CI discovery (utils/github/variant_bundles.py) splits roles with more
than INFINITO_VARIANT_BUNDLE_SIZE (default 3) variants into bundles of
consecutive indices, one matrix runner each (5 -> 0,1,2 + 3,4), threaded
via the variant env. test-deploy-{local,universal,workstation,server}.yml
switch to matrix.include with a comma-free variant_slug for artifact/job
names (GitHub expressions have no replace()).
- runtime_budget honours INFINITO_VARIANT_TIME_OVERFLOW (cut|fail) when a
runner cannot finish its slice within INFINITO_MAX_RUNTIME.
Unit tests + docs (variants.md, deploy.md) included.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add INFINITO_MAX_RUNTIME (env handler: 6h on real GitHub Actions, 48h otherwise)
and a RuntimeBudget guard in the dev-deploy variant loop. Before each next
variant round the guard projects elapsed + longest round so far + 30m buffer;
if it would exceed the budget it stops and emits a CI warning annotation
(never fails — the rounds that ran stay deployed).
- utils/env/handlers/infinito_max_runtime.py: new handler (canonical SPOT for
the var; uses ctx.on_gha, materialised into .env by the generator).
- cli/administration/deploy/development/runtime_budget.py: RuntimeBudget +
duration parser (6h/48h/90m/3600/3600s), extracted to keep deploy.py lean.
- deploy.py: loop uses budget.exhausted()/start_round()/end_round().
- tests for both the handler and RuntimeBudget.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* cli/administration/deploy/swarm/closure.py: drop dep-walked roles
that are not present in the inventory's role groups so the
`validate_application_ids` strict in-inventory check no longer
collides with `disable=` (which removes provider roles from the
inventory). Operator-supplied seed IDs and the
`default_placement: manager` safety net are kept unchanged.
* tests/unit/cli/administration/deploy/swarm/test_closure.py: add
`test_operator_ids_dep_walked_not_in_inventory_dropped` to pin
the new behaviour and update the existing scenarios to put the
filtered roles in their test inventories.
* tests/lint/ansible/shell/test_container_exec_is_stack_host_gated.py:
recognise `include_tasks: { file: X, apply: { ... } }` dict form
alongside the free-form string. Without it the lint's include
graph dropped the apply-include edge and false-positived gated
files reached by it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a `swarm:` subcategory under `svc:` in roles/categories.yml so the
shared-cluster role is properly invokable, and renames the existing
`svc-swarm` role + inventory group to `svc-swarm-node` (with
`get_entity_name`-aligned primary key `node:` in meta/services.yml).
`svc-swarm-manager` stays as the manager-marker role, becoming invokable
via the same new category and re-keyed to `manager:`.
Consumers of the cluster-orchestration service (svc-registry-cache,
svc-registry-docker, svc-swarm-manager) now reference the service-key
`node:` (was `swarm:`), matching the renamed primary entry. The planner
empirically resolves `service_registry['node']` -> `svc-swarm-node` and
emits it into --include for stack-host deploys, so the CI failure on
`universal-svc-registry-docker` (`svc-swarm: not present in inventory`)
no longer reaches `ValidDeployId`.
Breaking change for operators: inventory groups named `svc-swarm:` must
be renamed to `svc-swarm-node:`. `DEPLOYMENT_MODE` reads
`groups['svc-swarm-node']`, so a stale `svc-swarm:` group silently falls
back to compose mode.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
svc-cache-registry -> svc-registry-cache
svc-docker-registry -> svc-registry-docker
svc-docker-swarm -> svc-swarm
svc-docker-swarm-manager -> svc-swarm-manager
categories.yml: drop svc.docker, add svc.registry. Deliberately no svc.swarm
category so get_entity_name('svc-swarm') stays 'swarm' instead of collapsing
to '' (which would break discover_role_services).
Entity keys updated to match the new svc.registry prefix:
services.cache-registry.* -> services.cache.*
services.registry.* -> services.docker.*
01_constructor: replace svc-docker bootstrap-loop entry with explicit
include_role of svc-swarm + bootstrap loop over [svc-registry, svc-storage].
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* tests/unit/cli/administration/deploy/swarm/test_closure.py — drop
the `parents[6]` + `nocheck: project-root-import` combo. The
file-path loader still bypasses the dedicated CLI's heavy package
__init__, but the path now comes from the canonical
`from utils import PROJECT_ROOT` so the project-root-import lint
passes without a per-line opt-out.
* utils/tests/swarm/write_extras.py — restructure the
`Path.read_text()` call onto a single line so the
`# nocheck: cache-read` annotation lands on the call site itself.
Autoformat had previously line-broken the chained `.strip()`
expression and pushed the marker onto a different line, making
the suppression invisible to the lint scanner.
* tests/integration/infrastructure/compose/test_compose_volumes_call_required.py —
the `test_no_template_uses_deprecated_pipe_form` walker used a raw
`roles_dir.rglob("templates/**/*.j2")`, which trips
`test_no_raw_project_walk` in the lint suite. Switch to the cached
`iter_project_files()` helper with a roles-prefix + `/templates/`
filter; behaviour is unchanged but the walk goes through the
shared SPOT.
* roles/svc-cache-registry/meta/services.yml — annotate the role
with `# nocheck: required-by-coverage`. svc-cache-registry is
intentionally opt-in via `default_placement: manager` plus the
swarm CLI closure, NOT via a downstream `required_by` chain, so
the lint's non-invokable-without-required_by check would otherwise
reject it.
`make autoformat` passes. `make test` is now green across lint /
test-external / test-integration / test-lint / test-unit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CI failures from run 26987751848:
* `cache-read` (3 hits): annotate the `Path.read_text()` calls in
utils/tests/swarm/{write_extras,extend_inventory}.py — every one
reads a file the same script just wrote, so the cache helper would
serve stale content. Per `# nocheck: cache-read` carve-out.
* `project-root-import`: annotate the `parents[6]` in the swarm
closure test loader. The file-path loader bypasses the package
__init__ to avoid pulling in the heavy dedicated CLI imports.
* No-unittest-tests: tests in test_closure.py inherited only via
`_BaseClosureCase` (which itself extends `unittest.TestCase`), but
the AST lint walks bases literally and does not follow indirect
inheritance. Switch each `Test*` class to multi-inherit
`(_BaseClosureCase, unittest.TestCase)` so the discovery check
recognises them as runnable.
* `hardcoded-dns-resolver`: the per-block head suppression in
inventories/development/swarm.yml was insufficient; the lint scans
per-line and needs the marker on the actual `dns: "1.1.1.1"` line.
Verified locally: 16 of the previously-failing tests now pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New ``cli.administration.deploy.swarm`` package wraps the strict
``cli.administration.deploy.dedicated`` runner with a layered closure
so swarm deploys never require operators (or test scripts) to
enumerate every infra role:
effective --id = seed (operator --id, else inventory_role_groups)
∪ dep-walk(seed via applications_if_group_and_all_deps)
∪ default-placement safety net
The dep-walker is the same SPOT
(``utils.roles.applications.in_group_deps.applications_if_group_and_all_deps``)
the runtime constructor consumes via ``lookup('applications_current_play')``,
so the CLI-side closure and the in-play role set stay in sync.
The dedicated CLI keeps strict explicit-only semantics: ``main`` was
split into ``parse_args`` + ``run_from_args`` so the swarm wrapper
can share parser, mode wiring, validator and runner verbatim and
only mutate ``args.id``.
Meta-side: ``svc-db-postgres`` gains ``default_placement: manager``
(semantic symmetry with mariadb). ``default_placement`` is now a
PLACEMENT hint only ("if deployed, run on the manager"); deploy
decisions come from the dep-walk. Apps that declare ``mariadb:``
pull svc-db-mariadb; apps that declare ``postgres:`` pull
svc-db-postgres; apps that declare neither get no central DB.
Shared utils extracted so the four consumers stay aligned:
* ``utils.inventory.groups.inventory_has_group``: YAML + INI static
inventory group probe (extracted from ValidDeployId).
* ``utils.roles.meta_lookup.{get_role_default_placement,
iter_roles_with_default_placement}``: single source of truth for
``default_placement`` discovery (lookup plugin is now a thin wrapper).
Test pipeline (``scripts/tests/deploy/swarm/*``):
* Heredoc-Python shells converted to proper modules under
``utils.tests.swarm.*`` and invoked via ``python -m``:
- extend_inventory.py (was 08_extend_inventory.sh)
- write_extras.py (was 09_write_extras.sh dynamic half)
- derive_includes.py (replaces 06's hardcoded includes array)
- write_hosts_entries.py (replaces 02's hardcoded /etc/hosts heredoc;
derives every canonical + www-redirect domain from APP_ID via
canonical_domains_map + generate_all_domains, so the test
works for any app in the matrix without per-app maintenance)
* 01_create_network.sh deleted (single docker command inlined in workflow).
* 11_deploy.sh deleted (swarm CLI now called directly in workflow).
* 09_write_extras.sh: static half moved to inventories/development/swarm.yml.
* Remaining shell scripts renumbered 01-13 without gaps.
* 08_wait_converge.sh: case block on DB_DEP replaced with single
conditional (container name == DB_DEP).
* 06_provision_inventory.sh: hardcoded includes array replaced with
`mapfile -t includes < <(python -m utils.tests.swarm.derive_includes)`.
* Workflow YAML step references updated to new script names + module
invocations.
New tests (39 covering the closure surface):
* tests/unit/cli/administration/deploy/swarm/test_closure.py (14)
* tests/unit/utils/inventory/test_groups.py (11)
* tests/unit/utils/roles/test_meta_lookup.py (+3 for default_placement)
* tests/unit/utils/roles/validation/test_deploy_id.py (unchanged
8 tests still pass after refactor onto the shared util).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root cause of step-08 deploy failures across all 4 test-development
distros in run 26572559126: `make compose-deploy mode=reinstall
apps=web-app-dashboard INFINITO_SERVICES_DISABLED=matomo` (step 07)
made make export the cmdline var into the recipe's shell env, which
reached `runner.py` → `make setup` → `make dotenv`. The dotenv
generator's `setdefault` then baked the runtime-only value into the
on-disk `.env`. Step 08 sourced that .env, filtered `web-app-matomo`
out of its primary apps, and exited with "All primary apps disabled".
Fix:
- Drop the 7 routing knobs (apps, mode, purge, bundles, disable,
full_cycle, variant) from `passthrough.STATIC_KEYS` and from
`default.env`. They are per-invocation; .env is the persistent SPOT
and must not capture them.
- Switch every consumer from `os.environ["KEY"]` strict access to
`os.environ.get("KEY", "")` so the missing-key path stops at "no
filter" instead of a KeyError.
- Rename the env-var surface to match the make-alias surface 1:1
(lowercase): INFINITO_APPS → apps, INFINITO_SERVICES_DISABLED →
disable, INFINITO_BUNDLES → bundles, INFINITO_FULL_CYCLE →
full_cycle, INFINITO_VARIANT → variant, INFINITO_DEPLOY_MODE → mode,
INFINITO_PURGE_ENTITIES → purge. The make alias was the user-facing
name in every doc; routing-knob env vars now use the same name so
the cmdline form, the recipe env-prefix, and the consumer reads
all line up.
- Collapse the Makefile `compose-deploy` recipe to just the two
knobs whose alias differs from the env name (`type` →
INFINITO_DEPLOY_TYPE, `debug` → INFINITO_DEBUG). GNU Make
auto-exports cmdline variables into the recipe's shell env, so the
identical-name `$(if $(apps),apps="$(apps)")` lines were pure
noise.
- Drop `disable: ${disable}` from compose.yml. The dev CLI builds
`extra_env` per `docker exec` call and forwards `disable` via
`-e`, so the container env never needs the value frozen at
compose-up time.
- Update the 3 actual leak call sites
(07/08/09_deploy_*.sh) to the make-alias form
(`disable=matomo`, `full_cycle=true`, `variant=1`).
- Add `SIM112` to ruff's ignore set: the lowercase env names are
the deliberate make/env parity, not a typo.
Repro before fix: `INFINITO_SERVICES_DISABLED=matomo make dotenv`
wrote `INFINITO_SERVICES_DISABLED=matomo` to `.env`. After fix:
`disable=matomo make dotenv` leaves `.env` untouched.
Tests: `make test` and `make autoformat` green (2711 tests, 8 lint
sub-targets).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Since 576fad84f (2026-05-10) moved role images from
`defaults/main.yml` into `meta/services.yml`, matrix-deploy's variant
baker has been writing the FULL effective config (services.yml +
variants.yml deep-merged via `get_variants`) into host_vars through
the `--vars` JSON consumed by `apply_vars_overrides`. That populates
`applications.{role}.services.{svc}.image` (and `.version`, .cpus,
etc.) with the upstream value, which then silently disables
`apply_mirror_overrides` for every role with an explicit image
declaration: the default `if_missing` policy short-circuits when the
field is already non-blank, so the rendered compose / Dockerfile FROM
keeps pulling from the upstream registry rather than the GHCR mirror.
The regression went unnoticed because Docker Hub usually serves the
pull, but it eventually surfaced as a BuildKit 429 on
`docker.io/library/php:8.2-apache` during the suitecrm Dockerfile
build (job 77602110531 on run 26362676803). Nextcloud, moodle,
mariadb consumers, postgres consumers, simpleicons, phpmyadmin,
phpldapadmin, jenkins, baserow, fider, ... all share the same latent
defect — only suitecrm happened to hit the rate limit first.
Fix the root cause instead of papering over per-role with
`mirror_policy: force`: introduce `get_variant_overrides_only` next
to `get_variants` and switch payload.py to it. The new function
returns the raw `meta/variants.yml` entries per role — no merge with
`meta/services.yml`. matrix-deploy now bakes only the variant-
specific overrides into host_vars; the role's services.yml fields
flow through the normal Ansible lookup chain at deploy time, leaving
`image`/`version` blank in host_vars so `apply_mirror_overrides`
populates them from `mirrors.yml` under its default `if_missing`
policy.
Behavior preserved for the other `get_variants` consumers:
* utils/manager/inventory.py:104 (schema-driven credentials) keeps
the full merged variant payload.
* planner.py:58 (variant-count only) is unaffected.
* legacy_resolver.py:42 (CombinedResolver runtime services_overrides)
keeps the full merged view.
Coverage:
* 3 new tests in test_applications.py for the new helper (single
empty variant when no meta/variants.yml, raw override without base
merge, cache invariance).
* test_payload.py + test_builder.py mocks repointed from `get_variants`
to `get_variant_overrides_only`.
* `make test` green across all 5 targets (lint, unit, integration,
external, test-lint) — 188 tests total.
Side-effects assessed:
* Repo-wide grep confirms no template / task accesses
`applications.X.services.Y.image` directly via hostvars — every
consumer goes through `lookup('config', …)` which still deep-merges.
* Roles previously pulling upstream now resolve via the GHCR mirror
whenever the image is already synced (the `images-mirror-missing.yml`
workflow keeps the destination populated).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
InventoryManager.resolve_schema_includes_recursive previously read the
root role's base meta/services.yml when discovering shared providers,
regardless of which matrix variant the planner had pinned for the
round. For a role whose variants.yml enables additional services per
variant (e.g. svc-bkp-container-2-local variant 2 enabling
services.ldap), the base config had no `enabled+shared` flag for those
services, so the recursive resolver never visited the variant's
shared providers — their credentials were never generated, and the
deploy crashed in the provider's env.j2 with "credentials is defined
in schema at roles/<provider>/meta/schema.yml but not set in
application '<provider>'".
Wires the active variant through the existing inventory pipeline:
- utils/manager/inventory.py: InventoryManager(variant=None|int). When
set, load_role_config_by_path overlays variants.yml[N] onto the root
role's meta config (only for the root — transitive providers keep
their own base lookup). variant=None preserves the pre-refactor base
behavior for every standalone caller.
- cli/administration/inventory/credentials: new --variant N flag,
forwarded to InventoryManager.
- cli/administration/inventory/provision: new --app-variants <JSON>
flag ({app_id: variant_index}), forwarded per-app to the credentials
subprocess.
- cli/administration/deploy/development/inventory/builder.py: emits
--app-variants from spec.variant_selectors() when the round pins any
variant. Apps without variants.yml collapse to a single empty
overlay in get_variants so this is a no-op for them.
10 unit tests cover: variant overlay for root role, variant=None
fallback to base, out-of-range fallback, no overlay on non-root
paths, --variant CLI forwarding (set + default), --app-variants
generator forwarding (set + omit), builder --app-variants emission
(set + omit).
Caveat: bkp-c2l CI variant 1 still loads svc-db-openldap transitively
at runtime (outside the planner's variant closure) — a separate
sys-service-loader / role-meta bug, not addressed here.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Four follow-ups uncovered by \`make test\` after 3c9db6142:
* tests/lint/ansible/variables/test_role_and_group_vars_used:
TLS_ENABLED is declared in group_vars/all/02_tls.yml but never
read via \`{{ TLS_ENABLED }}\`; it is consumed by
plugins/lookup/{tls,email,cert}.py, which the YAML/Jinja-scoped
lint cannot see. Tag the declaration with the project's standard
\`# nocheck: unused-var\` marker and document the consumer.
* tests/lint/filesystem/python/test_no_raw_project_walk: the new
test_no_ansible_env_lookups.py used \`Path.rglob("*")\` for its
walk, which is the exact shape the no-raw-walks lint forbids in
test code. Switch to \`iter_project_files(extensions=_SCAN_EXTS)\`
with a relative-path prefix filter; same coverage, shared cache.
* tests/unit/cli/administration/deploy/development/test_inventory_vars_file_spot:
the new lint test legitimately names the dev inventory path
(\`inventories/development/default.yml\`) in its failure message.
Add it to the SPOT-drift-guard allowlist with a justification.
* tests/integration/ansible/jinja/test_variable_definitions: the
debug task added in 0884eb4e2 prints \`ansible_env\` and \`vars\`,
both Ansible magic variables. Add them to the test's reserved-name
list so the static analyzer stops flagging them as undefined.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
compose.yml no longer hides services behind \`profiles: ["ci"]\`. The
profile existed solely as a no-op safety net against a bare
\`docker compose up\` in the repo root, but the top-level compose.yml is
already explicitly scoped to local dev / CI testing (production goes
through the Ansible roles under roles/<role>/templates/), so the safety
net is redundant with the documented scope.
Every \`--profile ci\` callsite collapses:
- compose.yml: \`profiles: ["ci"]\` removed from coredns + infinito
- Profile.args() deleted; Profile keeps is_ci() and
registry_cache_active() for the cache-override gating
- compose.py / down.py: profile.args() splat removed from the
docker-compose command builder; down.py drops its now-unused
Profile import
- 5 shell wrappers (install, lint, tests/code, meta/resolve/apps,
meta/resolve/diff/affected_roles): plain \`docker compose\` instead
of \`docker compose --profile ci\`
- tests/unit/.../test_profile.py: TestProfileArgs class removed
- tests/unit/.../test_compose.py: two cmd-shape asserts updated to
reflect the missing --profile pair
Docs:
- docs/contributing/artefact/files/compose.yml.md: the
\"every service MUST be behind a profiles: entry\" rule is gone
- docs/contributing/actions/debugging/ci.md: the
\`docker compose ... --profile ci config\` debug snippet drops the
flag; while touching the file the two em-dash clause separators in
the reproduction-profiles shell block are rewritten to colons
(documentation.md em-dash rule), and one list-item in
compose.yml.md is collapsed to one sentence per line for
consistency with its siblings
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
With env.development and env/ci.env both gone, env/default.env was the
last inhabitant of the directory. Hoisting it to the repo root removes
the empty-shell indirection: the SPOT is now [default.env](default.env),
sitting next to .env (its generated companion) and other repo-root
configuration artefacts like compose.yml, Dockerfile, and Makefile.
Path updates land in:
- generator: cli/meta/env/__main__.py (REPO_ROOT / "default.env")
- workflow: .github/workflows/test-install-make.yml (awk reads from
default.env, error message updated in lockstep)
- lint tests: test_used_keys_declared.py and test_infinito_prefix.py
switched from globbing env/*.env to targeting default.env directly;
test_default_env_keys_handled.py points at the new path
- drift test: test_inventory_vars_file_spot.py whitelist + STATIC_ENV
- smoke test: tests/integration/meta/env/test_dotenv_generator.py
STATIC_ENV + every docstring/error message
Doc-string and prose references in utils/env/*, scripts/*,
cli/administration/deploy/development/{common,coredns}.py,
Makefile, Dockerfile, and every doc page that mentioned env/default.env
are rewritten in the same pass so future readers see one path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The four ci.env keys (INFINITO_DNS_IP, INFINITO_DOMAIN, INFINITO_DOMAIN_RE,
INFINITO_IP4) were only honored by docker compose --env-file, so processes
that sourced .env via load.sh saw INFINITO_IP4 as unset. As a result
networks.internet.ip4 in the Ansible playbook fell through to the
127.0.0.1 default and the DinD DNS-check handler failed on the deploy
server matrix.
Moving the defaults into env/default.env makes them flow through the same
.env / load.sh / os.environ chain as every other INFINITO_* key, so
\${INFINITO_IP4} in compose.yml, lookup('env', 'INFINITO_IP4') in
group_vars, and envsubst over the CoreDNS Corefile template all see the
same value deterministically.
Side fix: utils/env/parser.py now undoes the writer's \\\\/\\" escaping for
quoted values, closing a round-trip gap that surfaced once
INFINITO_DOMAIN_RE (the only value containing a literal backslash) started
flowing through the generator.
All --env-file env/ci.env call sites (compose.py, down.py, install/lint/
test wrappers, resolve scripts, test-dns workflow + script) are removed.
coredns.py now reads .env via parse_static_env, docs and unit/integration
tests follow.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Promote the key from GHA-only (gha_passthrough) to always-emitted (passthrough)
so the deploy CLI can read it strictly via os.environ[...]. The previous
INFINITO_NO_BUILD layout left the key unset locally, which crashed
cli/.../up.py and cli/.../compose.py with KeyError after the env-pipeline
refactor that introduced gha_passthrough.
New semantics:
INFINITO_BUILD=1 build the local image (default; local mode)
INFINITO_BUILD=0 skip build, pull image (CI overrides via -e)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pulls VENV/PYTHON/PIP/PYTHONPATH out of the dotenv handler pipeline
into scripts/meta/python/detect.sh. load.sh now sources detect.sh and
prepends ${VENV}/bin to PATH above the INFINITO_ENV_LOADED guard, so
every BASH_ENV re-entry re-resolves PYTHON the moment a venv
materialises mid-chain — the same fix as b79ed2869 but at the SPOT,
which retires the reload-trick from ansible.sh, python.sh and
pre-commit.sh.
Also renames INVENTORY_VARS_FILE → INFINITO_INVENTORY_VARS_FILE under
the new env-prefix lint, drops the 5 dead INFINITO_ENV_*_LOADED unsets
left over from the multi-loader era, and adds a lint guarding the
INFINITO_ prefix invariant on env/*.env keys.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three new lint tests close gaps in the INFINITO_* SPOT contract:
* tests/lint/filesystem/env/test_default_env_keys_handled.py
Every default.env key must be wired through a handler.
* tests/lint/filesystem/env/test_used_keys_declared.py
Every INFINITO_* in .sh/.py must live in env/{default,ci}.env or
a handler; per-line `# nocheck:` suppresses runtime-only refs.
* tests/lint/filesystem/python/test_no_infinito_get_defaults.py
Forbid ANY default in `.get("INFINITO_*", ...)` -- including the
empty-string fallback (which silently masks an unset key). The
nocheck check walks every physical line the call spans so an
autoformat-split call still honours the marker.
Cache namespace flattens to a single prefix:
INFINITO_PACKAGE_CACHE_* -> INFINITO_CACHE_PACKAGE_*
INFINITO_REGISTRY_CACHE_* -> INFINITO_CACHE_REGISTRY_*
The five previously driver-injected paths (APT_LIST, NPMRC,
PIP_CONF, REGISTRY_PROXY_CONF, FRONTEND_CA_FILE) move into
env/default.env and passthrough.STATIC_KEYS; the
`cache_env_overrides()` helper and its two call sites are gone
along with the test-fixture override for FRONTEND_CA_DIR.
SPOT-strict rewrites:
- 19 sites: `something.get("INFINITO_X", "")` -> subscript
`something["INFINITO_X"]` (loud KeyError when env loader did
not run; all keys live in env/default.env).
- 2 handler-internal bootstrap reads suppressed with
`# nocheck: handler-bootstrap-read`.
- 30 runtime-only / driver-injected / test-fixture references
annotated with `# nocheck: <reason>` per-line.
- Unit tests that previously removed SPOT-guaranteed keys via
`{k: v ... if k \!= KEY}` or `clear=True` switch to setting
the key to "" explicitly, matching the new contract.
Side fixes:
- INFINITO_DNS_DOMAIN + INFINITO_DNS_HOSTS_FILE move from
shell-side defaults in common.sh to env/default.env and
passthrough.STATIC_KEYS.
- dns_systemd_is_operational() drops the dead
INFINITO_DNS_FORCE_HOSTS_FALLBACK override branch (the
variable was never set anywhere).
- INFINITO_WORKER_FETCH switches from `os.environ.get(KEY, "1")`
to `os.environ[KEY]`.
- scripts/tests/code/wrapper.sh exec args move into a bash
array so `# nocheck:` on INFINITO_TEST_TYPE can live on its
own element line (SC2215 fix).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rename env/static.env → env/default.env across the entire repo
(string label, doc links, Python file-path constants) — the
`parse_static_env*` function names stay as a stable API surface.
Sort all `*.env` / `*.env.j2` entries alphabetically (52 in
default.env, 4 in ci.env, 46 of 49 j2 templates; 3 templates
with Jinja2 `{% %}` control flow are skipped because conditional
blocks would tear apart under a naive sort). Sort all 83
Makefile target blocks alphabetically; .PHONY group headers
stay in place.
Collapse the two remaining multi-line Makefile recipe comments
(dotenv-force, test-signed) to single-line WHY comments.
Add four lint tests under tests/lint/filesystem/ to keep the
invariants enforced going forward:
- env/test_alphabetical_sort.py
- env/test_single_line_comments.py (# nocheck: markers exempt)
- makefile/test_alphabetical_targets.py
- makefile/test_single_line_comments.py (# nocheck: markers exempt)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace 8 bash env modules with a single Python generator
(cli.meta.env) that materialises env/static.env + runtime context
into a gitignored .env. Each variable lives in its own handler
under utils/env/handlers/; defaults carry inline one-line `# ...`
comments.
Rename to the INFINITO_* schema:
- workflow inputs: APPS, SERVICES_DISABLED, CMD, DEBUG, WHITELIST,
BUNDLES, FULL_CYCLE, VARIANT, RUN_FLAGS, LIMIT_HOST, PARALLEL
- inventory: INVENTORY_DIR / INVENTORY_FILE / HOST_VARS_FILE,
PW_FILE -> INVENTORY_PASSWORD_FILE
- venv: VENV_BASE / VENV_FALLBACK / VENV_NAME
- new SPOT defaults: COMPILE_SILENCE, CONTAINER_NO_CA,
WAIT_HEALTH_TIMEOUT_S, PACKAGE_CACHE_FRONTEND_INIT_IMAGE,
promoted IMAGE_TAG from GHA-only to general passthrough
compose.yml hardening:
- inline \${VAR:-default} removed; strict \${VAR:?...} everywhere
- new integration lint blocks regressions
Tooling:
- scripts/install/all.sh owns the stamp-file install logic
(Makefile target delegates)
- scripts/meta/env/load.sh is the single shell entry point;
8 bash modules deleted
- new shell lint: no INFINITO_* defaults outside env/static.env
Tests:
- 93 new unit tests for utils/env/{builder,parser,runtime,writer}
- restructure tests/lint into repository/ (file-content rules) vs
filesystem/{shell,python,env}/ (filesystem-shape rules)
- drift + smoke integration tests against the generator
Docs:
- variables.md is now a pointer; per-variable docs live next to
the key in env/static.env / env/ci.env
- compose.yml.md drops the Default column
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Matrix-deploy orchestrator now pushes `VARIANT_INDEX` (0-based round
id) alongside the existing `ASYNC_ENABLED` extra var for both PASS 1
(sync) and PASS 2 (async) runs. Test-e2e-playwright's archive step
consumes both to derive a per-(variant, pass) destination so a
subsequent round cannot overwrite the previous one's reports.
- New `roles/test-e2e-playwright/files/archive_playwright_reports.sh`
moves the playwright-junit.xml / playwright-report/ / test-results/
outputs into `variant-<N>/<sync|async>/` under the role's reports
dir, idempotently (re-running the same tuple replaces rather than
fails). `run_one.yml` invokes the script via `ansible.builtin.script`
so the YAML stays a single 7-line task.
- `VARIANT_INDEX: 0` lives in `defaults/main.yml` to keep the variable
defined for dev / non-matrix invocations. Role defaults are not
considered SPOT, so the `| default(0)` belt-and-suspenders inside
the script invocation stays valid against the spot-default lint.
- Summary script now renders an 8-column table: Time | App | Variant |
Pass | Test | Status | Duration | Message. Per-test ISO timestamps
are derived from the JUnit `<testsuite timestamp>` attribute plus the
cumulative duration of preceding cases; file mtime is the fallback
when the attribute is absent. App / Variant / Pass are extracted
from the artefact path layout produced by the archive script.
- Matrix-deploy unit tests refreshed to expect the new
`extra_ansible_vars` shape; summary unit tests gain coverage for
`_to_iso_z`, `_parse_iso_timestamp`, the four supported path
layouts, and the file-mtime fallback (55 tests total).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`3c1d5d821` solved the cross-round backbone-abriss problem by making
`include` equal the union of every variant's closure for every round.
Side-effect: every matrix round then ALSO deployed the full union, so
`08_deploy_performance.sh` saw `dashboard.infinito.example` still
answering HTTP 200 after a matomo-only `deploy-fresh-purged-apps` —
the deploy step was no longer matomo-only at all.
Split the two concerns instead:
- `include` (PlanEntry[3]) goes back to the per-round variant-closure.
Both `init`/`builder` (`--include` for the inventory bake) and
`deploy` (`--id` for ansible) consume it. Each round therefore
deploys ONLY its own variant-closure again — the pre-`3c1d5d821`
semantics.
- `purge_set` (PlanEntry[4], new) carries the union across all rounds.
`deploy.py` uses it for the inter-round wipe instead of the previous
round's include. Result: between rounds the WHOLE matrix footprint
is purged, not just the previous round's variant slice — a variant
pinning `services.<X>.shared: false` cannot leave its bundled
provider on the host for a sibling round that wires the same service
shared. The backbone-abriss `3c1d5d821` was hunting stays fixed:
whichever round needs the backbone re-deploys it immediately after
the wipe (purge runs at the start of each round, not the end).
`previous_round_include` tracking is gone. The first iteration skips
the purge (nothing to wipe yet), which also covers `total_rounds == 1`
and `--variant N` pinned single-folder modes cleanly.
Unrelated, but staged together because it is part of the same
investigation:
- `roles/dev-nix/templates/nix.conf.j2`: enable `keep-derivations` /
`keep-outputs`, so `nix-collect-garbage -d` (run by the aggressive
prune step) does not drop the build closure of pkgmgr's flake; this
was making every deploy re-fetch the build inputs from
cache.nixos.org and overshoot pkgmgr's update timeout.
- `roles/pkgmgr/tasks/01_core.yml`: raise the pkgmgr update timeout
from 600s to 1800s and lower the retry count from 6 to 3, so a slow
fetch on a single try no longer pushes total wallclock past three
hours.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Commit 16f323319 added a strict variant-only include rule + conflict
detector in `inventory/variants.py`, gated behind the presence of any
primary's `meta/variants.yml`. In practice the rule never paid off:
* The CI failure that motivated it (matomo's matrix transition leaving
a stale dashboard nginx vhost -> 502) is already addressed by the
nginx-purge primitive shipped in commit bfd04e4a1.
* The existing `CombinedResolver` path already implements the "any
role says true or `in group_names` -> pull" semantics via
`utils.roles.applications.services.registry.is_explicit_truth`, so
every needed service still gets pulled and services that all roles
mark False stay out of the include.
* The strict literal-equality conflict detector was a false-positive
factory: 79/80 variant-equipped primaries failed it against the
current repo state because roles legitimately disagree on `enabled`
flags for shared services.
This commit removes the entire variant-only branch and restores the
single-path planner. Round-index clamping for non-primary deps is left
exactly as before 16f323319 -- only the additions are reverted.
The file-split refactor from 16f323319 is intentionally kept; only the
`variants.py` submodule, the corresponding `test_variants.py`, and the
variant-only gate test class in `test_planner.py` are removed. The
`__init__.py` re-exports + `__all__` are pruned to match.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two changes ship together because the variant-only matrix-deploy logic
was written into the legacy 697-line inventory.py and then re-split:
* Source package (max 240 lines per submodule, well under the 300 budget):
- spec.py DevInventorySpec + PlanEntry data types
- payload.py _resolve_variant_payloads + _bake_overrides
- legacy_resolver.py CombinedResolver fallback (default-mode path)
- variants.py strict variant-only include resolution
- planner.py plan_dev_inventory_matrix + filter_plan_to_variant
- builder.py build_dev_inventory + build_dev_inventory_matrix
- __init__.py public-API re-exports + back-compat shims
* Test package mirrors the source 1:1 with a shared _fixtures.make_spec
to avoid duplicating the DevInventorySpec defaults across files.
* New variant-only matrix mode (gated on any primary shipping
meta/variants.yml): only services whose merged variant payload pins
both enabled and shared to literal True become inventory groups;
Jinja flags never qualify (group_names does not exist yet at build
time); run_after only orders the include, never expands it;
cross-primary disagreement on a (service-key, round) raises ValueError.
Default mode (no variants.yml on any primary) keeps the legacy
CombinedResolver path untouched.
* Docs in design/{inventory,variants}.md updated to point at the new
per-submodule paths; test_nginx_vhosts.py's Test classes now also
inherit directly from unittest.TestCase so the AST-level lint in
test_all_test_files_have_tests.py picks them up (transitive base via
NginxVhostsTestBase alone is not enough).
Coverage: 458 cli unit tests, 349 utils unit tests, 99 dev-deploy tests,
all green; ruff check clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Strip every `req <NNN>`, `req-<NNN>`, `Requirement <NNN>`,
`requirement-<NNN>`, and `docs/requirements/<NNN>-...md` mention from
non-Markdown source files. Comments anchored to requirement numbers
rot when requirements are renumbered or rewritten and conflate
present-state behaviour (which the comment should describe) with the
audit trail of how the code got there (which belongs in commit
messages, PR descriptions, and the requirement file itself).
* Document the rule in `docs/contributing/documentation.md` under
"Comments". Code MUST describe the *why* in present-state language;
contract enforcement lives in tests, not folklore.
* Clean 254 source files across `roles/`, `tests/`, `plugins/`,
`cli/`, `scripts/`, `utils/`, `group_vars/`, and the repo-root
`eslint.config.js`. Pure comment / docstring / log-message edits;
no behavioural change.
* Add `tests/lint/repository/test_no_req_references_in_code.py` to
enforce the rule going forward across `.js`, `.py`, `.yml`,
`.yaml`, `.j2`, and `.sh` files. The legacy
`test_no_stale_requirement_refs.py` is exempt as the only file
whose purpose is to detect these very tokens.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Removes 67 live references to docs/requirements/<NNN>-*.md files across
43 contributor docs, role READMEs, code comments, task files, and tests.
Requirement docs are initial-creation-only artefacts; live references
freeze a snapshot that rots once the per-feature SPOT (a contributor
doc, a code module, a lint test) takes over the live behaviour.
In the same pass, fixes documentation.md violations on the touched
files: drops em-dashes used as thought breaks, removes "SPOT for ..."
self-labels, strips historical phrasing ("the legacy", "previously",
"used to", "is no longer used", "matching the historical"), and
reflows soft-wrapped sentences so each sentence stays on one logical
line.
The `test_inventory_vars_file_spot.py` whitelist for the literal
inventory path drops the docs/requirements/003 entry and instead
skips docs/requirements/ entirely; future requirement docs that
mention the literal path are then ignored automatically.
Verified by tests/lint/repository/test_no_stale_requirement_refs.py
(green tree-wide).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Module moves (with imports, paths, argv literals, scripts, docs, and
parents[N] formulas updated in lockstep):
- cli/deploy -> cli/administration/deploy
- cli/validate/inventory -> cli/administration/inventory/validate
- cli/vault -> cli/administration/secrets/vault
- cli/administration/inventory/encrypt -> cli/administration/secrets/seal
- cli/mirror -> cli/contributing/mirror
The new cli/administration/secrets cluster collocates the whole-file
ansible-vault wrapper next to the inventory-aware per-field sealer.
The user-facing CLI subcommand paths follow the package layout, e.g.
"infinito administration deploy dedicated <inv>" and
"infinito administration inventory validate <inv>".
Help system redesign in cli/core/{discovery,help,app}.py:
- Discovery now distinguishes runnable commands (directories with
__main__.py) from category folders (directories that only group
further commands). Empty support packages are filtered out.
- Each category folder MAY carry a README.md; its H1 heading is used
as the title and its first paragraph as the one-line description in
the help output.
- "infinito" / "infinito --help" prints only the top-level entries
(one level deep). "infinito <category>" navigates into a folder and
shows its sub-folders and commands one level deep, with the README
title as the page heading and the invocation as a dim subtitle.
- New "--tree [N]" flag walks the tree under the (optional) path
prefix recursively. N caps the depth (1-based); omit N for an
unbounded walk. The output uses Unicode tree connectors, an emoji
per entry (folder for categories, gear for commands), a globally
aligned description column, and multi-line wrapped descriptions
with tree-continuity bars on continuation lines.
- Category and command listings use the full invocation
("infinito meta callorder") rather than the bare leaf name, so the
output is copy-pasteable.
Convention doc + 17 CLI READMEs:
- New docs/contributing/artefact/files/cli/readme_md.md describes the
H1+paragraph layout the help system reads, and how it differs from
role/docs READMEs.
- README.md added for cli/{administration, build, contributing, meta}
and their immediate sub-folders, plus cli/meta/roles/applications
and its resolution sub-folder. Each follows the "title with trailing
emoji + single descriptive paragraph" convention.
Mirror resolver/sync description strings (the help system reads them
from argparse --help output) added so they no longer surface as
"options:" in the global index. cli/mirror/cleanup already had a
description and is unchanged.
Fixes uncovered by the move:
- import unittest.mock added to four dedicated-deploy tests so
unittest.mock.patch resolves under stdlib unittest discovery
(Python 3.13 no longer surfaces unittest.mock through "import
unittest" alone).
- parents[N] index in the new
cli/administration/inventory/validate/__init__.py corrected from
3 to 4 to match the deeper layout.
- test_paths.INVENTORY_VALIDATOR_PATH assertion and test_runner
substring assertions updated to match the new validator path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
