Single SPOT for matrix-deploy topology
=======================================
Before this change two paths derived the deploy plan from different
sources and could disagree:
* Group inclusion (`--include` / `group_names`): driven by
`cli.meta.applications.resolution.combined.CombinedResolver`,
which read each role's `meta/services.yml` raw and ignored
`meta/variants.yml` entirely. A variant that flipped
`services.<X>.enabled` to a literal `true` did NOT pull `<X>`'s
provider role into the deploy plan.
* Inventory bake (`host_vars/applications.<app>.*`): driven by the
variant-aware payload merge in `cli.deploy.development.inventory`,
so the role saw the variant-merged services map at deploy time.
The split surfaced concretely for `web-app-wordpress` variant 0
(`services.discourse.enabled: true` in `meta/variants.yml`): the
inventory baked `discourse.enabled = true`, the wp-discourse plugin
task gate fired, and the install loop tried `docker exec discourse`
against a container the variant-blind resolver had never pulled.
This commit collapses both consumers onto the same per-round,
variant-merged services map. The variant-aware resolver (Phase 1+2)
is the additive plumbing, but the deploy/init wiring (Phase 3-5) is
what actually changes behaviour at runtime.
Variant-aware resolver (Phase 1+2)
----------------------------------
`role_introspection.load_shared_service_roles_for_app` accepts a
`services_override` kwarg; `CombinedResolver.__init__` accepts a
`{role_name: services_dict}` map and routes it through the loader.
A `CombinedResolver` instance is round-specific when overrides are
supplied — its per-role edge cache is variant-resolved, do NOT reuse
across rounds. The combined-resolver CLI gains
`--services-overrides <path>`.
`cli.deploy.development.deps.resolve_run_after` threads the
in-container path of the overrides JSON into the resolver
invocation.
Per-round include planner (Phase 3)
-----------------------------------
`cli.deploy.development.inventory.plan_dev_inventory_matrix` now
takes `primary_apps` (just the user-named apps) instead of a
pre-resolved `include` list and returns a 4-tuple per round
`(round_index, inv_dir, round_variants, include_R)`. Per round it:
1. Computes `round_variants` clamped to each primary's variant
count.
2. Builds variant-merged services overrides for every role with a
`meta/variants.yml`, picking that role's variant for the round
(clamped) — so transitive deps with their own variants follow
the round.
3. Runs `CombinedResolver(services_overrides=...)` per primary and
unions the result with the primary apps in user order.
4. Extends `round_variants` with the discovered deps' clamped
variant indices so the deploy step's per-round purge logic stays
accurate.
`build_dev_inventory_matrix` keeps its name but signature changes:
it now takes `primary_apps` and an optional `include_filter` for
SERVICES_DISABLED-style consumer filters.
Init / deploy wiring (Phase 4-5)
--------------------------------
`init.py` and `deploy.py` consume the per-round `include_R`
directly, drop the old `resolve_deploy_ids_for_apps` pre-resolution
step, and apply the SERVICES_DISABLED provider filter on the
include set so the inventory `--include` flag and the final ansible
deploy ids stay in lockstep.
Behaviour invariance for non-variant roles
------------------------------------------
Apps with no `meta/variants.yml`, or without a `services.<X>.enabled`
override on any variant, behave identically to before:
* The override map for those roles is empty — resolver falls
through to the disk-only path.
* Per-round `include_R` collapses to the same set every round.
* Inventory bake outputs the same `applications.<app>.*` shape.
The behaviour change is scoped to roles that legitimately use
variants to flip topology. In the current repo only
`web-app-wordpress` is affected (variant 0 pulls
`web-app-discourse` in; variant 1 leaves it out, exercising the
multisite-without-discourse path).
Test surface
------------
* `tests/unit/cli/deploy/development/test_inventory.py` updated
for the new planner signature, mocking
`_resolve_round_include` / `_build_services_overrides_for_round`
so the planner's tuple shape is asserted independently of the
live resolver.
* `tests/unit/cli/deploy/development/test_matrix_deploy.py` adds a
`_entry()` helper that defaults `include` to the keys of
`round_variants`, so existing fixtures stay terse — round 0 of
the deploy loop deploys whatever is in `include_R`, and the
variant-aware planner naturally emits the same set.
* `tests/unit/utils/cache/test_data.py` pins the planner contract
(`primary_apps=`, 4-tuple plan entries with `include_R`).
* `tests/utils/__init__.py` added so the
`tests.lint.filesystem.test_init_files` lint test passes
end-to-end (the directory existed in the tree without an
__init__.py and tripped the lint).
Verification
------------
* `make test` green (162 tests, 1 skipped) on the merged tree.
* `make deploy-fresh-kept-apps APPS="web-app-wordpress"
FULL_CYCLE=true SERVICES_DISABLED="matomo,email"` end-to-end:
round 0 (variant 0, single-site + Discourse integration) and
round 1 (variant 1, multisite) each ran PASS 1 (sync) and PASS 2
(async), all four passes with `failed=0 rescued=0`. Round 0's
per-round include contained `web-app-discourse`; round 1's did
not — exactly the variant-driven topology the resolver is
supposed to produce.
The unrelated Joomla CLI install race (CI run 25079786137 `Deploy
web-app-joomla`) is parked separately as infinito-nexus/core#209;
its candidate fix is in that issue body, not in this commit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`cli.deploy.development.init` plans matrix rounds with the FULL include
set (`deploy_ids` = primary apps + transitive run_after deps); deploy
was planning with `primary_app_ids` (just what the user named). When
the only multi-variant role was a transitive dep, init created
`<dir>-0` / `<dir>-1` while deploy looked at `<dir>` and the inventory
file silently went missing. Fix passes `deploy_ids` to
`plan_dev_inventory_matrix` so both stages agree on the round count.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-up to d002796a5. The formula `infinito_nexus_${INFINITO_DISTRO}` is
now defined in exactly one place — `scripts/meta/env/defaults.sh:18`,
exported outside the load-once guard so it tracks INFINITO_DISTRO across
matrix iterations (caller re-sources defaults.sh after distro changes).
Every other site reads INFINITO_CONTAINER strictly from env; nothing
re-derives the formula, no lazy/colon-dash fallbacks, no per-call defaults.
Production code aligned:
- cli/deploy/development/common.py: add resolve_container() (strict env
reader, raises SystemExit on unset). make_compose() calls it as an
early-fail guard so a missing defaults.sh source surfaces here instead
of as a confusing compose.yml interpolation error later.
- cli/deploy/development/deploy.py: stop re-deriving from compose.distro;
use resolve_container().
- compose.yml: container_name reads ${INFINITO_CONTAINER:?...} strictly,
so callers that bypass defaults.sh fail fast with a clear hint.
- scripts/meta/resolve/image/local.sh: drop ${INFINITO_DISTRO:-debian}
fallback; require INFINITO_DISTRO via :? error.
- scripts/tests/deploy/ci/dedicated.sh: drop local _container fallback.
- scripts/tests/deploy/local/deploy/fresh-kept-app.sh: drop redundant
${INFINITO_DISTRO:-debian} reassignment after sourcing env/all.sh
(defaults.sh already defaults the value).
- scripts/tests/deploy/local/deploy/fresh-purged-app.sh: drop
target_container fallback; INFINITO_CONTAINER is already in env.
- scripts/tests/deploy/local/exec/container.sh: drop the lazy
${INFINITO_CONTAINER:=infinito_nexus_...} default; defaults.sh source
in env/all.sh already guarantees the value.
- scripts/tests/dns.sh: source defaults.sh so callers only need to set
INFINITO_DISTRO (the implicit dependency was previously a footgun).
- scripts/system/tls/trust/{wsl2,linux}.sh: same — source defaults.sh,
drop the ${INFINITO_CONTAINER:-infinito_nexus_debian} fallback in
linux.sh.
- .github/workflows/test-dns.yml: drop redundant
INFINITO_CONTAINER: infinito_nexus_${{ matrix.distro }} from per-step
env blocks (3 sites). The env now flows from INFINITO_DISTRO via the
defaults.sh sourced inside the scripts.
- tests/unit/cli/deploy/development/test_matrix_deploy.py: replace the
module-level os.environ.setdefault('INFINITO_CONTAINER', 'infinito_nexus_arch')
shim — which itself encoded the formula — with a module-level patch of
resolve_container that returns an opaque test-fixture string. Drop the
now-unused distro="arch" field from the _args() Namespace fixture
(deploy.handler stopped consuming it after the --distro arg removal).
Documentation aligned:
- docs/contributing/environment/variables.md: rewrite the INFINITO_CONTAINER
row to spell out the SPOT contract — set automatically by defaults.sh,
never re-derived elsewhere, callers re-source defaults.sh after
changing INFINITO_DISTRO mid-process.
- docs/contributing/artefact/files/compose.yml.md: replace the
INFINITO_DISTRO row (which described the now-removed
`infinito_nexus_${INFINITO_DISTRO:-debian}` interpolation) with an
INFINITO_CONTAINER row noting the strict ${INFINITO_CONTAINER:?...} read.
Verified:
- Formula derivation: 1 production hit (the SPOT in defaults.sh:18).
- INFINITO_CONTAINER fallback patterns (:-/:=): 0.
- --distro CLI arg refs: 0 (one comment explaining the removal).
- make test green: 1553 unit + 168 integration tests, no skipped suites.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related improvements anchored on the same principle: the env var
INFINITO_DISTRO is the single source of truth for the development distro,
and INFINITO_CONTAINER is purely derived from it.
1. Fix the matrix-deploy purge failure (CI run 24942172018, job 73038526513,
web-app-wordpress @ fedora):
scripts/meta/env/defaults.sh eager-exported INFINITO_CONTAINER once
inside its load-once guard; ci/all.sh's iteration loop re-exported
INFINITO_DISTRO per round but the cached INFINITO_CONTAINER stayed at
the initial-default infinito_nexus_debian, so _purge_app_entities()
between rounds tried to docker exec into a container from the wrong
distro and failed with "No such container".
Move the INFINITO_CONTAINER derivation OUTSIDE the load-once guard so
every re-source picks up a CONTAINER name in lock-step with the current
INFINITO_DISTRO. ci/all.sh now re-sources defaults.sh after each
export INFINITO_DISTRO=\$distro to trigger that re-derivation. defaults.sh
stays the single spot that owns the formula.
2. Retire the --distro CLI arg from cli.deploy.development {up,down,stop,
restart,init,deploy,exec,logs,build}. The arg duplicated INFINITO_DISTRO
(its default already read from the env), creating a hidden second source
of truth that drifted from the env across matrix iterations. Replace it
with cli.deploy.development.common.resolve_distro() which reads
INFINITO_DISTRO strictly (raises SystemExit on unset/invalid), and
simplify make_compose() to take no args. All bash callers
(scripts/tests/deploy/{ci,local}/**, Makefile restart target) drop the
redundant --distro \"\${INFINITO_DISTRO}\" pass-through.
Verified: make test passes (1553 unit + 168 integration). The mechanism of
the WP fix is verified by a synthetic distro-cycling test of defaults.sh
re-derivation; the original CI multi-distro repro can only be confirmed
end-to-end by the next CI push (local single-distro deploys do not exercise
the stale-env path).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Playwright E2E gate previously fired through `MODE_CI`, a boolean
that OR-ed three env-var markers (`GITHUB_ACTIONS`, `ACT`,
`INFINITO_MAKE_DEPLOY`) AND-ed with an `INFINITO_SKIP_E2E` opt-out.
That mechanism duplicated information `RUNTIME` already carries:
- `make deploy-*` already bakes `RUNTIME=dev` into the inventory at
init time (`RUNTIME_VARS_JSON='{"RUNTIME":"dev"}'`).
- CI runners (act / GitHub Actions) detect `RUNTIME=act|github` from
their canonical env markers via `cli.meta.runtime.detect_runtime`
during init.
- A bare `ansible-playbook` from a production shell stays at the
default `RUNTIME=host`.
Switching the gate to `RUNTIME in ['dev', 'act', 'github']` therefore
covers every case `MODE_CI` covered, AND folds the gate into the
single existing runtime SoT rather than maintaining a parallel one.
Changes:
- `tasks/stages/02_server.yml` E2E include now keys on `RUNTIME`.
- `group_vars/all/01_modes.yml` drops the `MODE_CI` block; a short
note explains the retirement.
- `scripts/meta/env/ci.sh` deleted (it only existed to export
`INFINITO_MAKE_DEPLOY=1`).
- The 5 `scripts/tests/deploy/local/deploy/*.sh` entry-points stop
sourcing `ci.sh`.
- `cli/deploy/development/deploy.py` no longer forwards the four
retired env markers into the container; the runtime decision is
already serialised into the inventory the deploy stage consumes.
- `scripts/tests/deploy/local/deploy/reuse-kept-app.sh` drops the
same four `-e` flags from its `docker exec`.
- `docs/requirements/006-playwright-service-gated-tests.md` gets a
prepended "MODE_CI retired" update note; the historical sections
remain for context but the dead `ci.sh` link target was reduced to
inline code.
Behaviour: any runtime where E2E was fired before still fires it now,
and the previous opt-outs have a single replacement: pass
`RUNTIME=host` (or `--vars '{"RUNTIME":"host"}'`) to skip Playwright on
a deploy that would otherwise have run it.
Introduces per-role `meta/variants.yml` for declaring multi-variant
deploys (e.g. WordPress Single-Site vs. Multisite) and reshapes the
development deploy CLI to materialise one inventory folder per variant
and iterate them with co-located sync+async passes.
Why:
- The previous single-inventory + Pass-1-then-Pass-2 split deployed each
variant twice and ran the async pass against a host left in a
different variant's state, masking real bugs.
- Inventory and runtime concerns were tangled: variant selection lived
as a runtime extra-var and required cross-stage coordination.
How:
- `roles/<role>/meta/variants.yml` (file format docs at
docs/contributing/artefact/files/role/variants.md): YAML list of
deep-merge overrides on top of `config/main.yml`. Empty entry `{}` is
variant 0 (= legacy payload). 6 roles get their dev overrides moved
here from inventories/dev.yml; WordPress declares 2 variants.
- `utils/runtime_data.py` gains `_build_variants` / `get_variants` and
drops the retired `_active_variants` runtime selector. Loader always
returns variant 0 as the default; non-zero variants reach
`get_merged_applications` as inventory-level `applications.<app>`
overrides baked at init time.
- New `cli/deploy/development/inventory.py`: `DevInventorySpec`,
`plan_dev_inventory_matrix` (pure planner shared between init and
deploy), `build_dev_inventory_matrix` (materialises one folder per
round at `<dir>-<R>`, or `<dir>` when single-variant). Single-folder
paths preserve full backward compat.
- `cli/deploy/development/deploy.py`: matrix loop with per-round
cleanup (only apps whose variant changed AND that are about to
re-deploy), `--variant <idx>` (or `VARIANT=` env) to pin a single
round, and `--full-cycle` (or `FULL_CYCLE=true`) which interleaves
sync + async per round so the async re-deploy targets exactly the
host state the matching sync deploy just produced.
- `cli/deploy/development/exec.py` gains repeatable `--env KEY=VAL` so
bash callers can inject per-run context into in-container helpers
without inlining heredocs.
- `scripts/meta/env/inventory.sh` exports `INVENTORY_VARS_FILE`
(Python mirror in `cli.deploy.development.common.DEV_INVENTORY_VARS_FILE`;
test_inventory_vars_file_spot.py locks them against drift).
- `make deploy-fresh-purged-apps` and `scripts/tests/deploy/ci/dedicated.sh`
switch to a single init + single deploy `--full-cycle`. The
`make deploy-reuse-*` bash scripts honour `VARIANT=` by suffixing
`INVENTORY_DIR`/`INVENTORY_FILE`.
- The 5 `scripts/tests/deploy/local/deploy/*.sh` scripts now delegate
their in-container heredocs to dedicated helpers in
`scripts/tests/deploy/local/utils/`, removing the bash-in-bash
quoting hell.
Tests:
- `tests/unit/utils/test_variants.py` (renamed from
`test_application_variants.py`): loader edge cases, variant-zero
default, retired-`_active_variants` guard.
- `tests/unit/cli/deploy/development/test_inventory.py` (new): planner,
variant baking, builder, `filter_plan_to_variant`, `DevInventorySpec`.
- `tests/unit/cli/deploy/development/test_matrix_deploy.py` (new):
per-round deploy filter, cleanup decisions, `--variant` pinning,
`--full-cycle` per-round interleave.
- `tests/unit/cli/deploy/development/test_inventory_vars_file_spot.py`
(new): drift guard for the bash <-> Python literal pair.
- `tests/unit/cli/deploy/development/test_exec.py` extended for `--env`.
Docs:
- `docs/contributing/artefact/files/role/variants.md` (new): file format.
- `docs/contributing/design/matrix-deploy.md` (new): runtime mechanism
(folder layout, --variant, --full-cycle, cleanup logic).
- `docs/contributing/actions/deploy.md` (new): contributor dev-deploy
flow (make targets, --variant pinning).
- `docs/administration/deploy.md` slimmed to admin-only (was mixed
admin+dev), cross-links to the new contributor doc.
- `docs/agents/action/iteration/role.md` keeps only agent-iteration
recipes; mechanism explanation moved out and linked.
- `docs/contributing/documentation.md` adds the "no self-SPOT
declaration" rule; `applications.md` renamed its SoT header for
consistency.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements requirements 005, 006, and 007.
005: per-application Keycloak `group-ldap-mapper`s anchor each app's
role groups under `/roles/<application_id>/<role_name>` (and, for
tenant-aware apps, `/roles/<application_id>/<tenant>/<role_name>`).
The flat-CN LDAP layout keeps each CN globally unique so Keycloak's
group-tree resolver never collides; per-app mappers use the LDAP
`description` as the visible Keycloak group name to keep the path
free of redundant prefixes. Migration purges legacy pre-005 entries
and forces a Keycloak `logout-all` so stale tokens drop pre-005
group paths.
006: per-service `<SERVICE>_SERVICE_ENABLED` env flags and the
`service-gating.js` helper let role specs `skipUnlessServiceEnabled`
cleanly when a shared service is missing or `SERVICES_DISABLED` is
set; `MODE_CI` markers (`INFINITO_MAKE_DEPLOY`, `GITHUB_ACTIONS`,
`ACT`) propagate through the deploy CLI into the test stage.
007: WordPress->Discourse round-trip Playwright scenario gated on the
`web-app-discourse` deployment.
WordPress Playwright runs 5/5 green on the single-site flow; the
multisite + Discourse scenarios are correctly skipped when their
service prerequisites are absent. The `rbac_group_path` lookup is the
single sanctioned producer for OIDC group paths and is wired into
every consumer (yourls, lam, minio, openwebui, pgadmin, phpldapadmin,
phpmyadmin, prometheus, xwiki).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- deploy.py: use find_provider_roles() to map service names (e.g. 'matomo')
to provider app IDs (e.g. 'web-app-matomo') before filtering deploy_ids;
forward SERVICES_DISABLED into the container via extra_env
- container.sh: rename INSPECT_CMD to CMD for consistency; fix indentation
- Makefile: expose CMD variable on exec target for non-interactive use
- environment.sh: source scripts/meta/env/all.sh for INVENTORY_DIR; add
inventory verification after SERVICES_DISABLED deploy using make exec
- variables.md: document CMD variable
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Makes clear that multiple space- or comma-separated app ids are
supported. Renames the env var, CLI flag (--app → --apps), and all
references in scripts, workflows, and documentation.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Resolves deps for each app individually and merges the results,
enabling APP="web-app-taiga web-app-pixelfed make deploy-fresh-purged-app".
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Capture and print deploy stdout/stderr on failure (development/deploy.py)
- Forward INVENTORY_DIR into init and deploy handlers (development/run.py)
- Add missing when_down flag for up handler
- Print full traceback on orchestrator failure for CI visibility
https://chatgpt.com/share/6970fcf5-5860-800f-89ae-0d601405489b
- add scripts/inventory/resolve.sh to compute INVENTORY_DIR based on runtime context (act/github/local)
- resolve INVENTORY_DIR dynamically in Makefile via resolve.sh
- enforce INVENTORY_DIR usage across all local deploy/test scripts
- propagate INVENTORY_DIR into container and development CLI commands
- remove hardcoded /etc/inventories paths
https://chatgpt.com/share/69700d69-b270-800f-ad50-d9ff486378b9
