Node image now builds FROM the distro infinito image (python, ansible, CLI baked in) instead of installing a per-node .deb, so there is no per-node bootstrap or timeout; the swarm CI job depends on the distribution build. Move compose/swarm.yml to compose/swarm/compose.yml and relocate the swarm matrix orchestrator from cli.administration.deploy.swarm to utils.tests.swarm (test code out of the shipped CLI).
Back the NFS export root with a disk-backed named volume instead of a RUNNER_TEMP bind: local-act RUNNER_TEMP resolves onto tmpfs and ganesha FSAL_VFS cannot export a tmpfs, so it loaded zero exports and the controller mount got ENOENT. Add optional compose/swarm/cache.override.yml wiring swarm nodes to the package-cache frontend when present, leaving CI without a cache unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The backup node no longer rides in the cluster inventory: extend_inventory
writes a sibling backup.yml (same host_vars dir) holding only the
svc-bkp-remote-2-local and svc-bkp-local-2-device groups, and the matrix
deploys it as a second play after the cluster converges, right before the
DR drill. Without a svc-swarm-node group that play resolves
DEPLOYMENT_MODE=compose and IS_STACK_HOST=true on the backup host, so the
backup roles and their alarm/cleanup units (cln-faild-bkps, alm-compose)
deploy without cluster-membership special cases; the interim group-gate
on sys-ctl-cln-faild-bkps is gone again. A stack host that should also
pull backups joins the bkp groups of its own cluster inventory instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The DR-drill applications block in swarm-nfs-extras.yml reached the
deploy as extra-vars; -e replaces the whole inventory applications dict,
so every generated credential vanished and the first schema-checked
lookup (svc-db-mariadb credentials) killed all swarm deploys. The block
now enters through the provisioner's host_vars merge (_bake_overrides
base_overrides), the deploy loads the applications-free
swarm-nfs-extras.deploy.yml twin, and the full extras file stays for the
drill, which reads the device paths from it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the per-node docker-run bring-up with a single node image
(compose/swarm/Dockerfile bakes python3/dnsmasq/openssh/rsync/cryptsetup)
declared in compose/swarm.yml, one compose up, and one ansible play
(compose/swarm/playbook.yml over ansible_connection: docker) that waits for
systemd, captures IPs, wires lab DNS and installs the .deb per node. Reshape
the harness into topology/ (naming SPOT), routine/ (01_bootstrap + the
sequenced 02-08 steps) and utils/ (clean/, collect/, shared helpers); the
workflow, matrix.py and Makefile follow the new paths. swarm init/join was
already ansible-owned (svc-swarm-node), corrected in the README diagram.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rename the meta/services.yml key across all roles, the roles_with_default_placement lookup (now roles_with_placement), the get_role_default_placement / iter_roles_with_default_placement helpers, and every consumer (swarm closure, compose_volumes filter, nfs_prep, container deploy template, constructor stage, extend_inventory) plus their tests and docs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
derive_includes applies the same shared:true flip as force_shared_db, so
variant rounds that pin a consumer's DB to shared:false keep the central
provider (and its generated credentials) in the include set the deploy
will actually load. extend_inventory derives its manager-placed groups
from the same variant-aware closure instead of a base-config walk, which
re-added pruned provider groups without credentials. matrix.py passes the
round's variant map to the extend step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Embedded per-app DBs (shared:false) resolve to mariadb-on-NFS with no placement pin in swarm, which InnoDB does not support (locking, split-brain); embedded DBs are a compose-only pattern. After provisioning each swarm-test round's inventory, force shared:true on every svc-db-* provider service so consumers route to the central provider.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Conflicts resolved:
- .claude/.gitignore: union of ignore patterns
- plugins/lookup/rbac_group_path.py: drop both unused conflicting imports
(body fetches apps via lookup_loader)
- roles/sys-svc-compose/handlers/main.yml: keep HEAD's refactor (compose-up
handlers live in compose.yml, included per DEPLOYMENT_MODE)
- utils/cache/base.py: keep HEAD's merge_with_defaults/FILE_TOKENS imports
Post-merge fixes for main's newly-enabled lints:
- pyproject.toml: grandfather 5 files under the BLE001 ratchet
- justify 6 pre-existing security # noqa markers (S108/S311/S701)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The merged main renamed the inventory variant filter (filter_plan_to_variant -> filter_plan_to_variants) and added the shared variant_select selector. Adapt cli.administration.deploy.swarm.matrix to consume the variant CSV bundle via variant_select (add_variant_args + apply_variant_filter) instead of its bespoke single-int parser, so it iterates exactly the runner's bundle of rounds and the stale import no longer breaks.
Mirror test-deploy-compose.yml in the swarm workflow: pass the variant bundle to the orchestrator (re-passed through sudo so env_reset can't drop it), show the variant in the job name, and suffix the playwright artifact with variant_slug. Also singularise the discover job names in both workflows.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The swarm e2e now iterates one provision+deploy+e2e round per app variant,
mirroring the compose matrix deploy, so the keycloak TOTP env-gate is no
longer needed: the variant's services.keycloak.realm.totp_enabled=false is
baked into host_vars per round and resolves at deploy time in swarm too.
New cli/administration/deploy/swarm/matrix.py drives the rounds by reusing
the compose planner (plan_dev_inventory_matrix, _resolve_variant_payloads,
_bake_overrides): per round it provisions with the baked --vars overlay plus
--app-variants, extends the swarm topology, deploys (e2e in-deploy), waits
for convergence, and purges prior-round stacks between rounds via the new
purge_stacks.sh. The workflow collapses its per-step provision/deploy/verify
into one orchestrator step; chaos (seed/drain/assert) still runs once.
- cli/administration/deploy/swarm/matrix.py: NEW round orchestrator
- scripts/tests/deploy/swarm/purge_stacks.sh: NEW per-round swarm purge
- scripts/tests/deploy/swarm/05_provision_inventory.sh: per-round inv dir plus
conditional --vars / --app-variants (standalone path unchanged)
- .github/workflows/test-deploy-swarm.yml: single orchestrator step, timeout 360
- default.env, utils/env/handlers/passthrough.py: register INFINITO_VARS_PAYLOAD
plus INFINITO_APP_VARIANTS passthrough keys
- roles/web-app-keycloak/vars/main.yml: drop the ENVIRONMENT env-gate
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* cli/administration/deploy/swarm/closure.py: drop dep-walked roles
that are not present in the inventory's role groups so the
`validate_application_ids` strict in-inventory check no longer
collides with `disable=` (which removes provider roles from the
inventory). Operator-supplied seed IDs and the
`default_placement: manager` safety net are kept unchanged.
* tests/unit/cli/administration/deploy/swarm/test_closure.py: add
`test_operator_ids_dep_walked_not_in_inventory_dropped` to pin
the new behaviour and update the existing scenarios to put the
filtered roles in their test inventories.
* tests/lint/ansible/shell/test_container_exec_is_stack_host_gated.py:
recognise `include_tasks: { file: X, apply: { ... } }` dict form
alongside the free-form string. Without it the lint's include
graph dropped the apply-include edge and false-positived gated
files reached by it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
svc-cache-registry -> svc-registry-cache
svc-docker-registry -> svc-registry-docker
svc-docker-swarm -> svc-swarm
svc-docker-swarm-manager -> svc-swarm-manager
categories.yml: drop svc.docker, add svc.registry. Deliberately no svc.swarm
category so get_entity_name('svc-swarm') stays 'swarm' instead of collapsing
to '' (which would break discover_role_services).
Entity keys updated to match the new svc.registry prefix:
services.cache-registry.* -> services.cache.*
services.registry.* -> services.docker.*
01_constructor: replace svc-docker bootstrap-loop entry with explicit
include_role of svc-swarm + bootstrap loop over [svc-registry, svc-storage].
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New ``cli.administration.deploy.swarm`` package wraps the strict
``cli.administration.deploy.dedicated`` runner with a layered closure
so swarm deploys never require operators (or test scripts) to
enumerate every infra role:
effective --id = seed (operator --id, else inventory_role_groups)
∪ dep-walk(seed via applications_if_group_and_all_deps)
∪ default-placement safety net
The dep-walker is the same SPOT
(``utils.roles.applications.in_group_deps.applications_if_group_and_all_deps``)
the runtime constructor consumes via ``lookup('applications_current_play')``,
so the CLI-side closure and the in-play role set stay in sync.
The dedicated CLI keeps strict explicit-only semantics: ``main`` was
split into ``parse_args`` + ``run_from_args`` so the swarm wrapper
can share parser, mode wiring, validator and runner verbatim and
only mutate ``args.id``.
Meta-side: ``svc-db-postgres`` gains ``default_placement: manager``
(semantic symmetry with mariadb). ``default_placement`` is now a
PLACEMENT hint only ("if deployed, run on the manager"); deploy
decisions come from the dep-walk. Apps that declare ``mariadb:``
pull svc-db-mariadb; apps that declare ``postgres:`` pull
svc-db-postgres; apps that declare neither get no central DB.
Shared utils extracted so the four consumers stay aligned:
* ``utils.inventory.groups.inventory_has_group``: YAML + INI static
inventory group probe (extracted from ValidDeployId).
* ``utils.roles.meta_lookup.{get_role_default_placement,
iter_roles_with_default_placement}``: single source of truth for
``default_placement`` discovery (lookup plugin is now a thin wrapper).
Test pipeline (``scripts/tests/deploy/swarm/*``):
* Heredoc-Python shells converted to proper modules under
``utils.tests.swarm.*`` and invoked via ``python -m``:
- extend_inventory.py (was 08_extend_inventory.sh)
- write_extras.py (was 09_write_extras.sh dynamic half)
- derive_includes.py (replaces 06's hardcoded includes array)
- write_hosts_entries.py (replaces 02's hardcoded /etc/hosts heredoc;
derives every canonical + www-redirect domain from APP_ID via
canonical_domains_map + generate_all_domains, so the test
works for any app in the matrix without per-app maintenance)
* 01_create_network.sh deleted (single docker command inlined in workflow).
* 11_deploy.sh deleted (swarm CLI now called directly in workflow).
* 09_write_extras.sh: static half moved to inventories/development/swarm.yml.
* Remaining shell scripts renumbered 01-13 without gaps.
* 08_wait_converge.sh: case block on DB_DEP replaced with single
conditional (container name == DB_DEP).
* 06_provision_inventory.sh: hardcoded includes array replaced with
`mapfile -t includes < <(python -m utils.tests.swarm.derive_includes)`.
* Workflow YAML step references updated to new script names + module
invocations.
New tests (39 covering the closure surface):
* tests/unit/cli/administration/deploy/swarm/test_closure.py (14)
* tests/unit/utils/inventory/test_groups.py (11)
* tests/unit/utils/roles/test_meta_lookup.py (+3 for default_placement)
* tests/unit/utils/roles/validation/test_deploy_id.py (unchanged
8 tests still pass after refactor onto the shared util).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
