It runs the role like any member, but it drives the deploy rather than being
deployed to, so it keeps a local connection while the nodes keep the container
one.
With the axis off the role is left out of both inventories entirely rather
than deployed disabled, so that arm proves swarm works with no mesh logic
anywhere in the path.
The role only reached the three swarm nodes, which pull it as a dependency.
The NFS and backup hosts are spokes of the data mesh and got no placement at
all, so they never brought an interface up while the writer had already given
them keys and addresses.
That fails in a misleading place: the inventory describes the mesh correctly
and the hub holds peers for both, so the first symptom is the manager's gate
reporting peers unreachable rather than anything pointing at placement.
Observed in a live drill, where the role was included for swarm-mgr-01 and the
two workers only.
ruff wraps the ternary in _sandbox_workers (8ce66075b) and the yaml formatter
reflows db_key_base's description (e3d5ab287); both lines were written over the
limit. No behaviour changes.
autoformat runs as quality-high's first phase, so it corrects a line only on the
run AFTER the one that introduced it -- which is why git status is worth reading
after every gate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two Moodle#1 jobs of run 33445831564 -- tor/arch and clearnet/debian -- end with the
same line, 65 minutes apart:
FAILURE: no running moodle task on a worker node
The deploy is clean; PLAY RECAP reports failed=0 on all four hosts. Only the
post-deploy drain drill fails, and it fails because nothing is left to drain.
The chain, each link measured rather than assumed:
meta/variants.yml variant 1 turns openwebui/hermes/openclaw/flowise on;
variant 0 turns all four off
extend_inventory.py:88-89 svc-virt-kata in the closure put BOTH workers in the group
svc-virt-kata/00_core.yml:61 labels every group member kata-capable=true
(2 label tasks in each moodle#1 log, 0 in moodle#0)
deploy.yml.j2:18-23 non-kata services get node.labels.kata-capable != true
05_seed_content.sh:12-22 filters running tasks with grep -vx "${MGR}"
With both workers labelled, moodle and nginx have no eligible node but the stack
host. `docker service ps` at failure shows all three moodle_nginx replicas Running
on mgr-01. The drill then looks for a worker task and finds none.
Controls in the same run: moodle#0 carries no kata group and 0 label tasks, and its
drill was reachable; Nextcloud#10 (green, no kata) logs
"initially scheduled on: ...wrk-01" -> "Rescheduled ... to ...mgr-01" -> drill complete.
Exactly three of the run's 18 swarm inventories carry the kata group, and all three
are red.
b20a70646f widened _WORKERS[-1] to *_WORKERS and said so in its own message: "whether
the stack host holds up under it is what the next swarm run measures". The stack host
held up. What broke is the drill's precondition.
The discriminator is the app's own declaration, and it separates cleanly -- executed,
not reasoned:
web-app-moodle kata.enabled=None -> ('swarm-wrk-02',)
web-app-nextcloud kata.enabled=None -> ('swarm-wrk-02',)
web-app-openclaw kata.enabled=True -> ('swarm-wrk-01', 'swarm-wrk-02')
web-app-hermes kata.enabled=True -> ('swarm-wrk-01', 'swarm-wrk-02')
_WORKERS[-1:] restores exactly the pre-b20a70646f tuple for the non-sandboxed case, so
that commit's drain fix stays intact for the rounds it was written for.
svc-virt-kata's own assert (00_core.yml:80-88, labelled_nodes | length > 0) holds with
one labelled worker.
applications_for_round() is a thin public wrapper over the existing
_applications_for_active_variants(get_merged_applications(...)); extend_inventory would
otherwise have to import a neighbour module's private helper.
Three tests cover the branch, which had none: grep -c "kata\|_WORKERS" on
test_extend_inventory.py returned 0 before this.
Not covered: the full chain -- label, placement, drill -- has not run. One
`make act-swarm apps=web-app-moodle variant=1` closes that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Dedicating a single worker to the sandbox left the drain step with nowhere to
put the agent: draining the only labelled node failed with `no suitable node
(scheduling ...)`. Two labelled workers are also what the requirement asks for,
so the earlier single-worker choice traded one defect for another.
The cost is deliberate: the inverse placement constraint pushes every
non-sandbox service off both workers, so the stack host carries the rest of the
platform. That is the topology the requirement describes, and whether the stack
host holds up under it is what the next swarm run measures.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The inventory groups and the remote-2-local provider list are the same fact about which node holds a backup repository. extend_inventory now consumes backup_repos.repo_placements instead of restating the closure checks, so the two cannot drift apart again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A sandbox node runs gVisor as its daemon-wide default runtime and every
container scheduled there inherits it, so keycloak failed to start on one.
Swarm has no taints, which leaves the inverse placement constraint as the only
lever: sandbox consumers are pinned to the labelled nodes, everything else is
kept off them. Both halves were measured on a live cluster first, because a
docker constraint has no presence test and rejects an empty comparison value:
`!= true` does match a node without the label, and three replicas under it
avoided two labelled workers entirely.
That makes the set of labelled nodes load-bearing for the whole platform, and
it was decided by accident. Without an `svc-virt-kata` inventory group the
service loader runs the role on every host, so every worker becomes a sandbox
node and the rest of the platform is evacuated onto the stack host. The role
now refuses to run in swarm mode until the operator names those nodes.
With the group present the opposite gap opened: the provisioner puts every role
group on the single `--host` it is given, so the role lands on the manager
alone, the label task skips itself there, and no node ends up labelled. The old
assert only proved that a worker exists, which says nothing about where the
role ran, so the deploy passed the check and then hung until the convergence
wait gave up. The assert now reads the labels back from the stack host.
`docker node ls --filter label=` matches engine labels, not swarm node labels;
the filter has to be `node.label`.
The constraint itself is emitted only where a sandbox tier exists, so a deploy
without an agent role is untouched by any of this, and the lab dedicates a
single worker: labelling both would leave the manager as the only home for
every other service.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`docker stack deploy` drops the per-service `runtime:` key, so a swarm worker
can only carry the sandbox as its daemon-wide `default-runtime`. That was never
set: the role reached the manager only, no node was labelled, and one of three
agent replicas executed model-generated code under plain runc.
Setting it from the base render is not possible either. The container role
writes `daemon.json` before the runtime binary exists, and dockerd refuses to
start when a runtime path does not resolve ("stat /usr/local/bin/runsc: no such
file or directory"). The role's own attempt to re-render afterwards was a silent
no-op, because `include_role: sys-svc-container` hits the
`run_once_sys_svc_container` guard.
Move the probes plus the render into `daemon_config.yml` and let the role
re-enter it via `tasks_from` after installing runsc. Only runtimes whose binary
the probe found are registered, and `default-runtime` is written only for a
present shim on a non-stack host, so the stack host keeps the shared kernel for
the platform's own services.
The placement constraint compares against a fixed literal because Docker
constraints have no presence test: label the node `kata-capable=true` rather
than with the runtime name, otherwise a Kata node fails a `== runsc`
comparison. Without a worker outside the stack host no node is eligible at all,
so refuse the deploy instead of leaving the service pending forever.
`select_runtime.yml` is gone: hermes and openclaw included it by absolute path
out of a foreign role, and one probe in the container role now serves every
consumer.
Verified live on a three node cluster: worker `DefaultRuntime=runsc`, both
workers labelled, manager unlabelled, service constrained, and the running
replica reports `runtime=runsc`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
extend_inventory appended the volume, secrets and nfs backup groups to the test inventory unconditionally, so the DR-drill ran on hosts whose roles declare no backup. For roles whose only volume is backup-disabled (registries, unbound, redis, openresty) the volume drill produced a payload-less generation and failed its non-empty assertion. Each leg now follows the closure of the role that induces it: volume and (transitively, via volume-2-local's secrets_backup consumer) secrets on the app closure, nfs on svc-storage-nfs-server's closure via its nfs_backup consumer. Enumeration confirms no consumer-active role loses its leg; a README documents the group derivation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The backup node no longer rides in the cluster inventory: extend_inventory
writes a sibling backup.yml (same host_vars dir) holding only the
svc-bkp-remote-2-local and svc-bkp-local-2-device groups, and the matrix
deploys it as a second play after the cluster converges, right before the
DR drill. Without a svc-swarm-node group that play resolves
DEPLOYMENT_MODE=compose and IS_STACK_HOST=true on the backup host, so the
backup roles and their alarm/cleanup units (cln-faild-bkps, alm-compose)
deploy without cluster-membership special cases; the interim group-gate
on sys-ctl-cln-faild-bkps is gone again. A stack host that should also
pull backups joins the bkp groups of its own cluster inventory instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
routine/08_cleanup.sh becomes utils/clean/teardown.sh and topology/
becomes utils/topology/ so routine/ holds only the sequenced flow; all
consumers (workflow steps, compose/swarm.yml env guards, relative
sources, README) follow. The shell rescue collector and its lint are
gone: utils/diagnostics/container.py covers compose and swarm
identically. extend_inventory gains the backup-host placements the DR
drill deploys.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
extend_inventory maps svc-bkp-remote-2-local + svc-bkp-local-2-device onto
the backup host and write_extras configures them (backup_providers =
manager + NFS server IPs; device mount/target/source), so the matrix
deploy installs the real systemd units there. The drill now exercises
them: 01_ssh_trust.sh only installs the pull identity, the remote-2-local
unit pulls every provider (replacing both manual pull calls), and after
02_luks_device.sh simulates the USB plug the local-2-device unit performs
the device sync; the device recover passes the configured target subpath
via --device-target. Unit absence on the backup host is now a hard
failure since the roles deploy with the round.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add routine/backup/: base.sh drives a marker seeded into the live NFS
volume and the manager secrets through the whole chain (volume+secrets+nfs
backup units -> remote-2-local pull -> local-2-device LUKS 'USB') and back
via every recover.py (device -> local root -> nfs export, docker volume and
host secrets), asserting each marker survives. Each per-host step is one
in-node routine run over docker exec. extend_inventory maps the volume/
secrets/nfs backup roles onto their hosts so the units install; write_extras
generates the pull keypair; the INFINITO_SWARM_BACKUP_* keys are added to
default.env + the passthrough handler.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rename the meta/services.yml key across all roles, the roles_with_default_placement lookup (now roles_with_placement), the get_role_default_placement / iter_roles_with_default_placement helpers, and every consumer (swarm closure, compose_volumes filter, nfs_prep, container deploy template, constructor stage, extend_inventory) plus their tests and docs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
derive_includes applies the same shared:true flip as force_shared_db, so
variant rounds that pin a consumer's DB to shared:false keep the central
provider (and its generated credentials) in the include set the deploy
will actually load. extend_inventory derives its manager-placed groups
from the same variant-aware closure instead of a base-config walk, which
re-added pruned provider groups without credentials. matrix.py passes the
round's variant map to the extend step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Conflicts resolved:
- .claude/.gitignore: union of ignore patterns
- plugins/lookup/rbac_group_path.py: drop both unused conflicting imports
(body fetches apps via lookup_loader)
- roles/sys-svc-compose/handlers/main.yml: keep HEAD's refactor (compose-up
handlers live in compose.yml, included per DEPLOYMENT_MODE)
- utils/cache/base.py: keep HEAD's merge_with_defaults/FILE_TOKENS imports
Post-merge fixes for main's newly-enabled lints:
- pyproject.toml: grandfather 5 files under the BLE001 ratchet
- justify 6 pre-existing security # noqa markers (S108/S311/S701)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Formatter-only changes from make autoformat: wrap a long Jinja
expression in the dashboard manager-ops task and reformat the swarm
extend_inventory test util. No behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
extend_inventory._host_topology placed the app-under-test on the workers whenever the role had no default_placement: manager. When APP_ID is svc-swarm-manager itself, that leaked the workers into the svc-swarm-manager group, which is the marker group IS_STACK_HOST keys on, so IS_STACK_HOST went truthy on the workers and sys-ca-selfsigned skipped its CA fetch (when: not IS_STACK_HOST). root-ca.crt never landed and 'CA | enforce permissions' failed on both workers. Guard the worker extension so this marker group stays manager-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Hoist the swarm-test node, network and NFS-export-base names out of the shell
harness (00_topology.sh) into default.env (INFINITO_SWARM_*), so the shell
harness and the Python harness (extend_inventory, write_extras) read one source
instead of each hardcoding the suffixes. 00_topology.sh, extend_inventory.py,
write_extras.py and 01/09/11 now read the SPOT; passthrough.py exports the keys
into the generated .env. 00_topology.sh reads default.env directly (not .env)
because the workflow sources it before make dotenv runs.
Rename topology.sh -> 00_topology.sh (sorts ahead of 01_/02_, the order it is
sourced in) and update all 10 references.
Add scripts/tests/deploy/swarm/clean_all.sh + make act-swarm-clean to reclaim
wedged act-swarm state between runs, label-scoped via INFINITO_SWARM_TEST_LABEL
so it never touches unrelated containers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every swarm-test cluster is named so parallel clusters get distinct container and network names. topology.sh and extend_inventory require SWARM_NAME; act-swarm-zombie defaults it to the app id (name= overrides), the CI workflow derives it from the matrix app, and act-swarm-down/shell/playwright require name=. The bare swarm-mgr-01 names are gone (node refs now carry the cluster prefix).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Tier-1 changes verified live in act-12 (mariadb stack converged, postgres
stack started, swarm-readiness gate fired correctly for docker/mariadb/
postgres/openresty).
* `plugins/filter/compose_volumes.py` + `plugins/lookup/compose_volumes.py`:
thread `dir_var_lib` through filter (required keyword, no default)
and lookup (reads from `vars_["DIR_VAR_LIB"]`, raises if missing). Replace
native NFS driver with a bind-mount from the host's already-NFS-mounted
`DIR_VAR_LIB` because native `type: nfs` re-handshakes portmap, which is
unreachable in nested DinD. Drop the FilterModule wrapper so the deprecated
pipe form keeps failing fast per the lookup-only SPOT contract.
* `tests/unit/plugins/{filter,lookup}/test_compose_volumes.py`:
thread `DIR_VAR_LIB` through every fixture (no default-fallbacks).
* `roles/sys-svc-compose/tasks/utils/post_deploy.yml`:
compose-mode db-health probe gated to `DEPLOYMENT_MODE \!= 'swarm'`; new
swarm-mode convergence gate runs the extracted `swarm_stack_ready.sh`
via `ansible.builtin.script` with `STACK` env, addressed by absolute
`playbook_dir`-rooted path so the include resolves from caller roles.
* `roles/sys-svc-compose/files/swarm_stack_ready.sh` (new): probe
`docker stack services` for a single `STACK` and exit non-zero until
every service's running/desired replica count matches.
* `roles/svc-db-mariadb/vars/main.yml`: new `MARIADB_CONTAINER_ADDRESS`
constant wraps the `container_address` lookup so callers reference one
SPOT instead of duplicating the lookup.
* `roles/svc-db-mariadb/tasks/01_core.yml` + `02_init.yml`:
switch root-password probes and bootstrap SQL calls to
`container exec -i {{ MARIADB_CONTAINER_ADDRESS }} sh -c 'mariadb -p"$MARIADB_ROOT_PASSWORD"'`.
Using the container's own `MARIADB_ROOT_PASSWORD` env var mirrors the
healthcheck and matches whatever quoting Docker swarm preserves from
the env_file, sidestepping the compose-vs-swarm env_file quote-strip
divergence.
* `roles/svc-db-postgres/files/set_postgres_superuser_password.sh`:
`eval "container=$container_raw"` so the swarm-mode lookup payload
`"$(/usr/bin/resolve-container-id <stack> <svc>)"` is re-evaluated by
bash and resolves to the live task id at exec time.
* `roles/svc-db-postgres/tasks/01_core.yml`: rename arg to use the new
swarm-aware `POSTGRES_CONTAINER_ADDRESS`.
* `utils/tests/swarm/extend_inventory.py`: when the seed app's primary
service declares `default_placement: manager`, do not add the app to
the worker group, so the swarm test inventory matches the
single-replica placement-pinned semantics.
Adds a `swarm:` subcategory under `svc:` in roles/categories.yml so the
shared-cluster role is properly invokable, and renames the existing
`svc-swarm` role + inventory group to `svc-swarm-node` (with
`get_entity_name`-aligned primary key `node:` in meta/services.yml).
`svc-swarm-manager` stays as the manager-marker role, becoming invokable
via the same new category and re-keyed to `manager:`.
Consumers of the cluster-orchestration service (svc-registry-cache,
svc-registry-docker, svc-swarm-manager) now reference the service-key
`node:` (was `swarm:`), matching the renamed primary entry. The planner
empirically resolves `service_registry['node']` -> `svc-swarm-node` and
emits it into --include for stack-host deploys, so the CI failure on
`universal-svc-registry-docker` (`svc-swarm: not present in inventory`)
no longer reaches `ValidDeployId`.
Breaking change for operators: inventory groups named `svc-swarm:` must
be renamed to `svc-swarm-node:`. `DEPLOYMENT_MODE` reads
`groups['svc-swarm-node']`, so a stale `svc-swarm:` group silently falls
back to compose mode.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
svc-cache-registry -> svc-registry-cache
svc-docker-registry -> svc-registry-docker
svc-docker-swarm -> svc-swarm
svc-docker-swarm-manager -> svc-swarm-manager
categories.yml: drop svc.docker, add svc.registry. Deliberately no svc.swarm
category so get_entity_name('svc-swarm') stays 'swarm' instead of collapsing
to '' (which would break discover_role_services).
Entity keys updated to match the new svc.registry prefix:
services.cache-registry.* -> services.cache.*
services.registry.* -> services.docker.*
01_constructor: replace svc-docker bootstrap-loop entry with explicit
include_role of svc-swarm + bootstrap loop over [svc-registry, svc-storage].
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CI failures from run 26987751848:
* `cache-read` (3 hits): annotate the `Path.read_text()` calls in
utils/tests/swarm/{write_extras,extend_inventory}.py — every one
reads a file the same script just wrote, so the cache helper would
serve stale content. Per `# nocheck: cache-read` carve-out.
* `project-root-import`: annotate the `parents[6]` in the swarm
closure test loader. The file-path loader bypasses the package
__init__ to avoid pulling in the heavy dedicated CLI imports.
* No-unittest-tests: tests in test_closure.py inherited only via
`_BaseClosureCase` (which itself extends `unittest.TestCase`), but
the AST lint walks bases literally and does not follow indirect
inheritance. Switch each `Test*` class to multi-inherit
`(_BaseClosureCase, unittest.TestCase)` so the discovery check
recognises them as runnable.
* `hardcoded-dns-resolver`: the per-block head suppression in
inventories/development/swarm.yml was insufficient; the lint scans
per-line and needs the marker on the actual `dns: "1.1.1.1"` line.
Verified locally: 16 of the previously-failing tests now pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New ``cli.administration.deploy.swarm`` package wraps the strict
``cli.administration.deploy.dedicated`` runner with a layered closure
so swarm deploys never require operators (or test scripts) to
enumerate every infra role:
effective --id = seed (operator --id, else inventory_role_groups)
∪ dep-walk(seed via applications_if_group_and_all_deps)
∪ default-placement safety net
The dep-walker is the same SPOT
(``utils.roles.applications.in_group_deps.applications_if_group_and_all_deps``)
the runtime constructor consumes via ``lookup('applications_current_play')``,
so the CLI-side closure and the in-play role set stay in sync.
The dedicated CLI keeps strict explicit-only semantics: ``main`` was
split into ``parse_args`` + ``run_from_args`` so the swarm wrapper
can share parser, mode wiring, validator and runner verbatim and
only mutate ``args.id``.
Meta-side: ``svc-db-postgres`` gains ``default_placement: manager``
(semantic symmetry with mariadb). ``default_placement`` is now a
PLACEMENT hint only ("if deployed, run on the manager"); deploy
decisions come from the dep-walk. Apps that declare ``mariadb:``
pull svc-db-mariadb; apps that declare ``postgres:`` pull
svc-db-postgres; apps that declare neither get no central DB.
Shared utils extracted so the four consumers stay aligned:
* ``utils.inventory.groups.inventory_has_group``: YAML + INI static
inventory group probe (extracted from ValidDeployId).
* ``utils.roles.meta_lookup.{get_role_default_placement,
iter_roles_with_default_placement}``: single source of truth for
``default_placement`` discovery (lookup plugin is now a thin wrapper).
Test pipeline (``scripts/tests/deploy/swarm/*``):
* Heredoc-Python shells converted to proper modules under
``utils.tests.swarm.*`` and invoked via ``python -m``:
- extend_inventory.py (was 08_extend_inventory.sh)
- write_extras.py (was 09_write_extras.sh dynamic half)
- derive_includes.py (replaces 06's hardcoded includes array)
- write_hosts_entries.py (replaces 02's hardcoded /etc/hosts heredoc;
derives every canonical + www-redirect domain from APP_ID via
canonical_domains_map + generate_all_domains, so the test
works for any app in the matrix without per-app maintenance)
* 01_create_network.sh deleted (single docker command inlined in workflow).
* 11_deploy.sh deleted (swarm CLI now called directly in workflow).
* 09_write_extras.sh: static half moved to inventories/development/swarm.yml.
* Remaining shell scripts renumbered 01-13 without gaps.
* 08_wait_converge.sh: case block on DB_DEP replaced with single
conditional (container name == DB_DEP).
* 06_provision_inventory.sh: hardcoded includes array replaced with
`mapfile -t includes < <(python -m utils.tests.swarm.derive_includes)`.
* Workflow YAML step references updated to new script names + module
invocations.
New tests (39 covering the closure surface):
* tests/unit/cli/administration/deploy/swarm/test_closure.py (14)
* tests/unit/utils/inventory/test_groups.py (11)
* tests/unit/utils/roles/test_meta_lookup.py (+3 for default_placement)
* tests/unit/utils/roles/validation/test_deploy_id.py (unchanged
8 tests still pass after refactor onto the shared util).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
