make roundtrip with no apps= now iterates every application, most-complex first (complexity --sort total --order desc --format string). Also marks roundtrip.sh executable and moves the roundtrip target into alphabetical order.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
make roundtrip apps="<app> ..." iterates each role through every deploy mode in order (compose, then swarm), stopping at the first failure. Per-step logs under TMPDIR; each validated swarm cluster is released unless keep=true. The mode sequence is overridable (modes="compose swarm") so kubernetes can be appended later.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every swarm-test cluster is named so parallel clusters get distinct container and network names. topology.sh and extend_inventory require SWARM_NAME; act-swarm-zombie defaults it to the app id (name= overrides), the CI workflow derives it from the matrix app, and act-swarm-down/shell/playwright require name=. The bare swarm-mgr-01 names are gone (node refs now carry the cluster prefix).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
down_act_outer: match the act outer runner by name prefix instead of a hardcoded image ancestor, so it works regardless of ACT_PLATFORM_IMAGE. 13_cleanup: warn loudly when a node survives 'docker rm -f' (kernel D-state / stuck NFS, reboot needed) instead of swallowing the error.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A killed prior act-swarm run leaves the nfs-server container behind, so 'docker run --name nfs-server' then conflicts on the next run. Remove any existing one first.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Thread a single `variant` selector (CSV) end-to-end so CI can split a
role's matrix deploy across runners.
- variant_select.py (SPOT, shared by init + deploy): one --variant taking
a CSV (2 pins one round, 0,1,2 is a runner bundle, empty = full matrix);
replaces the old separate --variant/--variants.
- planner.filter_plan_to_variants filters the plan to a round subset.
- CI discovery (utils/github/variant_bundles.py) splits roles with more
than INFINITO_VARIANT_BUNDLE_SIZE (default 3) variants into bundles of
consecutive indices, one matrix runner each (5 -> 0,1,2 + 3,4), threaded
via the variant env. test-deploy-{local,universal,workstation,server}.yml
switch to matrix.include with a comma-free variant_slug for artifact/job
names (GitHub expressions have no replace()).
- runtime_budget honours INFINITO_VARIANT_TIME_OVERFLOW (cut|fail) when a
runner cannot finish its slice within INFINITO_MAX_RUNTIME.
Unit tests + docs (variants.md, deploy.md) included.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The inner-loop rerunner copied only top-level *.js and personas/, never the role's addons/ (or fixtures/) tree, so addon-spec edits were invisible and every rerun silently used the stale deploy-time addon specs. Mirror the deploy-time stager so addon coverage reruns against the edited repo copy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Recent Docker tightened docker cp tar extraction, so act aborts at job setup when the stock catthehacker runner image's /var/run symlink is unpacked into. Add a make act-runner-image target that builds local/act-runner-fixed (stock image with /var/run removed); pass it via ACT_PLATFORM_IMAGE. Document the symptom + fix in iteration/workflow.md and cross-reference from playwright.md and README.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The swarm e2e now iterates one provision+deploy+e2e round per app variant,
mirroring the compose matrix deploy, so the keycloak TOTP env-gate is no
longer needed: the variant's services.keycloak.realm.totp_enabled=false is
baked into host_vars per round and resolves at deploy time in swarm too.
New cli/administration/deploy/swarm/matrix.py drives the rounds by reusing
the compose planner (plan_dev_inventory_matrix, _resolve_variant_payloads,
_bake_overrides): per round it provisions with the baked --vars overlay plus
--app-variants, extends the swarm topology, deploys (e2e in-deploy), waits
for convergence, and purges prior-round stacks between rounds via the new
purge_stacks.sh. The workflow collapses its per-step provision/deploy/verify
into one orchestrator step; chaos (seed/drain/assert) still runs once.
- cli/administration/deploy/swarm/matrix.py: NEW round orchestrator
- scripts/tests/deploy/swarm/purge_stacks.sh: NEW per-round swarm purge
- scripts/tests/deploy/swarm/05_provision_inventory.sh: per-round inv dir plus
conditional --vars / --app-variants (standalone path unchanged)
- .github/workflows/test-deploy-swarm.yml: single orchestrator step, timeout 360
- default.env, utils/env/handlers/passthrough.py: register INFINITO_VARS_PAYLOAD
plus INFINITO_APP_VARIANTS passthrough keys
- roles/web-app-keycloak/vars/main.yml: drop the ENVIRONMENT env-gate
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The node dnsmasq runs with no-resolv, so swarm container names (notably the swarm-mgr-01:5000 registry host) stopped resolving once the node resolver was pointed at it: workers could not pull custom-built images and rejected every task with "No such image". Route the node names to Docker's embedded DNS (127.0.0.11) in both the manager and worker dnsmasq configs, keeping the public upstreams and the domain override.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Workers resolve the app domain to the manager IP (openresty is mode:host +
manager-pinned), so sys-ctl-hlth-webserver can run on every node instead of
being gated to the stack host. The shared Root CA is minted once on the
stack host and fetched by each worker, so a worker's trust store matches the
manager's leaf signer and the healthcheck's TLS chain verifies.
- sys-ctl-hlth-webserver: drop the IS_STACK_HOST gate (reachability, not
data-locality, now that workers resolve the domain to the manager)
- sys-ca-selfsigned: mint the Root CA once on STACK_HOST (run_once +
delegate_to), extract the generation steps to 02_mint.yml; non-stack hosts
slurp the cert from STACK_HOST and write it to their host-local path
(cert only, never the key)
- svc-swarm-node: anchor the shared CA early on every swarm node so it
exists before any per-node consumer reaches it; bundle the swarm-only
tasks under one DEPLOYMENT_MODE block
- sys-front-tls-selfsigned: gate leaf signing to the stack host (workers
have no CA key and run no manager-pinned openresty)
- scripts/tests/deploy/swarm/02_start_swarm_nodes.sh: workers run dnsmasq
resolving the domain to MGR_IP; excluded from the 127.0.0.1 /etc/hosts
entries that would shadow it
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
With NFS shared storage the DB dependency runs as its own single-replica swarm service (<dep>_<dep>) instead of a stateful compose container, so 07_wait_converge inspects it via 'docker service ls/ps' (N/N replicas + Running) rather than 'docker inspect' on a container name.
09_seed_content now drains a WORKER node (never the manager, which pins the registry/DB/edge-proxy a rescheduled task depends on) and probes with 'curl -sS' instead of '-fsS': many app roots legitimately 404 (UI/API live elsewhere), so the reschedule check asserts the HTTP server responds, not that / is 2xx. 11_assert_state applies the same -sS reachability semantics.
Verified against swarm #20 (green): db(postgres) 1/1 Running -> converged; drain wrk-01 -> reschedule wrk-02; baserow reachable after reschedule.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror compose-playwright for the act-swarm DinD cluster: copy the working-tree's modified files into the node's frozen bootstrap copy (the nodes are not bind-mounted), then rerun via rerun-spec.sh. rerun-spec.sh gains an opt-in TEST_E2E_PLAYWRIGHT_NETWORK_HOST that switches it to --network host (mirroring the deploy-time swarm branch in 02_run_one.yml) so cross-app canonical domains are reachable on the node; the compose path is unchanged. Document the all-green-before-redeploy gate in playwright.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A hung NFS server refuses 'docker rm -f' and keeps its lab-network endpoint, blocking both the network removal and the fixed container name on the next run. Kill + force-disconnect from the lab network + remove each node before removing the network, so a wedged DinD NFS container no longer leaves the cluster un-redeployable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Makefile: `make act-swarm-zombie app=<id> disable=<keys>` now
forwards the `disable=` value into the act run via ACT_ENV. The
swarm provision CLI inside the workflow reads it through
`apply_services_disabled_from_env` and removes the named provider
roles from the test inventory + sets their `enabled: false /
shared: false` overrides in host_vars. Matches the iteration spec
in docs/agents/action/iteration/role.md.
* scripts/tests/deploy/act/workflow.sh: switch the ACT_ENV pair
separator from comma to semicolon so values can themselves carry
commas (the disable= list is comma-separated). Only one in-tree
caller (the Makefile target above) currently uses ACT_ENV with
multiple pairs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Sandboxed sessions leave behind devtmpfs bind-mounts under
${ACT_ACTION_CACHE_PATH}/<action>@<ver>/.claude/{hooks,settings.json,
settings.local.json,skills}/ and .mcp.json. Act tries to refresh the
cached action repo on the next run, can't `os.RemoveAll` the mounts
(EBUSY), and dies in `Set up job` with no actionable error.
Add a preflight that detects stale `devtmpfs` mounts under the active
cache path and aborts with exit 3 and a copy-pasteable
`sudo umount … && sudo rm -rf …` remediation BEFORE invoking act. Stays
silent when no stale mounts are present.
Confirmed firing in iter 2 of a sandboxed act-swarm-zombie run; silent
in every subsequent t-series run after the operator cleaned the cache.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Migrate every role's meta/volumes.yml from ad-hoc dict-of-named-volumes
to a canonical dict-of-dicts schema where the YAML key is the semantic
short name and the entry body carries type / name / source / mounts /
nfs / mode / read_only. The optional 'name:' field is the docker volume
name (defaults to the key); the previous 'docker_name:' field is gone.
What's in this commit (210 paths):
Infrastructure (23)
- New filter plugins/filter/{compose,container}_volumes.py — render the
top-level volumes:/configs:/secrets: section and per-service mount
blocks from the canonical meta. container_volumes uses the SEMANTIC
name as per-service mount source so it resolves against compose_volumes'
top-level dict key (docker_name as source produced 'undefined volume'
on swarm).
- New lookup plugins/lookup/{compose,container,volume}_volumes.py.
volume(application_id, name) returns a normalised single entry;
volume(application_id) returns the whole canonical dict.
- utils/cache/applications.py keeps a sibling registry
_CANONICAL_VOLUMES_BY_ROLE outside the merged applications payload so
raw Jinja source: strings never enter the templar walk (root-cause of
the 200x cache-miss + recursive get_merged_applications storm).
- utils/roles/applications/mounts.py — schema validator + normaliser;
list-shape is rejected.
- plugins/lookup/nginx.py: _volume_path_by_name reads canonical directly.
- scripts/tests/deploy/swarm/_context.sh: NFS_VOLUMES extraction handles
both legacy dict and canonical dict shapes.
- roles/svc-prx-openresty/handlers/restart_and_validate.yml: gate
compose-only validate+reload behind a single when-block; swarm uses
service-update instead (bind-mounts of a FILE pin source inode at
task-create time, so atomic write-tmp+rename leaves the running task
on a deleted inode).
- roles/sys-svc-compose/tasks/utils/nfs_prep.yml: switch to
lookup('volume', application_id) | dict2items so NFS subdir
pre-creation works under the new cache split.
Tests (12)
- New strict lints flagging inline mount sections in compose templates,
single-file bind mounts, and legacy lookup('config', ..., 'volumes.X.Y')
call sites — all volume-mount surfaces MUST flow through container_volumes
/ compose_volumes / volume lookups.
- Integration: test_volumes_compose_chain + test_volume_constants_are_lookups
accept both the legacy and the new lookup('volume', app, 'X').name forms
during the migration window.
- Unit tests for the new lookup, filter, mounts helper, plus updated
nginx lookup test.
- Prometheus role-wiring test reads canonical meta for the
alert_rules.yml bind mount.
Role migrations (175)
- 62 meta/volumes.yml converted to dict-of-dicts canonical shape
- 63 templates/compose*.yml.j2 replace inline volumes:/configs:/secrets:
with lookup('container_volumes', application_id, '<service>') |
indent(N) per service + a single trailing lookup('compose_volumes',
application_id) for the top-level block
- 50 vars/main.yml drop dead volume constants and refactor 23 remaining
legacy lookup('config', ..., 'volumes.X.Y') to lookup('volume',
application_id, 'X').Y across 9 roles
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds `act-swarm-zombie app=<id>` which runs the swarm matrix-app test and leaves
the DinD cluster + lab network alive afterwards (INFINITO_KEEP_SWARM_NODES=true
forwarded via the new ACT_ENV passthrough in workflow.sh) so failures can be
post-mortem inspected with `make act-swarm-exec node=… cmd=…`,
`make act-swarm-shell node=…`, and finally released with `make act-swarm-down`.
Consolidates the swarm-test topology (manager / worker / nfs-server names, lab
network, NFS export base) into scripts/tests/deploy/swarm/topology.sh as the
single source of truth: 13_cleanup.sh and shell_node.sh source it directly,
the GitHub Actions workflow ingests it into $GITHUB_ENV via a setup step,
replacing the duplicate env: block entries.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Resolves CI swarm rc=69 ("msmtp: needs a password"): controller-side
Ansible lookups (lookup('email').password etc.) couldn't see the
tokens.yml that Mailu writes target-side, because the controller has
its own filesystem in swarm mode.
Mount NFS-shared /var/lib/infinito on every swarm host AND on the
controller during deploy (in-play delegate_to: localhost task at end
of svc-storage-nfs-server role). State subdir exported separately
with no_root_squash + wildcard CIDRs so root-owned writes succeed
and the controller can mount from outside the svc-swarm IP set.
Single-writer guarantee on FILE_TOKENS via delegate_to: STACK_HOST +
run_once on sys-token-store/{01_core,write}.yml. Backup roles
(svc-bkp-container-2-local, sys-ctl-cln-bkps, sys-ctl-cln-faild-bkps,
user-backup/03_permissions_folders, three direct callers of
04_seed-database-to-backup.yml) gated on IS_STACK_HOST to prevent
N-host write races on the shared NFS backup tree.
Fail-fast assertion in svc-storage-nfs-client + 05_controller_mount
when pre-existing local /var/lib/infinito would be hidden by mount-
over. Operator must migrate or wipe before re-deploying.
SPOTs: SWARM_LAB_NETWORK in workflow env block, storage.nfs.state_subdir
in group_vars/all/15_storage.yml, DIR_VAR_LIB unchanged. UPPER_CASE
constants in NFS roles. Cleanup script umounts controller view before
container teardown to prevent stale-mount leak between matrix jobs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
12_collect_diagnostics.sh starts with `set +e` before sourcing
_context.sh, which immediately re-enables `set -euo pipefail` for the
remainder of the script. The first `docker exec ... mount | grep -i nfs`
that finds no nfs mount returned rc=1, pipefail propagated it, and
errexit aborted the script halfway through. GitHub Actions reported a
red tab on the diagnostic step next to the actual failing deploy step.
Move `set +e`/`set +o pipefail`/`set +u` after the source so the
post-failure helper continues through every probe even when individual
commands return non-zero, and the trailing `exit 0` still pins the
script's return code to success.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Loading default.yml via `-e @"${INFINITO_INVENTORY_VARS_FILE}"` (commit
a3c75bde7) made the dev vars available on workers but at extra-vars
precedence (level 22), which overrode the `set_fact` merge in
tasks/stages/01_constructor.yml that reconciles defaults_oidc with the
inventory-supplied OIDC. Workers and mgr-01 ended up with OIDC =
{CLIENT: {SECRET: ...}} only, so the templates that resolved
OIDC.CLIENT.LOGOUT_URL (sys-front-inj-logout etc.) aborted with "dict
has no attribute 'LOGOUT_URL'".
Mirror the provisioned host_vars/${MGR}.yml to host_vars/${WRK1}.yml and
host_vars/${WRK2}.yml at the end of 05_provision_inventory.sh instead.
Same default.yml source via the existing INFINITO_INVENTORY_VARS_FILE
SPOT, same content on every swarm host, but at host_vars precedence
(level 9) so the role-level merge wins on access. Removes the
now-redundant `-e` line from the workflow.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
12_collect_diagnostics.sh now ends with an explicit `exit 0` so the
diagnostic step never propagates a non-zero rc to GitHub Actions.
zip_swarm_for_upload.sh switches from `set -euo pipefail` to `set +e`
plus a trailing `exit 0` for the same reason. Keeps the workflow's red
tab on the actual failing step instead of doubling up on the post-mortem
collectors.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add `# nocheck: hardcoded-dns-resolver` on the two `server=` lines
inside the dnsmasq heredoc in 02_start_swarm_nodes.sh so the
test_no_hardcoded_dns_resolvers lint guard recognises the literals as
intentional (dnsmasq parses `#` as a comment, so the marker is
inert at runtime).
Re-stage three pure-formatting tightenings ansible-lint applied to
existing `template: # nocheck` lines (double space -> single space)
in sys-daemon, sys-svc-mail-msmtp and web-app-moodle.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The MODE_DEBUG-gated DinD DNS handler in sys-svc-container spawned a
busybox container with `--dns 1.1.1.1` and expected the mocked
infinito.example to resolve to 127.0.0.1, which Cloudflare obviously
cannot do. Run an in-cluster dnsmasq inside mgr-01 instead of patching
the symptom out:
- Stop and mask systemd-resolved so port 53 is free.
- Bootstrap /etc/resolv.conf with 1.1.1.1 so apt-get can fetch dnsmasq.
- Install dnsmasq with bind-dynamic, no-resolv, address=/${INFINITO_DOMAIN}/127.0.0.1
and 1.1.1.1/8.8.8.8 as upstream forwarders; bind-dynamic catches docker0
when the Ansible-installed dockerd creates it later.
- Switch /etc/resolv.conf to 127.0.0.1 and verify the chain via
`getent ahosts ${INFINITO_DOMAIN}` so the script fails fast on misconfig.
- Pin mgr-01's docker0 to 172.17.0.1/16 via daemon.json `bip`, then point
the swarm.yml networks.internet.dns at that gateway so DinD-spawned
containers reach dnsmasq via their default route.
Also re-stage three pure-formatting tightenings ansible-lint made to
existing `template: # nocheck` lines (double space -> single space).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous commit's default.yml load turned MODE_DEBUG (and therefore
MODE_ASSERT) on for the swarm CI, which activates the DNS check in
tasks/stages/01_constructor.yml:12 that resolves
test.{{ DOMAIN_PRIMARY }}. The pipeline failed on all four hosts
(mgr-01, wrk-01, wrk-02, nfs-server) because:
1. write_hosts_entries did not emit test.<domain> at all.
2. 02_start_swarm_nodes wrote /etc/hosts only on the manager.
Add test.<domain> to the emitted set and write the resulting block to
mgr + both workers + nfs-server so the assertion succeeds everywhere
the play targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add 08_verify_reachable.sh between wait_converge (07) and seed_content
(09, renumbered from 08): TCP-connect via the manager's local
routing-mesh entry confirms incoming traffic reaches a live task. For
multi-replica services, additionally probe each node that hosts a
Running task to verify the mesh works locally on every task-bearing
node.
Skips services without published ports (tag-only roles, internal-only
services). Logs replica count and port list for debugging.
Renumber 08-12 -> 09-13 to keep the sequence contiguous.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
svc-cache-registry -> svc-registry-cache
svc-docker-registry -> svc-registry-docker
svc-docker-swarm -> svc-swarm
svc-docker-swarm-manager -> svc-swarm-manager
categories.yml: drop svc.docker, add svc.registry. Deliberately no svc.swarm
category so get_entity_name('svc-swarm') stays 'swarm' instead of collapsing
to '' (which would break discover_role_services).
Entity keys updated to match the new svc.registry prefix:
services.cache-registry.* -> services.cache.*
services.registry.* -> services.docker.*
01_constructor: replace svc-docker bootstrap-loop entry with explicit
include_role of svc-swarm + bootstrap loop over [svc-registry, svc-storage].
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
inventories/development/swarm.yml carries only swarm-mode-specific
overrides and omits cross-cutting cred seeds (OIDC.CLIENT.SECRET,
biber, administrator password). Until now the swarm CI provision step
did not pass --vars-file, so default.yml was never merged into the
generated host_vars; the first deploy that preloaded web-app-mailu as
a shared email service then hit object of type 'dict' has no attribute
'SECRET' when sys-svc-compose rendered roles/web-app-mailu/templates/
env.j2. Affected jobs in CI run 27006276488: web-app-chess,
web-app-espocrm, web-app-funkwhale, web-app-prometheus.
Match the server-CI pattern (scripts/tests/deploy/local/reset/
inventory.sh and apps/initialize/all.sh both pass --vars-file
${INFINITO_INVENTORY_VARS_FILE}). 05_provision_inventory.sh now does
the same: the env loader sourced via _context.sh already exports
INFINITO_INVENTORY_VARS_FILE in the script shell, so the path
resolves through the SPOT.
write_extras.py re-reads default.yml users dict and patches the
runtime SSH key onto admin authorized_keys before emitting the
extras file. Without this, the deploy-time -e @swarm-nfs-extras.yml
would REPLACE the host_vars users entry (ansible later-wins on
top-level dicts) and wipe biber + the admin password the
--vars-file just baked in.
Two distinct breakages that let the workflow fail before ever reaching
the ansible deploy:
* `python3 -m utils.tests.swarm.*` (and friends) ran with the
actions-setup interpreter that has no project deps, so the imports
of `plugins.filter.canonical_domains_map` blew up on `from ansible.errors`.
The swarm preamble scripts now source `scripts/meta/env/load.sh` so
PATH picks up the venv `make install` populated, and the workflow itself
uses `install-force` (under `act --bind` the build/install.stamp is the
host's and would short-circuit the in-container install) plus pushes
`\${VENV}/bin` onto GITHUB_PATH for downstream steps.
* `ansible-playbook -e @file` replaces top-level dicts rather than
deep-merging them. The runtime-only extras file was dropping
`storage.nfs.{export_base,version}`, `storage.backend`, and
`swarm.{registry.port,network}` that the static
`inventories/development/swarm.yml` ships. Re-state those keys
in write_extras.py so the merge ends up with everything the roles need.
The script's three lines were redundant: \`pip install -e .\` and the
ansible-galaxy collection step are already covered by \`make install\`
(now landed in the swarm matrix preamble), and \`PYTHON=python3 bash
scripts/setup.sh\` is exactly what \`make setup\` invokes. Replace the
step with a direct \`make setup\` call.
Renumber the remaining scripts to keep the sequence lückenlos:
06_provision_inventory.sh -> 05_provision_inventory.sh
07_debug_inventory.sh -> 06_debug_inventory.sh
08_wait_converge.sh -> 07_wait_converge.sh
09_seed_content.sh -> 08_seed_content.sh
10_drain_worker.sh -> 09_drain_worker.sh
11_assert_state.sh -> 10_assert_state.sh
12_collect_diagnostics.sh -> 11_collect_diagnostics.sh
13_cleanup.sh -> 12_cleanup.sh
Workflow YAML updated to the new names. No script content changed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New ``cli.administration.deploy.swarm`` package wraps the strict
``cli.administration.deploy.dedicated`` runner with a layered closure
so swarm deploys never require operators (or test scripts) to
enumerate every infra role:
effective --id = seed (operator --id, else inventory_role_groups)
∪ dep-walk(seed via applications_if_group_and_all_deps)
∪ default-placement safety net
The dep-walker is the same SPOT
(``utils.roles.applications.in_group_deps.applications_if_group_and_all_deps``)
the runtime constructor consumes via ``lookup('applications_current_play')``,
so the CLI-side closure and the in-play role set stay in sync.
The dedicated CLI keeps strict explicit-only semantics: ``main`` was
split into ``parse_args`` + ``run_from_args`` so the swarm wrapper
can share parser, mode wiring, validator and runner verbatim and
only mutate ``args.id``.
Meta-side: ``svc-db-postgres`` gains ``default_placement: manager``
(semantic symmetry with mariadb). ``default_placement`` is now a
PLACEMENT hint only ("if deployed, run on the manager"); deploy
decisions come from the dep-walk. Apps that declare ``mariadb:``
pull svc-db-mariadb; apps that declare ``postgres:`` pull
svc-db-postgres; apps that declare neither get no central DB.
Shared utils extracted so the four consumers stay aligned:
* ``utils.inventory.groups.inventory_has_group``: YAML + INI static
inventory group probe (extracted from ValidDeployId).
* ``utils.roles.meta_lookup.{get_role_default_placement,
iter_roles_with_default_placement}``: single source of truth for
``default_placement`` discovery (lookup plugin is now a thin wrapper).
Test pipeline (``scripts/tests/deploy/swarm/*``):
* Heredoc-Python shells converted to proper modules under
``utils.tests.swarm.*`` and invoked via ``python -m``:
- extend_inventory.py (was 08_extend_inventory.sh)
- write_extras.py (was 09_write_extras.sh dynamic half)
- derive_includes.py (replaces 06's hardcoded includes array)
- write_hosts_entries.py (replaces 02's hardcoded /etc/hosts heredoc;
derives every canonical + www-redirect domain from APP_ID via
canonical_domains_map + generate_all_domains, so the test
works for any app in the matrix without per-app maintenance)
* 01_create_network.sh deleted (single docker command inlined in workflow).
* 11_deploy.sh deleted (swarm CLI now called directly in workflow).
* 09_write_extras.sh: static half moved to inventories/development/swarm.yml.
* Remaining shell scripts renumbered 01-13 without gaps.
* 08_wait_converge.sh: case block on DB_DEP replaced with single
conditional (container name == DB_DEP).
* 06_provision_inventory.sh: hardcoded includes array replaced with
`mapfile -t includes < <(python -m utils.tests.swarm.derive_includes)`.
* Workflow YAML step references updated to new script names + module
invocations.
New tests (39 covering the closure surface):
* tests/unit/cli/administration/deploy/swarm/test_closure.py (14)
* tests/unit/utils/inventory/test_groups.py (11)
* tests/unit/utils/roles/test_meta_lookup.py (+3 for default_placement)
* tests/unit/utils/roles/validation/test_deploy_id.py (unchanged
8 tests still pass after refactor onto the shared util).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drop the required_by declaration on svc-cache-registry so the
post-deploy validator no longer enforces its execution for every
svc-docker / web-app deploy. The pull-through cache is a performance
optimization, not a correctness requirement; operators that omit it
from inventory should not see a deploy fail.
Compose-mode auto-place now scopes svc-prx-openresty to plays that
actually carry a web-* role (any group whose name matches ^web-),
so svc-only deploys do not get an orphan front-proxy. The match is
intentionally on the web-* prefix family so new web-<sub>-<entity>
categories pick up the auto-place rule without an allowlist edit.
The swarm-NFS test still wants the cache for the pull-through speedup,
so svc-cache-registry is listed explicitly in 07_provision_inventory.sh
alongside the swarm bootstrap roles.
Fixes test-deploy-server / Server web-app-mediawiki failing with
"required role(s) did not execute for deploy: svc-cache-registry".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add tests/lint/ansible/jinja/test_no_ansible_facts_in_templates.py: under
swarm-mode the rendered template lands once on the primary manager and is
slurp-distributed to every other node, so any per-host fact reference
(ansible_hostname, ansible_default_ipv4, ansible_fqdn, ansible_nodename,
ansible_machine_id, ansible_eth0, ansible_all_ipv*_addresses) would
silently produce wrong output on the other nodes. The lint blocks them
at PR time.
scripts/tests/deploy/act/workflow.sh forwards --action-offline-mode +
--action-cache-path so repeated local act runs do not refetch the actions
catalog.
roles/sys-ctl-cln-anon-volumes/tasks/01_core.yml quoting cleanup applied
by autoformat (single to double quotes, no semantic change).
.claude/settings.json adds Workflow to permissions.allow so the
multi-agent Workflow tool no longer prompts per invocation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
DinD DNS chain reworked so the Playwright container run --rm npm install
no longer EAI_AGAINs on registry.npmjs.org:
- 03_start_swarm_nodes.sh runs each DinD with --dns 1.1.1.1/8.8.8.8 and
pre-seeds /etc/docker/daemon.json. systemd-resolved still clobbers the
outer resolv.conf at boot, so the daemon-level DNS is the only thing
that survives for containers the inner daemon spawns.
- daemon.json.j2 omits the "dns" key when networks.internet.dns is empty
(default), so the rendered config no longer overwrites the pre-seeded
daemon.json with "dns": [""] and break inner-container resolution.
- 09_write_extras.sh sets networks.internet.dns to a public resolver for
the test rig and bakes RUNTIME via cli.meta.runtime (so e2e-playwright
actually runs in act/github instead of being skipped under "host").
07_provision_inventory.sh generates mirrors.yml via
cli.contributing.mirror.resolver and passes --mirror to the inventory
provisioner only when cli.meta.runtime == "github". act/local keeps
upstream pulls (no mirror infrastructure to point at).
12_wait_converge.sh and 15_assert_state.sh accept any N/N replica state
instead of hardcoded 1/1, so multi-host-replica deploys converge cleanly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Roles that declare default_placement: manager in meta/services.yml are now
auto-added to their own inventory group when the operator did not list them
explicitly. Both the runtime constructor add_host loop and the CLI
pre-flight inventory extender use the new roles_with_default_placement
lookup so swarm and CLI-validation stay consistent.
In compose-mode (no svc-docker-swarm-manager group), svc-prx-openresty
auto-joins the first play host. run_once + ansible_play_hosts | first
keeps that deterministic in multi-host compose deploys so the openresty
vhost view does not fragment.
The svc-cache-registry / svc-docker-registry entries previously hardcoded
in scripts/tests/deploy/swarm/08_extend_inventory.sh fall out of the new
dynamic loop; net inventory is unchanged.
ansible_play_hosts joins the magic-var allowlist in
test_variable_definitions so the constructor add_host passes integration
lint.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The install + update + admin maintenance commands all run as throwaway
container instances against the custom image, with the live service's
bind mounts attached. The previous container exec-against-static-name
pattern stops working in swarm where the container name is a
per-replica swarm task id and the task itself can be reaped by the
healthcheck mid-update.
container_name is gated under swarm because docker-compose's strict
validation rejects it once replicas > 1. Cross-host config replication
in 01_configuration.yml is removed entirely: the play now runs natively
on every host in the role's group, so the LocalSettings.php / oidc.php
/ debug.php templates render in place per host.
Test inventory adds the pilot role to all three simulated swarm hosts,
which is what flips DEPLOYMENT_MODE to swarm under the new
inventory-driven model.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New role that runs registry:2 in pull-through proxy mode on the swarm
manager. Worker nodes' daemon.json gets a registry-mirrors entry plus
the corresponding insecure-registries entry so plain-HTTP traffic to
the cache is permitted. The role's inventory group lists exactly one
host (manager) so the new inventory-driven topology model resolves
DEPLOYMENT_MODE to compose for the cache itself.
Within a single deploy run, the three swarm workers all pull the same
base images via the cache instead of hitting registry-1.docker.io
three separate times. Measured speedup on iter26 vs the no-cache iter21
baseline: pkgmgr/apt-install steps 2.4-2.8x faster, Docker image pulls
1.8-2.8x faster.
Cross-iter cache persistence is intentionally out of scope: act
recycles the DinD nodes between runs so the cache volume is wiped each
time. Cross-run reuse would require either bind-mounting the cache
from the host or pointing daemon.json at the dev compose stack's
registry-cache; both are tracked separately.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
iter11/iter12/iter14 each hung nfs-server's `apt-get update` in TCP
CLOSE_WAIT against apt-cache:3142 for 5-60 minutes, even with
Pipeline-Depth=0 + No-Keep-Alive + Retries=3 + Timeout=30. Pulling once
into the cache and fanning out to four nodes saves ~4 minutes of
bootstrap when it works, but every run that hit the hang lost 20+
minutes — net negative.
Remove the workflow cache restore step, the 02b_start_apt_cache.sh
launcher, the apt.conf.d/01-cacher write in 05_bootstrap_nodes.sh, and
the apt-cache cleanup in 17_cleanup.sh. Each node now talks directly
to the upstream mirrors. Slower-but-reliable beats faster-but-stuck.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
iter11 hung nfs-server's `apt-get update` for 16+ minutes in CLOSE_WAIT
against apt-cache:3142 even with Pipeline-Depth=0 — the cache closed
each TCP slot but apt's HTTP method held onto them via keep-alive and
never noticed the half-close.
Set `Acquire::http::No-Keep-Alive "true"` so every apt request opens a
fresh socket and tears it down on completion (no shared state to leak),
plus a 30 s `Acquire::http::Timeout` so any future stuck fetch errors
out fast enough for the Retries=3 backoff to actually kick in.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
In CI the workflow runs as the unprivileged `runner` user, so
`scripts/install/package.sh`'s `apt-get update && apt-get install
build-essential debhelper dpkg-dev rsync` aborted with `E: Could not open
lock file /var/lib/apt/lists/lock - open (13: Permission denied)` before
the .deb was ever produced. Local act runs as root inside
catthehacker/ubuntu so we never hit it.
Detect non-root in 05_bootstrap_nodes.sh and prepend `sudo -E` to the
build invocation. Inside the swarm-node containers package.sh still
runs as root and never goes through sudo (those exec calls are
unchanged).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Leaving the apt-cacher-ng proxy enabled past the .deb-deps install made the
later ansible-driven apt tasks (sys-svc-ssh, update, ...) hang on TCP
CLOSE_WAIT against the cache even with Pipeline-Depth=0: the connections
the proxy closed upstream were never picked up by ansible's apt method on
the next invocation. The cache only pays for itself during the parallel
4-node .deb-deps fetch anyway — afterwards the proxy is pure friction.
Remove /etc/apt/apt.conf.d/01-cacher at the end of provision_node so
every ansible apt task hits the upstream mirrors directly, undisturbed
by the cache's keep-alive semantics.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
