swarm-exec reaches a DinD node; reaching a swarm service container (moodle,
openldap) needed a nested docker exec. Add swarm-app-exec, which resolves one
running replica on the node and runs a base64-encoded command inside it, so
live in-container inspection stays on a sanctioned make helper. Adding a
second exec_* script tripped the prefix-group lint, so exec_node.sh moves to
exec/node.sh and the new one lands at exec/service.sh.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
07_assert_state.sh sources _context.sh under set -o pipefail, and the printf | head-n1 that picked the first NFS volume raised a broken-pipe write error the moment head closed the pipe after line one. Any role with two or more NFS volumes (erpnext ships erpnext_sites and erpnext_logs) crashed the reschedule assertion before it could check the surviving marker. Plain parameter expansion returns the same first line with no pipe.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The swarm deploy job now renders the same summary page as compose:
role runtimes, a swarm structure section (topology roles plus live node
list and service placement from the manager) and the Playwright table.
The matrix deploy tees its ansible output and prints the compose
matrix-deploy round/PASS markers, so the runtime parser segments swarm
runs per variant round too.
The runtime table unifies tasks and roles with a Type column (tasks
first), appends the recap total row and flags a mismatch between the
parsed sum and the recap total; rows are read from ROLES RECAP sections
only, which removes the double-counted TASKS RECAP task rows. Each role
row carries one column per host: executed, skipped or failed, with
ignored failures downgraded. The Playwright variant column drops the
redundant variant- prefix. Verified against the logs of runs
29656268771 (swarm) and 29620158441 (compose): every segment's parsed
sum matches the recap total within 0.05s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The context derived PRIMARY_NFS_VOLUME from the first declared volume, but gitlab's first volume is repositories with nfs: false (gitaly cannot run on NFS), so the seed step wrote its pre-drain marker into a state directory that never exists on the export and killed a job whose deploy had already converged. Volumes declaring nfs: false are now skipped, making the drain check target a volume that actually lives on the export (gitlab_shared).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The swarm convergence gate derived runner_runner-1 from the services
metadata and polled a stack service the role never creates: the runner
fleet deploys as node-local compose projects on every node because each
runner needs its own node's docker socket and identity. Declare
workload: node-local on the runner services entry and let the gate
context map it to HAS_SWARM_SERVICE=false, so the stack-service gates
skip themselves while the swarm deploy, the registry distribution chain
and the CLI e2e keep running. Roles without the attribute keep the
image/template heuristic; workload: stack forces the gates on. Document
the runner architecture for both deploy modes in the role README.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.
Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add DIR_BACKUPS to utils.paths and INFINITO_DIR_BACKUPS /
INFINITO_DIR_SECRETS env handlers, derive the recover-console constants
(including NFS_EXPORT_STATE) from utils.paths and the nfs export SPOT,
and switch every remaining self-built /var/lib/infinito subpath - the
secrets recover targets, the swarm extras backup source, the users
token-store candidate chain, the CI recover drill, the swarm NFS
diagnostics and the playwright reports dir - to those constants. Tests
consume the env constants directly; narration comments in the touched
files are dropped per the comment lint.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Node image now builds FROM the distro infinito image (python, ansible, CLI baked in) instead of installing a per-node .deb, so there is no per-node bootstrap or timeout; the swarm CI job depends on the distribution build. Move compose/swarm.yml to compose/swarm/compose.yml and relocate the swarm matrix orchestrator from cli.administration.deploy.swarm to utils.tests.swarm (test code out of the shipped CLI).
Back the NFS export root with a disk-backed named volume instead of a RUNNER_TEMP bind: local-act RUNNER_TEMP resolves onto tmpfs and ganesha FSAL_VFS cannot export a tmpfs, so it loaded zero exports and the controller mount got ENOENT. Add optional compose/swarm/cache.override.yml wiring swarm nodes to the package-cache frontend when present, leaving CI without a cache unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds make swarm-diagnostic (alias i8sadi): read-only probes across a
live swarm-test cluster's nodes via exec_node.sh - backup unit state +
recent journal, NFS mounts, D-state (wedged-NFS) processes, running
rsync/db-dump processes, and disk usage - so a hung backup unit can be
root-caused without hand-writing docker exec pipelines.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A wedged manager (act-swarm containerd stall) made the docker exec tar
block up to 900s; on a cancelled job that ate the runner's grace window
and the always() Cleanup step never ran, leaking the DiD cluster. Probe
the manager first (10s) and cap the copy at 120s with --kill-after so a
hung exec is force-reaped; teardown then always follows. Reports stay
collected before teardown, since they live inside the manager node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
swarm_stack_ready.sh piped 'docker stack services' through a process
substitution, so the timeout wrapper added in 8516e3167 turned a slow or
wedged manager into an empty listing, an empty not_running set and exit
0: the convergence gate would have gone green with zero running replicas.
Capture the listing, fail on a non-zero rc.
That exposed the second half: post_deploy gated the swarm wait on the
cluster-wide DEPLOYMENT_MODE, so forced-compose roles (matrix ansible
flavor, bigbluebutton) ran it against a stack that does not exist and
only passed because the old code swallowed the error. Gate it on
compose_mode, like the handlers already do.
teardown.sh: the NFS mountpoint and umount calls, the likeliest hang
sites, were the only ones left unwrapped, and the timeout around docker
inspect inverted the D-state warning (a hung inspect returned 124, the
if went false and the script reported a clean teardown). Warn on rc 0
and 124, and shrink the per-node budget so the script fits inside its
step timeout.
Swarm step budget: the bounded steps alone summed to 365 over a 360-min
cap, so cleanup was still starved. Deploy 240, diagnostics 15, playwright
15, rescue 25, cleanup 15.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
27 jobs of run 29147179243 burned their full 360-min budget. Three
systemic layers, all addressed here:
- The CLI e2e task ran the staged test.sh unbounded, and test.sh blocks
in 'systemctl start' on a oneshot backup unit with no JobTimeout; on a
wedged dockerd or a stalled NFS mount both freeze silently for hours.
The task now runs the script under 'timeout TEST_E2E_CLI_TIMEOUT'
(60 min via the timeout lookup, so TIMEOUT_FACTOR scales it) and the
unit starts are bounded by BKP_TEST_HEALTH_TIMEOUT.
- The converge probe re-ran the full per-service diagnostics (service ps,
service logs, inspect, journalctl) on EVERY retry, so 600 nominal
2-second attempts cost 25-33s each and silently outlived the step
timeout. The block is gone (the workflow-level container.py rescue
snapshot already collects it recursively on failure) and every docker
call is timeout-bounded. Same removal in the swarm collect script,
which keeps only the NFS and render evidence container.py does not see.
- Playwright-collect, rescue diagnostics and Cleanup had no
timeout-minutes, and teardown.sh calls docker against D-state nodes,
so a failed job hung there for hours until the job cap killed it
without artifacts. Every step is bounded now, every teardown call is
timeout-wrapped, and the deploy steps (swarm 270, compose 300) leave
room for diagnostics and cleanup below the 360-min cap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
routine/08_cleanup.sh becomes utils/clean/teardown.sh and topology/
becomes utils/topology/ so routine/ holds only the sequenced flow; all
consumers (workflow steps, compose/swarm.yml env guards, relative
sources, README) follow. The shell rescue collector and its lint are
gone: utils/diagnostics/container.py covers compose and swarm
identically. extend_inventory gains the backup-host placements the DR
drill deploys.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the per-node docker-run bring-up with a single node image
(compose/swarm/Dockerfile bakes python3/dnsmasq/openssh/rsync/cryptsetup)
declared in compose/swarm.yml, one compose up, and one ansible play
(compose/swarm/playbook.yml over ansible_connection: docker) that waits for
systemd, captures IPs, wires lab DNS and installs the .deb per node. Reshape
the harness into topology/ (naming SPOT), routine/ (01_bootstrap + the
sequenced 02-08 steps) and utils/ (clean/, collect/, shared helpers); the
workflow, matrix.py and Makefile follow the new paths. swarm init/join was
already ansible-owned (svc-swarm-node), corrected in the README diagram.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
