The discovery SPOT (cli.meta.ci.query) now runs one query shape per mode:
compose and host stay whole-role with variants packed into bundles, swarm
ranks and budget-cuts on per-variant rows and emits role#variant tokens
through apps.sh into variant_bundles, so the cut can pick a subset of a
role's variants and the --matrix view is the selection priority.
Replace the --unique flag with visible dna/clone columns (heaviest role
per dna cluster stays ahead of the cut), add the integrated column and
sort, count host bundles like compose, and give the deploy plan an id
counter, per-variant weights and display-width-aware alignment matching
the matrix renderer. Add i8ma*/i8pa* inspection aliases per mode.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dedicated.sh streams rescue diagnostics via docker exec tar since docker cp cannot see the systemd container's mount namespace and silently extracted nothing. recover_drill.sh guards on docker availability: non-stack role deploys now drill the nfs and secrets legs instead of hard-failing on the impossible volume leg; stack jobs still drill all three. 04_stack_rm_wait tolerates an already-removed stack and re-issues the rm once services are gone.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In compose/host mode the whole nested app stack lives inside INFINITO_CONTAINER, which dedicated.sh's EXIT-trap cleanup removes. The separate workflow-level rescue step ran container.py on the runner AFTER that teardown, so it saw an emptied Docker and captured only 12 host files (unlike swarm, whose DiD nodes outlive the deploy step and get recursed into).
Run the recursive container.py capture inside INFINITO_CONTAINER during cleanup, before the existing copy-out and the teardown, so it descends into the app containers. Drop the now-redundant, post-teardown 'Collect rescue diagnostics' step from the compose and host workflows (runner state stays covered by dump_state.sh; the upload step keeps uploading what dedicated.sh writes). container.py is now called once per mode: inside dedicated.sh for compose/host, at runner level for swarm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The drill entry gate treated 'deploys no swarm service' as 'has nothing
to restore', but the backup chain operates on NFS volumes and deployed
backup units, not on scheduler objects; bigbluebutton's seven
NFS-backed volumes went entirely unproven. Gate the drill solely on the
NFS-flagged volume and make only the full-disaster step
workload-aware: stack roles keep docker stack rm, node-local roles get
their compose project (label com.docker.compose.project=<entity>, the
compose handler's --project convention) stopped on every node so no
writer races the export wipe and restore. Declare
workload: node-local on bigbluebutton so the phantom-service
converge/chaos gates skip while play, e2e and now the drill keep
running; svc-runner stays skipped via its missing NFS volume. Document
the nine drill steps host-by-host in the README.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The marker discovery matched any .dr-drill-marker under files/, so
SRC_REL could resolve to a parent tree and step 8 restored the wrong
subtree into the live export as root, leaving n8n_data owned root:root
and n8n (uid 1000) crash-looping on EACCES. Scope both repo layouts to
the primary volume's own marker and declare uid/gid 1000 on the n8n
data volume so nfs_prep enforces ownership on every deploy instead of
relying on Docker's copy-on-first-use.
The drill also stacked four copies of the backup tree on shared space
(live export, pulled tree, fully-allocated LUKS image, final restore
into a tmpfs /tmp) and ENOSPCed on large apps. Restore to /var/tmp,
drop the pulled tree before the device recover, make the loop image
sparse and free it after the marker verify, and stage only the volume
generation instead of the whole machine tree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The swarm convergence gate derived runner_runner-1 from the services
metadata and polled a stack service the role never creates: the runner
fleet deploys as node-local compose projects on every node because each
runner needs its own node's docker socket and identity. Declare
workload: node-local on the runner services entry and let the gate
context map it to HAS_SWARM_SERVICE=false, so the stack-service gates
skip themselves while the swarm deploy, the registry distribution chain
and the CLI e2e keep running. Roles without the attribute keep the
image/template heuristic; workload: stack forces the gates on. Document
the runner architecture for both deploy modes in the role README.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.
Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Step 8 rewrites the export subtree directly on the ganesha server's
backing filesystem while the server keeps running and the nodes keep
their client mounts, so ganesha's mdcache and the client dentry caches
still describe the pre-wipe tree and update-pass tasks are Rejected
with readdirent ENOENT for the whole converge window. Step 8b restarts
ganesha, lazily remounts every swarm node and gates on a real write
probe through the manager's mount, each command timeout-bounded so a
wedged DinD node fails loudly instead of hanging the job.
The drill's tar extracts remapped ownership by name on the receiving
host like the rsync legs did; extract with --numeric-owner. Drill paths
now come from the INFINITO_DIR_* SPOT env vars.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add DIR_BACKUPS to utils.paths and INFINITO_DIR_BACKUPS /
INFINITO_DIR_SECRETS env handlers, derive the recover-console constants
(including NFS_EXPORT_STATE) from utils.paths and the nfs export SPOT,
and switch every remaining self-built /var/lib/infinito subpath - the
secrets recover targets, the swarm extras backup source, the users
token-store candidate chain, the CI recover drill, the swarm NFS
diagnostics and the playwright reports dir - to those constants. Tests
consume the env constants directly; narration comments in the touched
files are dropped per the comment lint.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reflect the folded drill: base.sh ends after recovery, the matrix update pass boots the recovered stack, and verify_recovered_marker.sh asserts the live marker.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A prior wedged drill run can leave loop devices bound (backing file deleted, or bound to the image path); a full loop pool makes cryptsetup luksFormat abort with "loop device with autoclear flag is required". Detach the orphaned and stale loops first so cryptsetup can autoclear-attach a fresh one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The drill (backup/base.sh) no longer runs its own step-10 redeploy; it recovers device -> nfs export -> volume -> secrets and writes a sentinel. The matrix update pass (which runs next anyway) boots the recovered stack, and verify_recovered_marker.sh asserts the live marker after it. Saves one full stack redeploy per drilled app; the marker-gone check before recovery stays in 05_wipe_export.sh.
Also derive recover_drill.sh REPO from the script location instead of an INFINITO_REPO_ROOT shell-side default (lint: no INFINITO_* / absolute-path defaults in .sh).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Node image now builds FROM the distro infinito image (python, ansible, CLI baked in) instead of installing a per-node .deb, so there is no per-node bootstrap or timeout; the swarm CI job depends on the distribution build. Move compose/swarm.yml to compose/swarm/compose.yml and relocate the swarm matrix orchestrator from cli.administration.deploy.swarm to utils.tests.swarm (test code out of the shipped CLI).
Back the NFS export root with a disk-backed named volume instead of a RUNNER_TEMP bind: local-act RUNNER_TEMP resolves onto tmpfs and ganesha FSAL_VFS cannot export a tmpfs, so it loaded zero exports and the controller mount got ENOENT. Add optional compose/swarm/cache.override.yml wiring swarm nodes to the package-cache frontend when present, leaving CI without a cache unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add scripts/tests/deploy/ci/recover_drill.sh: seed a backup root (nfs, volume, secrets, each with a marker) plus a docker volume, then run recover full <root> localhost to restore every backup-stored type into its live target and verify the markers. Runs between PASS 1 and PASS 2 of the compose development deploy.
Hook it via _maybe_recover_drill in cli.administration.deploy.development, gated by INFINITO_RECOVER_DRILL (default true).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add cli.administration.recover: one uniform type/source/target command wrapping each svc-bkp-* role files/recover.py (nfs, volume, secrets, device) plus full for whole-tree recovery. Colon-encoded sources are classified by shape; device supports an on-device subpath, a generation, and remote pull over ssh.
Replace the per-host swarm DR-drill scripts (03/06/07/08_recover_*.sh) with the recover CLI in backup/base.sh. Pass the device on-device subpath relative (the CLI reads absolute segments as the restore-root) and load the applications-free .deploy.yml extras twin on the step-10 redeploy so host_vars credentials survive the -e override.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds make swarm-diagnostic (alias i8sadi): read-only probes across a
live swarm-test cluster's nodes via exec_node.sh - backup unit state +
recent journal, NFS mounts, D-state (wedged-NFS) processes, running
rsync/db-dump processes, and disk usage - so a hung backup unit can be
root-caused without hand-writing docker exec pipelines.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a routine/README.md describing what each phase of the swarm lab
run does (bring-up, the variant-matrix round loop, the DR drill's nine
steps, the chaos phase, teardown), with the moments a backup-executing
systemd unit actually starts highlighted, and links to the per-step
scripts below the diagram.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The DR drill's simulated USB was a fixed 512M loop image (~449M usable
after LUKS2 + ext4), but the deployed local-2-device unit syncs the
whole pulled backup root, which after the remote pull holds two
machine-id trees (the manager's docker+secrets repos and the NFS
export repo). For an app the size of akaunting that overflows the
image mid-rsync and the drill fails with ENOSPC at step [5/9]. base.sh
now measures the pulled tree (du -sm) and passes 2x its size + headroom
(floor 2G) to 02_luks_device.sh, which sizes the loop image from that
argument; fallocate keeps the image sparse so the larger floor is free.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A wedged manager (act-swarm containerd stall) made the docker exec tar
block up to 900s; on a cancelled job that ate the runner's grace window
and the always() Cleanup step never ran, leaking the DiD cluster. Probe
the manager first (10s) and cap the copy at 120s with --kill-after so a
hung exec is force-reaped; teardown then always follows. Reports stay
collected before teardown, since they live inside the manager node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The provisioner copied the manager host_vars only to the workers, so the
service-loader preload on the nfs and backup nodes merged a defaults-only
applications dict (credentials lookups failed), and the bootstrap play
seeded /etc/hosts only on manager and nfs, so the backup node's msmtp
healthcheck died with EX_NOHOST. Both nodes now get the host_vars copy
and the app-domain hosts entries.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
swarm_stack_ready.sh piped 'docker stack services' through a process
substitution, so the timeout wrapper added in 8516e3167 turned a slow or
wedged manager into an empty listing, an empty not_running set and exit
0: the convergence gate would have gone green with zero running replicas.
Capture the listing, fail on a non-zero rc.
That exposed the second half: post_deploy gated the swarm wait on the
cluster-wide DEPLOYMENT_MODE, so forced-compose roles (matrix ansible
flavor, bigbluebutton) ran it against a stack that does not exist and
only passed because the old code swallowed the error. Gate it on
compose_mode, like the handlers already do.
teardown.sh: the NFS mountpoint and umount calls, the likeliest hang
sites, were the only ones left unwrapped, and the timeout around docker
inspect inverted the D-state warning (a hung inspect returned 124, the
if went false and the script reported a clean teardown). Warn on rc 0
and 124, and shrink the per-node budget so the script fits inside its
step timeout.
Swarm step budget: the bounded steps alone summed to 365 over a 360-min
cap, so cleanup was still starved. Deploy 240, diagnostics 15, playwright
15, rescue 25, cleanup 15.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
27 jobs of run 29147179243 burned their full 360-min budget. Three
systemic layers, all addressed here:
- The CLI e2e task ran the staged test.sh unbounded, and test.sh blocks
in 'systemctl start' on a oneshot backup unit with no JobTimeout; on a
wedged dockerd or a stalled NFS mount both freeze silently for hours.
The task now runs the script under 'timeout TEST_E2E_CLI_TIMEOUT'
(60 min via the timeout lookup, so TIMEOUT_FACTOR scales it) and the
unit starts are bounded by BKP_TEST_HEALTH_TIMEOUT.
- The converge probe re-ran the full per-service diagnostics (service ps,
service logs, inspect, journalctl) on EVERY retry, so 600 nominal
2-second attempts cost 25-33s each and silently outlived the step
timeout. The block is gone (the workflow-level container.py rescue
snapshot already collects it recursively on failure) and every docker
call is timeout-bounded. Same removal in the swarm collect script,
which keeps only the NFS and render evidence container.py does not see.
- Playwright-collect, rescue diagnostics and Cleanup had no
timeout-minutes, and teardown.sh calls docker against D-state nodes,
so a failed job hung there for hours until the job cap killed it
without artifacts. Every step is bounded now, every teardown call is
timeout-wrapped, and the deploy steps (swarm 270, compose 300) leave
room for diagnostics and cleanup below the 360-min cap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
routine/08_cleanup.sh becomes utils/clean/teardown.sh and topology/
becomes utils/topology/ so routine/ holds only the sequenced flow; all
consumers (workflow steps, compose/swarm.yml env guards, relative
sources, README) follow. The shell rescue collector and its lint are
gone: utils/diagnostics/container.py covers compose and swarm
identically. extend_inventory gains the backup-host placements the DR
drill deploys.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
extend_inventory maps svc-bkp-remote-2-local + svc-bkp-local-2-device onto
the backup host and write_extras configures them (backup_providers =
manager + NFS server IPs; device mount/target/source), so the matrix
deploy installs the real systemd units there. The drill now exercises
them: 01_ssh_trust.sh only installs the pull identity, the remote-2-local
unit pulls every provider (replacing both manual pull calls), and after
02_luks_device.sh simulates the USB plug the local-2-device unit performs
the device sync; the device recover passes the configured target subpath
via --device-target. Unit absence on the backup host is now a hard
failure since the roles deploy with the round.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add routine/backup/: base.sh drives a marker seeded into the live NFS
volume and the manager secrets through the whole chain (volume+secrets+nfs
backup units -> remote-2-local pull -> local-2-device LUKS 'USB') and back
via every recover.py (device -> local root -> nfs export, docker volume and
host secrets), asserting each marker survives. Each per-host step is one
in-node routine run over docker exec. extend_inventory maps the volume/
secrets/nfs backup roles onto their hosts so the units install; write_extras
generates the pull keypair; the INFINITO_SWARM_BACKUP_* keys are added to
default.env + the passthrough handler.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the per-node docker-run bring-up with a single node image
(compose/swarm/Dockerfile bakes python3/dnsmasq/openssh/rsync/cryptsetup)
declared in compose/swarm.yml, one compose up, and one ansible play
(compose/swarm/playbook.yml over ansible_connection: docker) that waits for
systemd, captures IPs, wires lab DNS and installs the .deb per node. Reshape
the harness into topology/ (naming SPOT), routine/ (01_bootstrap + the
sequenced 02-08 steps) and utils/ (clean/, collect/, shared helpers); the
workflow, matrix.py and Makefile follow the new paths. swarm init/join was
already ansible-owned (svc-swarm-node), corrected in the README diagram.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A default like ${BACKUP_KEY_PATH:-/tmp/swarm-nfs-backup.key} duplicates
a path contract that belongs in a SPOT (default.env or a role vars
file) and drifts silently. The guard covers every ALL-CAPS variable
whose inline default starts with a slash, allows per-line nocheck
suppression, and its failure message forbids dodging the syntax via
conditional assignments. Legacy offenders carry explicit nocheck
reasons (act tooling runs before default.env exists, POSIX TMPDIR
convention).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every Ansible rescue: block used to carry ad-hoc per-role failure dumps that
drifted apart and captured different subsets. Replace them all with one
diagnostics script (scripts/system/diagnostics/rescue.sh) that snapshots
containers (full logs plus inspect per container), swarm services (ps plus
logs per service), journal, host resources, runtime info and stats into a
per-run folder and always exits 1 so the deploy still fails. A new lint
enforces the shape: every rescue ends with the script, at most one
nocheck-justified extra task before it (postgres pg_hba dump, xwiki
extension-repo probe), and a nocheck rescue-recovery exemption for the one
block that must genuinely recover: keycloak object updates, where a
cross-node reschedule needs a swarm-wide node re-resolve plus a delegate_to
change that only a rescue can express; its recovery is kept and now also
captures diagnostics before the terminal failure.
The snapshot folder path is a group_vars SPOT: DIR_RESCUE_DIAGNOSTICS and
DIR_VAR_LIB live in group_vars/all/05_paths.yml as plain strings; new env
handlers derive INFINITO_RESCUE_DIAGNOSTICS_DIR and INFINITO_DIR_VAR_LIB
from it for the generated .env, replacing the default.env entries, and the
play-level environment forwards the value to every task host so rescue.sh
needs no second default. CI uploads the folder like the inventory and
playwright artifacts: compose copies it out of the deploy container in
dedicated.sh, swarm captures per node (manager and both workers) into
node-named subfolders.
Deleting the ad-hoc dumps orphaned helper files (postgres inspect_container,
terminator docker_diagnostics, peertube diagnostics, matomo bootstrap debug,
xwiki and openresty rescue includes) and 14 role vars; all removed. Generic
pieces of those dumps (runtime version and info, stats, dmesg oom evidence,
swapon) moved into the SPOT script instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
storage.nfs shrinks to the deployment-specific server address: export_base moves into svc-storage-nfs-server's services.yml (the server dictates where exports root) and the mount protocol version into svc-storage-nfs-client's services.yml (the client owns how it mounts). Both are read through utils.storage.nfs loaders by the nfs lookups, the env handler and 00_topology, eliminating the unrendered-variables class for these values; the group_vars assert shrinks accordingly and the swarm test extras stop restating role defaults.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
INFINITO_PYTHON_INSTALL_SCRIPT and INFINITO_DOCKER_CLI_INSTALL_SCRIPT join the default.env SPOT: the Dockerfile consumes them as required args (:? in the RUN as last defense), all three image builders (compose.yml, build.sh, push.sh) pass them hard-required, the shell consumers read the SPOT via default.env grep and the Makefile via BASH_ENV. INFINITO_SWARM_NFS_EXPORT_BASE is no longer a static default: a new env handler derives it from the deploy SPOT group_vars/all/15_storage.yml, 00_topology greps that file (and the STATE_SUBDIR constant) directly, and the swarm test extras stop restating the group_vars value. Stale hadolint directives are restored and recognised; leftover narration comments in the touched files are dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
04_bootstrap_nodes.sh consumed INFINITO_PACKAGE_INSTALL_SCRIPT and INFINITO_NODE_SRC_DIR from the ambient environment; local make targets provide them via BASH_ENV, but plain workflow steps do not, so every swarm CI job died at the required-var guard. Source load.sh explicitly like copy_worktree_to_node.sh.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
INFINITO_PACKAGE_INSTALL_SCRIPT in default.env replaces the scripts/install/package.sh literal in the Dockerfile (new required build arg), the environment install step, and both swarm bootstrap call sites. Shell consumers hard-require the variable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A wedged DinD dockerd hung a single diagnostics docker exec until the 360-minute job timeout cancelled the job together with its artifact uploads (job 86013153099 sat 5h in service logs). Route every diagnostics exec through a timeout wrapper, tail-limit the log dumps, bound showmount, cap the collect step at 30 minutes, and dump the pending services' log tails once on the final converge failure. The NFS state path moves into a single source of truth: STATE_SUBDIR in utils/storage/nfs.py feeds the nfs_state_subdir lookup (deploy side) and the derived INFINITO_SWARM_NFS_STATE_PATH env var (harness side); the diagnostics script requires both path vars instead of defaulting, and the swarm inventory drops its redundant export_base restatement.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-role existence asserts and the test script path were controller paths evaluated on the remote swarm node (docker connection, node checkout under /opt/infinito-nexus), so every svc-runner swarm job failed on a stat that could never succeed. Assert both required assets in one localhost-delegated loop, stage roles/<app>/files/test onto the node, run the staged script, and delegate the optional defaults/vars probes to the controller (they silently skipped before). svc-runner's test reaches its sources via RUNNER_SRC_DIR (compose: INFINITO_SRC_DIR dev-container mount; swarm: node checkout). The node checkout path itself moves into a single source of truth: INFINITO_NODE_SRC_DIR in default.env, consumed by the bootstrap/copy scripts, the swarm-playwright target, and the swarm inventory extras.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rename the meta/services.yml key across all roles, the roles_with_default_placement lookup (now roles_with_placement), the get_role_default_placement / iter_roles_with_default_placement helpers, and every consumer (swarm closure, compose_volumes filter, nfs_prep, container deploy template, constructor stage, extend_inventory) plus their tests and docs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
act/workflow.sh: build local/act-runner-fixed:latest on demand when that tag is selected but the image is absent, avoiding an image-not-found failure. swarm/_context.sh: probe_app_reachable falls back to a bash /dev/tcp connect and finally the container's own Docker health status, so toolless images (e.g. fider's debian-slim without curl/wget/nc, which exit 127 on every in-container probe) are no longer reported unreachable. Both additions are additive: the curl/wget/nc path and the non-matching act tag are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The previous commit read 'producer_rc=${PIPESTATUS[0]}' then
'consumer_rc=${PIPESTATUS[1]}', but the first scalar assignment resets
PIPESTATUS to a one-element array, so the second read hit an unbound
index and 'set -u' aborted the collector with exit 1 - failing every
swarm job whose deploy and tests actually passed (jenkins, pixelfed).
Snapshot the whole array in one statement, then index it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The collector piped 'docker exec tar | tar' with '|| true', so a hung
'docker exec' blocked the run indefinitely and failures were swallowed.
Bound it with 'timeout 900'; a producer exit of 124 (manager hung) now
fails the collector via PIPESTATUS, while other collection errors stay
best-effort so partial reports do not fail the job.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The post-reschedule assert mounted the state export with vers=3, which
ganesha does not serve ('Protocol not supported'), so the marker check
false-negatived and killed the gitea roundtrip; reproduced and the
v4 mount shown to read the marker on the live cluster. Replace the
single mount with a cascade covering every production client_src form:
nfs4 vers=4 against the full path (ganesha), against :/ (kernel fsid=0
root), then the previous vers=3 full-path arm (kernel v3). The block
also returned the umount status instead of the grep result, letting a
missing marker pass silently; propagate the grep rc explicitly.
The in-container reachability probe's last arm used bash /dev/tcp, but
e.g. the ejabberd image ships neither bash nor curl; use busybox
nc -z instead and move the probe chain into probe_app_reachable() in
_context.sh alongside a NFS_CHECK_MOUNTPOINT SPOT.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
container cp needs its source on the node running the docker CLI, so
role_path sources break on swarm workers. The joomla plugin dir now
ships as a controller-side tarball through container exec -i. A new
repository lint forbids docker/container cp in tracked .yml and .sh
files; the six legitimate same-node uses carry justified nochecks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/var/lib/infinito on the swarm manager is an NFS mount inside the container's own mount namespace. docker cp reads the daemon-side rootfs and sees the empty mountpoint, so it copied zero files and the uploaded Playwright artifact was always empty. Stream the reports out through docker exec (in-namespace), the same access pattern the swarm diagnostics already use.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
moodle mail failed two ways in a swarm cluster. (1) moodle's build_context
pulled the FULL sys-svc-mail-msmtp role early, which rendered /root/.msmtprc on
the manager before Mailu had created the no-reply token and tripped the role's
run-once flag, so the manager kept an empty-password msmtprc and the health
check failed 'authentication method PLAIN needs a password'. Pull only
tasks_from: ca_bundle so the manager re-renders on the normal post-token pass.
(2) msmtprc forced host=127.0.0.1 for any in-container host, but the NFS server
sits outside the swarm mesh and has no local Mailu, so it hit 'connect to
127.0.0.1:465 refused'. Narrow the 127.0.0.1 branch to compose or a real
svc-swarm-node AND to when Mailu is actually deployed (email.external); else use
email.host, and map the NFS node's test DNS to the manager IP. Also import the
shared self-signed Root CA for the relay's TLS trust.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
down_act_outer.sh is a best-effort pre-clean (set +e), but its last statement (docker rm -f) can race act's own --rm finalize and return non-zero, so the script exited non-zero and aborted swarm-zombie before the deploy. Force exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The converge-failure diagnostics truncated service logs to --tail 50 (and head -100), but a crashing service prints its exception at the TOP of the trace, so the root cause was cut off and non-converging services could not be debugged from CI. Remove the caps in the converge-failure rescue (wait_converged.yml) and the diagnostics collector (12_collect_diagnostics.sh) so the full log, including the exception, is captured.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
docker rm -f on a node stuck on a dead NFS mount races (did not receive
an exit event) and leaves the container stopped but not removed. Add a
SIGKILL + plain docker rm retry loop before the host-docker-restart
fallback, so make swarm-clean self-recovers without sudo.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A multi-variant app expanded into N swarm matrix entries, and
swarm-zombie ran them all on one host under identical SWARM_NAME
cluster names, so every deploy after the first aborted with
'network ... already exists'. Pin the act matrix to variant 0
(default, overridable via variant=) so a multi-variant app validates
a single cluster, in parity with the compose step's variant=0.
workflow.sh now splits ACT_MATRIX on ';' into multiple --matrix
filters so apps and variant can be pinned together.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
