The loop may drop a distro when the budget runs out, and both skip paths fell through to an implicit exit 0, so a run that never touched two distros still reported success. That hid the coverage gap: arch, fedora and centos get no package cache (cache.override.yml is apt-only), are therefore the slowest, and are the first the remaining < max_seen heuristic drops.
Skipping stays non-fatal and the order stays randomised, but every distro now shows up in a GITHUB_STEP_SUMMARY table in execution order, as passed, failed with its rc, or skipped with the reason. The table is also rendered before an aborting exit, since that is the run whose table matters. Two contract tests cover it; the order assertion checks the logged shuf order rather than the input list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
d9a4498b3 collapsed eleven CI steps into one, which broke the assumption every handover in the drill rested on: a $GITHUB_ENV write only comes back in the NEXT step. Inside one step nobody sees it. matrix.py reads os.environ["MGR_IP"], so every swarm job died with KeyError before the first deploy round; 05 -> 06 -> 07 passed INITIAL_NODE/NEW_NODE the same way and would have died next.
The node IPs now come from the topology SPOT: compose.yml already pins every node to those INFINITO_SWARM_*_IP keys, so the docker-inspect capture only read back what compose had set. It goes, together with the $GITHUB_ENV publish and lab_net. lab_domain_target reads the env instead, guarded by an assert task because lookup('env') yields '' for a missing key and an empty dnsmasq `address=/domain/` is valid syntax meaning NODATA -- the failure would have surfaced 30 retries later as a DNS problem.
INITIAL_NODE/NEW_NODE move to a SWARM_DRILL_ENV file, created fresh per distro so the previous distro's node cannot leak into the next drain. The out-of-drill callers of _context.sh keep working through a visible guard; both writers stay strict via :?.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The discover job was named after the full distro list and the plan summary printed all five, but test-deploy-swarm.yml never set INFINITO_DISTROS, so every run fell back to the default.env distro and only debian was ever exercised. The job name has been advertising coverage that did not exist.
The workflow now passes the distros input into the env and hands the drill to the shared loop from 1bbe7a0eb. Bootstrap, setup, matrix deploy, seed, drain and assert move out of eleven separate steps into routine/00_one.sh, which is what the loop runs once per distro.
Collection and teardown move with them, into an EXIT trap: it replaces the removed 'if: always()' and 'if: failure()' step conditions and returns the original exit code so cleanup cannot mask the deploy failure. The always() teardown step stays as a second net for a step that times out as a whole.
Playwright reports and rescue diagnostics gain a distro segment, matching the layout the compose path already writes, because a second distro would otherwise overwrite the first one's evidence. The structure summary carries the distro in its heading for the same reason.
The README described the old shape and is corrected: the loop entries are listed, topology_summary.sh was missing, and 01_bootstrap.sh is no longer 'one CI step'.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill configured svc-bkp-remote-2-local with a fixed manager plus NFS provider list for every app, while the manager only runs a backup repository when the app's include closure pulls svc-bkp-volume-2-local or svc-bkp-secrets-2-local in. For an app whose container_backup consumer is statically off, such as svc-dns-unbound, user-backup never ran on the manager, so the pull was rejected with 'Permission denied (publickey,password)' and the fail-if-any wrapper took the whole unit down at drill step 4/9. Four roles reach that step today: svc-dns-unbound, svc-db-redis, svc-registry-cache and svc-registry-docker.
backup_repos.py expresses the placement once and matrix derives the provider list from it per round. derive_includes accepts the round's variant map because INFINITO_APP_VARIANTS only reaches the subprocess callers. The extras file keeps the device paths the drill reads back and loses the provider list nothing read from it, and the drill banners stop claiming backup legs that were never installed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill configured svc-bkp-remote-2-local with a fixed manager plus NFS provider list for every app, while the manager only runs a backup repository when the app's include closure pulls svc-bkp-volume-2-local or svc-bkp-secrets-2-local in. For an app whose container_backup consumer is statically off, such as svc-dns-unbound, user-backup never ran on the manager, so the pull was rejected with 'Permission denied (publickey,password)' and the fail-if-any wrapper took the whole unit down at drill step 4/9. Four roles reach that step today: svc-dns-unbound, svc-db-redis, svc-registry-cache and svc-registry-docker.
backup_repos.py expresses the placement once and matrix derives the provider list from it per round. derive_includes accepts the round's variant map because INFINITO_APP_VARIANTS only reaches the subprocess callers. The extras file keeps the device paths the drill reads back and loses the provider list nothing read from it, and the drill banners stop claiming backup legs that were never installed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Restarting docker does not clear a container whose shim never sent an exit
event: that shim belongs to containerd, a separate unit that survives the
restart. Measured on a host carrying two such remnants - docker.service had
been restarted hours earlier while containerd.service had been up for three
days, and `rm -f`, plain `rm` and `stop -t 1` all still failed afterwards.
The hint also has to say what the restart costs. It wipes every exec instance,
so following it during a deploy kills the run: an act deploy died after 1h48m
with `failed to inspect exec: No such exec instance`, and the log carried no
ansible error at all, only post-steps failing in milliseconds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every teardown left one node container behind, always the same one, and the
warning blamed a kernel D-state. The cause was a budget: the client-unmount
step gave all four containers one shared 120s allowance, the unmount chain
costs up to three attempts per mount, and the third container in the list never
got its turn. That container was then killed with its mount still attached.
Killing was the second half of it. The node image sets STOPSIGNAL SIGRTMIN+3 so
that `docker stop` triggers a clean systemd poweroff, but the teardown only ever
ran `docker kill`, which skips it and leaves the nested engine and its task
containers to die mid-flight. Those task containers hold binds of the NFS
superblock inside their own mount namespaces, where an unmount in the node
namespace cannot reach them, so the nested engine has to go down first.
Under act the ansible controller is the act job container itself, so its own
NFS mount is invisible to the host mountpoint check and survived every cleanup.
Three more paths reached the same containers without any of this: the
cross-cluster reaper removed them with no unmount at all and then reported the
wedges it had just manufactured, the bootstrap pre-clean detached only inside
the NFS server, and the escape hatch filtered its candidates down to
`name=nfs-server` so it could never see the worker that actually survives.
The removal verdict itself was wrong: a single inspect fired immediately after
`rm -f` reports containers the daemon is still removing, and two of the three
warnings in the observed run were false alarms.
Verified by tearing a live five-node cluster down: all five containers removed,
no remnant, no warning.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The failover-drill assert polled service convergence for 60 iterations (120s).
OpenProject web boots in ~140s, so after 06_drain_worker rescheduled it the
replicas turned healthy ~20s after the poll gave up, and 07_assert_state.sh
reported 'Replicas not fully converged: 1/3' and exited 1 even though the stack
did converge moments later. The loop breaks early once converged, so raising the
ceiling to 120 iterations (240s) costs fast apps nothing and only widens the
window for heavy apps.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Conflicts: keep both new passthrough env vars; adopt the swarm-nfs
ollama manager-ops split while porting our model-pull retry hardening
into it. Strip three narration comments the stricter comment lint
rejects.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every DinD node is attached to a second network by the bootstrap cache connect, so the embedded DNS returns two A records per node name in undefined order; workers that resolved the manager onto the cache network could not reach the registry bound to the lab IP and rejected locally built images with 'No such image', pushing all replicas onto the manager. Pin a fixed lab ipv4_address per node and every node name via extra_hosts (files-first resolution wins over DNS); the subnet and IPs live in default.env as the INFINITO_SWARM_* SPOT consumed by compose interpolation and the topology exports. Proven by A/B: the identical friendica variant-2 swarm job went from worker rejects to fully green including the drain test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolves the runner Dockerfile to the lint-standard Acquire::Retries=3
and keeps the collabora TODO wording; hardens the merged tree for the
stricter lints (apt retries in the tor Dockerfile, shell-or-true
markers, comment cleanup).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pin node hostnames to their lab-net IPs (dual-homed nodes let Docker
DNS hand workers the cache-net IP while the registry binds only the
lab IP), quiesce the CI compose tor before bootstrap (it publishes the
same onion identity as the cluster), and reap wedged nodes with a
retried SIGKILL so leftovers cannot starve later phases.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Back to the pre-swarm-nfs-fix baseline:
- 5c2c8f531 + 985ec7eab: the swarm writer-quiesce DR drill. Deep-dive
attribution of CI run 29715516546 found it the dominant failure source
(17 of 23 jobs): [9b/9] handed-off generation mismatch, [6/9] quiesce
stack-rm network race, [4/9] remote-2-local pull stall, compose discourse
restore duplicate key.
- d6f980574: openproject swarm ingress to web:8080. Carried a leaking
proxy_internal_port lookup (no per-app default) that broke the seaweedfs
dependency front-proxy ('Key web not found'); the bypass-the-Apache-proxy
approach was never verified against downloads/X-Sendfile.
- 1dd153bad: xwiki replicas:1 + mem 3g. Did not resolve the xwiki superadmin
failure (still red in run 29715516546), so it is unverified.
Re-approach each from a clean baseline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up corrections to b449f8157 (swarm DR drill):
- export test.env in 06/07 via set -a so the child db restore/probe
scripts inherit BKP_TEST_DATABASES_CSV and BKP_TEST_RESTORE_BIN
instead of aborting at their :? guards before any restore runs
- move db_probe.sh/db_restore.sh into a db/ subfolder (probe.sh,
restore.sh) and update all callers, so they stop tripping the
prefix-grouped-files structure lint
- define TEST_E2E_SWARM_DRILL in group_vars/all and drop the now
redundant default() from test.env.j2 (satisfies the undefined-var
and spot-default lints together)
- fix a mermaid parse error in the drill README: a ; in a
sequenceDiagram note was read as a statement separator
make quality-high green (test + lint phases, 15 targets, 0 failures).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Defer Swarm SQL replay to the disposable DR drill so shared writers such as Discourse are stopped before the destructive restore. Verify the selected generation with before/after database probes while preserving the Compose restore path.
swarm-exec reaches a DinD node; reaching a swarm service container (moodle,
openldap) needed a nested docker exec. Add swarm-app-exec, which resolves one
running replica on the node and runs a base64-encoded command inside it, so
live in-container inspection stays on a sanctioned make helper. Adding a
second exec_* script tripped the prefix-group lint, so exec_node.sh moves to
exec/node.sh and the new one lands at exec/service.sh.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
07_assert_state.sh sources _context.sh under set -o pipefail, and the printf | head-n1 that picked the first NFS volume raised a broken-pipe write error the moment head closed the pipe after line one. Any role with two or more NFS volumes (erpnext ships erpnext_sites and erpnext_logs) crashed the reschedule assertion before it could check the surviving marker. Plain parameter expansion returns the same first line with no pipe.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The swarm deploy job now renders the same summary page as compose:
role runtimes, a swarm structure section (topology roles plus live node
list and service placement from the manager) and the Playwright table.
The matrix deploy tees its ansible output and prints the compose
matrix-deploy round/PASS markers, so the runtime parser segments swarm
runs per variant round too.
The runtime table unifies tasks and roles with a Type column (tasks
first), appends the recap total row and flags a mismatch between the
parsed sum and the recap total; rows are read from ROLES RECAP sections
only, which removes the double-counted TASKS RECAP task rows. Each role
row carries one column per host: executed, skipped or failed, with
ignored failures downgraded. The Playwright variant column drops the
redundant variant- prefix. Verified against the logs of runs
29656268771 (swarm) and 29620158441 (compose): every segment's parsed
sum matches the recap total within 0.05s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The context derived PRIMARY_NFS_VOLUME from the first declared volume, but gitlab's first volume is repositories with nfs: false (gitaly cannot run on NFS), so the seed step wrote its pre-drain marker into a state directory that never exists on the export and killed a job whose deploy had already converged. Volumes declaring nfs: false are now skipped, making the drain check target a volume that actually lives on the export (gitlab_shared).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dedicated.sh streams rescue diagnostics via docker exec tar since docker cp cannot see the systemd container's mount namespace and silently extracted nothing. recover_drill.sh guards on docker availability: non-stack role deploys now drill the nfs and secrets legs instead of hard-failing on the impossible volume leg; stack jobs still drill all three. 04_stack_rm_wait tolerates an already-removed stack and re-issues the rm once services are gone.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The drill entry gate treated 'deploys no swarm service' as 'has nothing
to restore', but the backup chain operates on NFS volumes and deployed
backup units, not on scheduler objects; bigbluebutton's seven
NFS-backed volumes went entirely unproven. Gate the drill solely on the
NFS-flagged volume and make only the full-disaster step
workload-aware: stack roles keep docker stack rm, node-local roles get
their compose project (label com.docker.compose.project=<entity>, the
compose handler's --project convention) stopped on every node so no
writer races the export wipe and restore. Declare
workload: node-local on bigbluebutton so the phantom-service
converge/chaos gates skip while play, e2e and now the drill keep
running; svc-runner stays skipped via its missing NFS volume. Document
the nine drill steps host-by-host in the README.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The marker discovery matched any .dr-drill-marker under files/, so
SRC_REL could resolve to a parent tree and step 8 restored the wrong
subtree into the live export as root, leaving n8n_data owned root:root
and n8n (uid 1000) crash-looping on EACCES. Scope both repo layouts to
the primary volume's own marker and declare uid/gid 1000 on the n8n
data volume so nfs_prep enforces ownership on every deploy instead of
relying on Docker's copy-on-first-use.
The drill also stacked four copies of the backup tree on shared space
(live export, pulled tree, fully-allocated LUKS image, final restore
into a tmpfs /tmp) and ENOSPCed on large apps. Restore to /var/tmp,
drop the pulled tree before the device recover, make the loop image
sparse and free it after the marker verify, and stage only the volume
generation instead of the whole machine tree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The swarm convergence gate derived runner_runner-1 from the services
metadata and polled a stack service the role never creates: the runner
fleet deploys as node-local compose projects on every node because each
runner needs its own node's docker socket and identity. Declare
workload: node-local on the runner services entry and let the gate
context map it to HAS_SWARM_SERVICE=false, so the stack-service gates
skip themselves while the swarm deploy, the registry distribution chain
and the CLI e2e keep running. Roles without the attribute keep the
image/template heuristic; workload: stack forces the gates on. Document
the runner architecture for both deploy modes in the role README.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.
Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Step 8 rewrites the export subtree directly on the ganesha server's
backing filesystem while the server keeps running and the nodes keep
their client mounts, so ganesha's mdcache and the client dentry caches
still describe the pre-wipe tree and update-pass tasks are Rejected
with readdirent ENOENT for the whole converge window. Step 8b restarts
ganesha, lazily remounts every swarm node and gates on a real write
probe through the manager's mount, each command timeout-bounded so a
wedged DinD node fails loudly instead of hanging the job.
The drill's tar extracts remapped ownership by name on the receiving
host like the rsync legs did; extract with --numeric-owner. Drill paths
now come from the INFINITO_DIR_* SPOT env vars.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add DIR_BACKUPS to utils.paths and INFINITO_DIR_BACKUPS /
INFINITO_DIR_SECRETS env handlers, derive the recover-console constants
(including NFS_EXPORT_STATE) from utils.paths and the nfs export SPOT,
and switch every remaining self-built /var/lib/infinito subpath - the
secrets recover targets, the swarm extras backup source, the users
token-store candidate chain, the CI recover drill, the swarm NFS
diagnostics and the playwright reports dir - to those constants. Tests
consume the env constants directly; narration comments in the touched
files are dropped per the comment lint.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reflect the folded drill: base.sh ends after recovery, the matrix update pass boots the recovered stack, and verify_recovered_marker.sh asserts the live marker.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A prior wedged drill run can leave loop devices bound (backing file deleted, or bound to the image path); a full loop pool makes cryptsetup luksFormat abort with "loop device with autoclear flag is required". Detach the orphaned and stale loops first so cryptsetup can autoclear-attach a fresh one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The drill (backup/base.sh) no longer runs its own step-10 redeploy; it recovers device -> nfs export -> volume -> secrets and writes a sentinel. The matrix update pass (which runs next anyway) boots the recovered stack, and verify_recovered_marker.sh asserts the live marker after it. Saves one full stack redeploy per drilled app; the marker-gone check before recovery stays in 05_wipe_export.sh.
Also derive recover_drill.sh REPO from the script location instead of an INFINITO_REPO_ROOT shell-side default (lint: no INFINITO_* / absolute-path defaults in .sh).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Node image now builds FROM the distro infinito image (python, ansible, CLI baked in) instead of installing a per-node .deb, so there is no per-node bootstrap or timeout; the swarm CI job depends on the distribution build. Move compose/swarm.yml to compose/swarm/compose.yml and relocate the swarm matrix orchestrator from cli.administration.deploy.swarm to utils.tests.swarm (test code out of the shipped CLI).
Back the NFS export root with a disk-backed named volume instead of a RUNNER_TEMP bind: local-act RUNNER_TEMP resolves onto tmpfs and ganesha FSAL_VFS cannot export a tmpfs, so it loaded zero exports and the controller mount got ENOENT. Add optional compose/swarm/cache.override.yml wiring swarm nodes to the package-cache frontend when present, leaving CI without a cache unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add cli.administration.recover: one uniform type/source/target command wrapping each svc-bkp-* role files/recover.py (nfs, volume, secrets, device) plus full for whole-tree recovery. Colon-encoded sources are classified by shape; device supports an on-device subpath, a generation, and remote pull over ssh.
Replace the per-host swarm DR-drill scripts (03/06/07/08_recover_*.sh) with the recover CLI in backup/base.sh. Pass the device on-device subpath relative (the CLI reads absolute segments as the restore-root) and load the applications-free .deploy.yml extras twin on the step-10 redeploy so host_vars credentials survive the -e override.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds make swarm-diagnostic (alias i8sadi): read-only probes across a
live swarm-test cluster's nodes via exec_node.sh - backup unit state +
recent journal, NFS mounts, D-state (wedged-NFS) processes, running
rsync/db-dump processes, and disk usage - so a hung backup unit can be
root-caused without hand-writing docker exec pipelines.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a routine/README.md describing what each phase of the swarm lab
run does (bring-up, the variant-matrix round loop, the DR drill's nine
steps, the chaos phase, teardown), with the moments a backup-executing
systemd unit actually starts highlighted, and links to the per-step
scripts below the diagram.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The DR drill's simulated USB was a fixed 512M loop image (~449M usable
after LUKS2 + ext4), but the deployed local-2-device unit syncs the
whole pulled backup root, which after the remote pull holds two
machine-id trees (the manager's docker+secrets repos and the NFS
export repo). For an app the size of akaunting that overflows the
image mid-rsync and the drill fails with ENOSPC at step [5/9]. base.sh
now measures the pulled tree (du -sm) and passes 2x its size + headroom
(floor 2G) to 02_luks_device.sh, which sizes the loop image from that
argument; fallocate keeps the image sparse so the larger floor is free.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A wedged manager (act-swarm containerd stall) made the docker exec tar
block up to 900s; on a cancelled job that ate the runner's grace window
and the always() Cleanup step never ran, leaking the DiD cluster. Probe
the manager first (10s) and cap the copy at 120s with --kill-after so a
hung exec is force-reaped; teardown then always follows. Reports stay
collected before teardown, since they live inside the manager node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The provisioner copied the manager host_vars only to the workers, so the
service-loader preload on the nfs and backup nodes merged a defaults-only
applications dict (credentials lookups failed), and the bootstrap play
seeded /etc/hosts only on manager and nfs, so the backup node's msmtp
healthcheck died with EX_NOHOST. Both nodes now get the host_vars copy
and the app-domain hosts entries.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
swarm_stack_ready.sh piped 'docker stack services' through a process
substitution, so the timeout wrapper added in 8516e3167 turned a slow or
wedged manager into an empty listing, an empty not_running set and exit
0: the convergence gate would have gone green with zero running replicas.
Capture the listing, fail on a non-zero rc.
That exposed the second half: post_deploy gated the swarm wait on the
cluster-wide DEPLOYMENT_MODE, so forced-compose roles (matrix ansible
flavor, bigbluebutton) ran it against a stack that does not exist and
only passed because the old code swallowed the error. Gate it on
compose_mode, like the handlers already do.
teardown.sh: the NFS mountpoint and umount calls, the likeliest hang
sites, were the only ones left unwrapped, and the timeout around docker
inspect inverted the D-state warning (a hung inspect returned 124, the
if went false and the script reported a clean teardown). Warn on rc 0
and 124, and shrink the per-node budget so the script fits inside its
step timeout.
Swarm step budget: the bounded steps alone summed to 365 over a 360-min
cap, so cleanup was still starved. Deploy 240, diagnostics 15, playwright
15, rescue 25, cleanup 15.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
27 jobs of run 29147179243 burned their full 360-min budget. Three
systemic layers, all addressed here:
- The CLI e2e task ran the staged test.sh unbounded, and test.sh blocks
in 'systemctl start' on a oneshot backup unit with no JobTimeout; on a
wedged dockerd or a stalled NFS mount both freeze silently for hours.
The task now runs the script under 'timeout TEST_E2E_CLI_TIMEOUT'
(60 min via the timeout lookup, so TIMEOUT_FACTOR scales it) and the
unit starts are bounded by BKP_TEST_HEALTH_TIMEOUT.
- The converge probe re-ran the full per-service diagnostics (service ps,
service logs, inspect, journalctl) on EVERY retry, so 600 nominal
2-second attempts cost 25-33s each and silently outlived the step
timeout. The block is gone (the workflow-level container.py rescue
snapshot already collects it recursively on failure) and every docker
call is timeout-bounded. Same removal in the swarm collect script,
which keeps only the NFS and render evidence container.py does not see.
- Playwright-collect, rescue diagnostics and Cleanup had no
timeout-minutes, and teardown.sh calls docker against D-state nodes,
so a failed job hung there for hours until the job cap killed it
without artifacts. Every step is bounded now, every teardown call is
timeout-wrapped, and the deploy steps (swarm 270, compose 300) leave
room for diagnostics and cleanup below the 360-min cap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
routine/08_cleanup.sh becomes utils/clean/teardown.sh and topology/
becomes utils/topology/ so routine/ holds only the sequenced flow; all
consumers (workflow steps, compose/swarm.yml env guards, relative
sources, README) follow. The shell rescue collector and its lint are
gone: utils/diagnostics/container.py covers compose and swarm
identically. extend_inventory gains the backup-host placements the DR
drill deploys.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
extend_inventory maps svc-bkp-remote-2-local + svc-bkp-local-2-device onto
the backup host and write_extras configures them (backup_providers =
manager + NFS server IPs; device mount/target/source), so the matrix
deploy installs the real systemd units there. The drill now exercises
them: 01_ssh_trust.sh only installs the pull identity, the remote-2-local
unit pulls every provider (replacing both manual pull calls), and after
02_luks_device.sh simulates the USB plug the local-2-device unit performs
the device sync; the device recover passes the configured target subpath
via --device-target. Unit absence on the backup host is now a hard
failure since the roles deploy with the round.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add routine/backup/: base.sh drives a marker seeded into the live NFS
volume and the manager secrets through the whole chain (volume+secrets+nfs
backup units -> remote-2-local pull -> local-2-device LUKS 'USB') and back
via every recover.py (device -> local root -> nfs export, docker volume and
host secrets), asserting each marker survives. Each per-host step is one
in-node routine run over docker exec. extend_inventory maps the volume/
secrets/nfs backup roles onto their hosts so the units install; write_extras
generates the pull keypair; the INFINITO_SWARM_BACKUP_* keys are added to
default.env + the passthrough handler.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the per-node docker-run bring-up with a single node image
(compose/swarm/Dockerfile bakes python3/dnsmasq/openssh/rsync/cryptsetup)
declared in compose/swarm.yml, one compose up, and one ansible play
(compose/swarm/playbook.yml over ansible_connection: docker) that waits for
systemd, captures IPs, wires lab DNS and installs the .deb per node. Reshape
the harness into topology/ (naming SPOT), routine/ (01_bootstrap + the
sequenced 02-08 steps) and utils/ (clean/, collect/, shared helpers); the
workflow, matrix.py and Makefile follow the new paths. swarm init/join was
already ansible-owned (svc-swarm-node), corrected in the README diagram.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every Ansible rescue: block used to carry ad-hoc per-role failure dumps that
drifted apart and captured different subsets. Replace them all with one
diagnostics script (scripts/system/diagnostics/rescue.sh) that snapshots
containers (full logs plus inspect per container), swarm services (ps plus
logs per service), journal, host resources, runtime info and stats into a
per-run folder and always exits 1 so the deploy still fails. A new lint
enforces the shape: every rescue ends with the script, at most one
nocheck-justified extra task before it (postgres pg_hba dump, xwiki
extension-repo probe), and a nocheck rescue-recovery exemption for the one
block that must genuinely recover: keycloak object updates, where a
cross-node reschedule needs a swarm-wide node re-resolve plus a delegate_to
change that only a rescue can express; its recovery is kept and now also
captures diagnostics before the terminal failure.
The snapshot folder path is a group_vars SPOT: DIR_RESCUE_DIAGNOSTICS and
DIR_VAR_LIB live in group_vars/all/05_paths.yml as plain strings; new env
handlers derive INFINITO_RESCUE_DIAGNOSTICS_DIR and INFINITO_DIR_VAR_LIB
from it for the generated .env, replacing the default.env entries, and the
play-level environment forwards the value to every task host so rescue.sh
needs no second default. CI uploads the folder like the inventory and
playwright artifacts: compose copies it out of the deploy container in
dedicated.sh, swarm captures per node (manager and both workers) into
node-named subfolders.
Deleting the ad-hoc dumps orphaned helper files (postgres inspect_container,
terminator docker_diagnostics, peertube diagnostics, matomo bootstrap debug,
xwiki and openresty rescue includes) and 14 role vars; all removed. Generic
pieces of those dumps (runtime version and info, stats, dmesg oom evidence,
swapon) moved into the SPOT script instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
storage.nfs shrinks to the deployment-specific server address: export_base moves into svc-storage-nfs-server's services.yml (the server dictates where exports root) and the mount protocol version into svc-storage-nfs-client's services.yml (the client owns how it mounts). Both are read through utils.storage.nfs loaders by the nfs lookups, the env handler and 00_topology, eliminating the unrendered-variables class for these values; the group_vars assert shrinks accordingly and the swarm test extras stop restating role defaults.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
INFINITO_PYTHON_INSTALL_SCRIPT and INFINITO_DOCKER_CLI_INSTALL_SCRIPT join the default.env SPOT: the Dockerfile consumes them as required args (:? in the RUN as last defense), all three image builders (compose.yml, build.sh, push.sh) pass them hard-required, the shell consumers read the SPOT via default.env grep and the Makefile via BASH_ENV. INFINITO_SWARM_NFS_EXPORT_BASE is no longer a static default: a new env handler derives it from the deploy SPOT group_vars/all/15_storage.yml, 00_topology greps that file (and the STATE_SUBDIR constant) directly, and the swarm test extras stop restating the group_vars value. Stale hadolint directives are restored and recognised; leftover narration comments in the touched files are dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
04_bootstrap_nodes.sh consumed INFINITO_PACKAGE_INSTALL_SCRIPT and INFINITO_NODE_SRC_DIR from the ambient environment; local make targets provide them via BASH_ENV, but plain workflow steps do not, so every swarm CI job died at the required-var guard. Source load.sh explicitly like copy_worktree_to_node.sh.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
