kevinveenbirkenbachandClaude Opus 5
3748e3ee00
Two places in the DR drill ruled on a systemd unit without looking at what the unit was doing. [8b/9] took the exit code of `systemctl try-restart nfs-ganesha` as the verdict. The unit carries Restart=on-failure from the role's own drop-in, and that drop-in also hangs a 20 second readiness probe on port 2049 as ExecStartPost. Right after the backing FS is wiped and restored, that probe runs out while the container still waits on its own stale NFS mount, so systemd kills the daemon it just started and reports a failed start - then restarts it three seconds later and succeeds. Measured in run 34699757369, jobs 103572236910 (Matrix#0) and 103638497583 (Gitea#0): 'Control process exited, code=exited, status=124', 'Failed to start', then 'Scheduled restart job, restart counter is at 1'. The restart attempt may now fail; the assertion is a bounded wait on is-active, and it aborts loudly with systemctl status when the unit never arrives. The coherence probe that follows is unchanged and still has to pass. [0/9] waited for in-flight backups with `systemctl is-active --quiet 'svc-bkp-*.service'` and did not wait. In job 103588089327 (Moodle#0) the volume unit ran from 01:00:56 to 01:01:12 while that loop returned after 0.36s for four nodes. The drill then attached to the running job instead of starting its own, and [3b/9] refused the generation it found because it predated the drill by ten seconds. That second change is a hardening, not a proven fix: the reason the loop returned early is still unknown. The obvious explanation does not hold - the systemd manual says is-active "Returns an exit code 0 if at least one is active", so a pattern matching one running unit among four idle ones should have blocked. list-units --state=active expresses the same intent without relying on that semantics. Closing the gap needs the call run on a node with exactly that mix, which no local container offers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Swarm deploy test harness
Simulates a full Docker Swarm cluster on a single CI runner (or host via
make swarm-zombie / make roundtrip) and deploys one application
through it end to end: provision an inventory, deploy every variant
round with the in-deploy Playwright e2e, run the backup+restore DR drill
between rounds, then prove state survives a worker reschedule.
The cluster consists of privileged systemd containers on a dedicated lab
bridge (192.168.244.0/24, MTU 1400): three swarm nodes (1 manager +
2 workers), a non-swarm NFSv4 server serving the shared volume storage,
and a non-swarm backup host for the DR drill. The node image
(compose/swarm/Dockerfile)
bakes python3 + dnsmasq and masks the zfs boot units the parent image's
userland enables; the containers and the lab network are declared
in compose/swarm/compose.yml (project
${SWARM_NAME}, backup host behind the drill profile) and started by
one docker compose up in routine/01_bootstrap.sh. Lab DNS is provisioned
by compose/swarm/playbook.yml over ansible_connection: docker (Ansible never
uses ssh here); swarm init/join itself happens inside the deploy via the
svc-swarm-node role, not in any bring-up script.
Flow
flowchart TB
subgraph infra["🥾 Cluster bring-up (per matrix app)"]
direction TB
topo["🗺️ utils/topology/base + export<br/>SPOT node names + NFS paths → GITHUB_ENV"]:::infra
cluster["🐝 01_bootstrap.sh<br/>host side: pre-clean, compose build + up<br/>all 5 nodes + lab network, sudo .deb build"]:::infra
boot["🧱 compose/swarm/playbook.yml (over docker connection)<br/>systemd wait, NFS-export wipe,<br/>lab DNS, repo + .deb install on every node"]:::infra
topo --> cluster --> boot
end
subgraph deploy["🚀 Matrix deploy (per variant round)"]
direction TB
prov["📋 02_provision_inventory<br/>per-round inventory + extend_inventory + write_extras"]:::deploy
dep["📦 deploy round N: swarm init/join (svc-swarm-node)<br/>+ docker stack deploy + Playwright e2e"]:::deploy
conv["✅ 03_wait_converge + 04_verify_reachable"]:::deploy
purge["🧽 purge_stacks (between rounds)"]:::deploy
subgraph drill["🔐 routine/backup/base.sh (DR drill, once, after round 1)"]
direction TB
s1["1️⃣ seed markers into the live NFS volume<br/>and the manager secrets"]:::drill
s2["2️⃣ trigger backup units<br/>volume + secrets @ manager, nfs-2-local @ NFS server"]:::drill
s3["3️⃣ locate the generation holding the marker<br/>3️⃣b assert the volume repo is fresh + non-empty<br/>(its NFS-backed volumes are excluded by design)"]:::drill
s4["4️⃣ pull to the backup host<br/>remote-2-local over the user-backup ssh wrapper"]:::drill
s5["5️⃣ space gate: pulled tree + watchdog floor,<br/>du breakdown on failure; then LUKS loop 'USB'<br/>via the local-2-device unit"]:::drill
s6["6️⃣ full disaster: stack rm"]:::drill
s7["7️⃣ wipe the pulled tree, recover device → local root<br/>(luksOpen; the device holds the only copy left)"]:::drill
s8["8️⃣ wipe the live export (marker proven gone),<br/>recover root → export, restore NFS coherence"]:::drill
s9["9️⃣ delete the marker server-side, recover the volume<br/>through its NFS-mounted mountpoint, re-verify on the<br/>server; recover the host secrets"]:::drill
s1 --> s2 --> s3 --> s4 --> s5 --> s6 --> s7 --> s8 --> s9
end
prov --> dep --> conv --> drill --> purge
end
subgraph chaos["💪 Reschedule chaos (after the matrix)"]
direction TB
seed["🌱 05_seed_content<br/>marker on the NFS volume"]:::chaos
drain["🩸 06_drain_worker<br/>drain the app's worker, force reschedule"]:::chaos
assert["🔎 07_assert_state<br/>marker survived + app reachable on new node"]:::chaos
seed --> drain --> assert
end
subgraph teardown["🧹 Always"]
direction TB
diag["📊 collect: diagnostics / playwright reports +<br/>workflow-level recursive container.py rescue snapshot"]:::teardown
clean["🗑️ utils/clean/teardown (kill nodes + lab network)"]:::teardown
diag --> clean
end
subgraph lab["🌐 Lab bridge 192.168.244.0/24 (MTU 1400)"]
direction LR
nMgr["👑 swarm-mgr-01 (manager)<br/>stacks, volume-2-local, user-backup, sshd"]:::mgr
nW1["⚙️ swarm-wrk-01 (worker)"]:::wrk
nW2["⚙️ swarm-wrk-02 (worker)"]:::wrk
nNfs["📦 nfs-server (non-swarm)<br/>NFS export, nfs-2-local, sshd"]:::nfsnode
nBkp["💽 backup-host (non-swarm)<br/>deployed remote-2-local + local-2-device units,<br/>LUKS device"]:::bkpnode
end
infra --> deploy --> chaos --> teardown
cluster -->|▶️ compose up all 5| nMgr & nW1 & nW2 & nNfs & nBkp
boot -->|🌐 hosts + dnsmasq| nMgr & nW1 & nW2 & nNfs
boot -->|📥 repo + .deb install| nMgr & nW1 & nW2 & nNfs & nBkp
dep -->|🚀 swarm init/join + docker stack deploy| nMgr
s2 -->|⏱️ trigger volume + nfs units| nMgr & nNfs
s4 -->|⬇️ pull backup@source| nBkp
s7 -->|♻️ wipe + recover export| nNfs
s8 -->|🚀 redeploy| nMgr
seed -->|🌱 marker on export| nNfs
drain -->|🩸 drain a worker| nMgr
clean -->|🗑️ kill + remove| nMgr & nW1 & nW2 & nNfs & nBkp
nMgr -. 📨 schedules app tasks .-> nW1 & nW2
nMgr -. 💾 NFS-backed volumes .-> nNfs
nW1 -. 🔗 NFS mount .-> nNfs
nW2 -. 🔗 NFS mount .-> nNfs
nBkp -. 🔑 pull over ssh .-> nMgr & nNfs
classDef infra fill:#f3f7fb,stroke:#d6e2ee,color:#455a64;
classDef deploy fill:#f3f9f4,stroke:#d9ecdc,color:#455a64;
classDef drill fill:#fbf6f0,stroke:#eee1cf,color:#455a64;
classDef chaos fill:#fbf4f4,stroke:#eed9d9,color:#455a64;
classDef teardown fill:#f7f8f9,stroke:#e3e6e9,color:#455a64;
classDef mgr fill:#f6f3fb,stroke:#e0d7ee,color:#455a64;
classDef wrk fill:#f8f4fa,stroke:#e6dcec,color:#455a64;
classDef nfsnode fill:#f1f8f6,stroke:#d5e7e2,color:#455a64;
classDef bkpnode fill:#fbf3f6,stroke:#eed9e2,color:#455a64;
style infra fill:#ffffff,stroke:#e6e6e6;
style deploy fill:#ffffff,stroke:#e6e6e6;
style drill fill:#ffffff,stroke:#e6e6e6;
style chaos fill:#ffffff,stroke:#e6e6e6;
style teardown fill:#ffffff,stroke:#e6e6e6;
style lab fill:#ffffff,stroke:#e6e6e6;
The drill proves the svc-bkp-* chain forward through the DEPLOYED
systemd units (nfs on the export host, volume + secrets on the manager when
the app's include closure pulls those roles in, remote pull + device sync on
the backup host) and every
recover.py back (device -> local root -> NFS export, docker volume and
host secrets into the live system paths), with marker files that must
survive the whole loop: the matrix update pass boots the recovered stack
and verify_recovered_marker.sh asserts the marker on the live volume. It
reuses the round-1 stack instead of spinning a dedicated cluster, and skips
cleanly when the app declares no NFS-flagged volume. The backup host is started by routine/01_bootstrap.sh (drill
profile) and receives its two roles via extend_inventory; the pull
trust (backup keypair) and the device mount/target/source come from
utils/tests/swarm/write/extras.py, while
remote-2-local.backup_providers is derived per round by
utils/tests/swarm/backup_repos.py from the same placements
extend_inventory turns into inventory groups, and reaches the deploy
through the provisioner's host_vars merge.
Scripts
The sequenced flow lives in routine/, the naming SPOT in utils/topology/,
shared helpers in utils/, and
the cluster declaration (image, containers, network, DNS play) in
compose/swarm/ + compose/swarm/compose.yml.
| Stage | Script | Purpose |
|---|---|---|
| loop | scripts/tests/deploy/distros.sh |
SPOT: run the drill once per distro in random order under the shared time budget, then table every outcome (✅ passed / ❌ failed / 🟦 skipped) into the job summary |
| loop | routine/00_one.sh |
the whole drill for one distro; collect + teardown run from its EXIT trap |
| bring-up | utils/topology/base.sh |
SPOT: node names + NFS export/state paths (sourced, not run) |
| bring-up | utils/topology/export.sh |
write the topology SPOT into $GITHUB_ENV |
| bring-up | compose/swarm/compose.yml + compose/swarm/Dockerfile |
declare the 5 node containers, node image + lab network (compose SPOT) |
| bring-up | routine/01_bootstrap.sh |
host side: pre-clean, compose build + one compose up, sudo .deb build, then the play |
| bring-up | compose/swarm/playbook.yml |
node side over docker connection: systemd wait, NFS-export wipe, lab DNS, repo + .deb install |
| deploy | routine/02_provision_inventory.sh |
provision the per-round inventory |
| deploy | routine/03_wait_converge.sh |
wait for every stack service to converge |
| deploy | routine/04_verify_reachable.sh |
probe the app is reachable in-cluster |
| deploy | routine/backup/base.sh (+ per-host routines in routine/backup/) |
backup+restore DR drill between rounds |
| deploy | utils/clean/purge_stacks.sh |
remove prior-round stacks between rounds |
| chaos | routine/05_seed_content.sh |
seed a marker on the NFS volume |
| chaos | routine/06_drain_worker.sh |
drain the app's worker + force reschedule |
| chaos | routine/07_assert_state.sh |
assert the marker + reachability survived |
| teardown | utils/collect/diagnostics.sh |
write stack/service diagnostics on failure, one file per topic under the rescue dir |
| teardown | ../utils/rescue_index.sh |
list the collected rescue tree so a reader knows what the artifact holds |
| teardown | utils/collect/playwright_reports.sh |
pull Playwright reports from the manager into /tmp/playwright-artifacts/<distro>/<app> |
| teardown | utils/collect/topology_summary.sh |
render the node list + service placement into $GITHUB_STEP_SUMMARY |
| teardown | utils/clean/teardown.sh |
kill the nodes + remove the lab network |
| helper | utils/_context.sh |
per-app facts (entity, service, NFS volume, probes) |
| helper | utils/unmount/nfs_mounts.sh |
best-effort NFS unmount before node removal |
| helper | utils/unmount/host_state.sh |
release the host mount of the shared state so the next cycle cannot inherit it |
| recovery | utils/clean/all.sh |
reap leftover clusters across every cluster id |
| recovery | utils/clean/stale_nfs.sh |
recover stale in-namespace NFS mounts |
The matrix orchestrator
(utils/tests/swarm/matrix.py) drives the deploy stage
per variant round; routine/00_one.sh drives the surrounding stages for one
distro, and the workflow .github/workflows/call-test-deploy.yml drives the
distro loop. Run one app locally with
make swarm-zombie app=<id> (keeps the cluster for inspection) or the
whole matrix via make roundtrip.