mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-08 03:38:23 +00:00
kevinveenbirkenbachandClaude Opus 5
3748e3ee00
Two places in the DR drill ruled on a systemd unit without looking at what the unit was doing. [8b/9] took the exit code of `systemctl try-restart nfs-ganesha` as the verdict. The unit carries Restart=on-failure from the role's own drop-in, and that drop-in also hangs a 20 second readiness probe on port 2049 as ExecStartPost. Right after the backing FS is wiped and restored, that probe runs out while the container still waits on its own stale NFS mount, so systemd kills the daemon it just started and reports a failed start - then restarts it three seconds later and succeeds. Measured in run 34699757369, jobs 103572236910 (Matrix#0) and 103638497583 (Gitea#0): 'Control process exited, code=exited, status=124', 'Failed to start', then 'Scheduled restart job, restart counter is at 1'. The restart attempt may now fail; the assertion is a bounded wait on is-active, and it aborts loudly with systemctl status when the unit never arrives. The coherence probe that follows is unchanged and still has to pass. [0/9] waited for in-flight backups with `systemctl is-active --quiet 'svc-bkp-*.service'` and did not wait. In job 103588089327 (Moodle#0) the volume unit ran from 01:00:56 to 01:01:12 while that loop returned after 0.36s for four nodes. The drill then attached to the running job instead of starting its own, and [3b/9] refused the generation it found because it predated the drill by ten seconds. That second change is a hardening, not a proven fix: the reason the loop returned early is still unknown. The obvious explanation does not hold - the systemd manual says is-active "Returns an exit code 0 if at least one is active", so a pattern matching one running unit among four idle ones should have blocked. list-units --state=active expresses the same intent without relying on that semantics. Closing the gap needs the call run on a node with exactly that mix, which no local container offers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Swarm test routine 🔁
Sequenced steps of the swarm lab run. Bring-up and the chaos phase are
workflow steps; the per-round loop in between is driven by
utils.tests.swarm.matrix (one round per variant of the
primary app). Highlighted boxes (⚡) are the moments a backup-executing
systemd unit actually starts. See ../README.md for
topology, naming SPOTs and helpers; the DR drill's own steps are
documented in backup/README.md.
flowchart TB
boot["Bring up the simulated cluster:<br/>build + start the 5 node containers, wait for systemd,<br/>wire lab DNS + IPs, install the project on every node"]
subgraph matrix["Variant matrix - one round per variant of the primary app"]
purge["Remove the prior round's stacks<br/>(rounds after the first)"]
prov["Provision the round's inventory and deploy the app stack"]
t_pre["⚡ Service-loader preload (MODE_BACKUP):<br/>volume + secrets units instant-start on the manager,<br/>nfs unit on the export host - pre-state snapshot<br/>before the deploy mutates anything"]
conv["Wait until every stack service converges"]
reach["Probe the app is reachable in-cluster"]
bkphost["Deploy the backup host<br/>(round 1 only: pull + device roles on bkp-01)"]
t_term["Terminator at that play's end:<br/>the lab deploys with --skip-backup, so MODE_BACKUP is false<br/>and remote-2-local stays out of the final run list -<br/>the drill is the only caller that starts the pull"]
subgraph drill["Disaster-recovery drill (round 1 only)"]
d1["Seed markers on the live NFS volume<br/>and in the manager secrets"]
t_d2["⚡ Trigger the deployed backup units again:<br/>volume + secrets on the manager,<br/>nfs on the export host - markers get captured"]
d3["Locate the backup generation holding the marker;<br/>assert the volume repo is fresh and non-empty<br/>(its NFS-backed volumes are excluded by design)"]
t_d4["⚡ Install the ssh pull identity, then start<br/>the remote-2-local unit: bkp-01 pulls every provider"]
t_d5["⚡ Space gate (pulled tree + watchdog floor,<br/>du breakdown on failure), then plug a LUKS loop<br/>device as simulated USB: the .mount unit fires<br/>the local-2-device unit"]
d6["Recover device -> local backup root<br/>(full LUKS open; the pulled tree is wiped first,<br/>so the device holds the only copy)"]
d7["Remove the stack, wipe the live NFS export<br/>(marker proven gone), recover it from the local<br/>root, restore NFS coherence"]
d8["Delete the marker server-side, recover the docker<br/>volume through its NFS-mounted mountpoint and<br/>re-verify on the server; recover the host secrets"]
d9["Redeploy the stack, assert the marker<br/>is back on the live volume"]
d1 --> t_d2 --> d3 --> t_d4 --> t_d5 --> d6 --> d7 --> d8 --> d9
end
purge --> prov --> t_pre --> conv --> reach --> bkphost --> t_term --> drill
drill -.->|next variant round| purge
end
subgraph chaos["Chaos phase (against the last round's stack)"]
seed["Seed a marker file on the app's NFS volume"]
drain["Drain the worker running the app,<br/>forcing a reschedule to another node"]
assertion["Assert the marker and reachability<br/>survived the reschedule"]
seed --> drain --> assertion
end
teardown["Tear down: collect diagnostics + Playwright<br/>reports on failure, kill nodes, remove the lab network"]
boot --> matrix --> chaos --> teardown
classDef trigger fill:#f7e28b,stroke:#b8860b,color:#000
class t_pre,t_d2,t_d4,t_d5 trigger
The nightly timers (volume 01:00, nfs 01:30, secrets 01:45, device 02:00, remote pull 00:30) run independently of every step above and never fire during a lab run.
Scripts per step: bring-up 01_bootstrap.sh - provision 02_provision_inventory.sh - converge 03_wait_converge.sh - reachability 04_verify_reachable.sh - drill backup/base.sh - marker 05_seed_content.sh - drain 06_drain_worker.sh - assert 07_assert_state.sh - purge ../utils/clean/purge_stacks.sh.