mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-10 04:36:40 +00:00
kevinveenbirkenbachandClaude Opus 5
246558a13e
The converge gate polls STACK_NAME_PRIMARY_SERVICE_KEY. Four roles never create that name and each was given a static `workload: node-local` to make the gate skip itself:21fe9e984web-svc-cdn ships only cdn_build, never cdn_cdn2c6c8b763web-app-matrix names its services matrix_synapse and friends76508c946svc-runner deploys node-local compose projects per nodecb52a72f2bigbluebutton phantom service, and the DR drill must still run Four hand-written claims for one observable fact. The claim is also static where the truth is not: web-app-discourse renders `services: {}` only while its sidecars are shared, and job 94965456921 of run 31864448027 polled discourse_discourse ninety times for a stack that had none. Declaring the role node-local would have blinded the gate in the variants where its sidecars are local and a stack does exist. has_swarm_service asks the deployed stack for the exact name instead. Every caller runs behind `if rc == 0`, so an absent service is the role's shape and not a broken deploy. Exact-line matching, not docker's name filter, which matches substrings. No coverage is lost: 03_wait_converge cannot pass without that service, and 05/06/07 plus the backup drill only run once it has. The four roles keep the skip they were given; discourse and every future case get it without a declaration. The teardown branch in the drill keeps its meaning - stack rm where the service exists, compose stop per node where it does not. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Swarm test routine 🔁
Sequenced steps of the swarm lab run. Bring-up and the chaos phase are
workflow steps; the per-round loop in between is driven by
utils.tests.swarm.matrix (one round per variant of the
primary app). Highlighted boxes (⚡) are the moments a backup-executing
systemd unit actually starts. See ../README.md for
topology, naming SPOTs and helpers; the DR drill's own steps are
documented in backup/README.md.
flowchart TB
boot["Bring up the simulated cluster:<br/>build + start the 5 node containers, wait for systemd,<br/>wire lab DNS + IPs, install the project on every node"]
subgraph matrix["Variant matrix - one round per variant of the primary app"]
purge["Remove the prior round's stacks<br/>(rounds after the first)"]
prov["Provision the round's inventory and deploy the app stack"]
t_pre["⚡ Service-loader preload (MODE_BACKUP):<br/>volume + secrets units instant-start on the manager,<br/>nfs unit on the export host - pre-state snapshot<br/>before the deploy mutates anything"]
conv["Wait until every stack service converges"]
reach["Probe the app is reachable in-cluster"]
bkphost["Deploy the backup host<br/>(round 1 only: pull + device roles on bkp-01)"]
t_term["Terminator at that play's end:<br/>the lab deploys with --skip-backup, so MODE_BACKUP is false<br/>and remote-2-local stays out of the final run list -<br/>the drill is the only caller that starts the pull"]
subgraph drill["Disaster-recovery drill (round 1 only)"]
d1["Seed markers on the live NFS volume<br/>and in the manager secrets"]
t_d2["⚡ Trigger the deployed backup units again:<br/>volume + secrets on the manager,<br/>nfs on the export host - markers get captured"]
d3["Locate the backup generation holding the marker;<br/>assert the volume repo is fresh and non-empty<br/>(its NFS-backed volumes are excluded by design)"]
t_d4["⚡ Install the ssh pull identity, then start<br/>the remote-2-local unit: bkp-01 pulls every provider"]
t_d5["⚡ Space gate (pulled tree + watchdog floor,<br/>du breakdown on failure), then plug a LUKS loop<br/>device as simulated USB: the .mount unit fires<br/>the local-2-device unit"]
d6["Recover device -> local backup root<br/>(full LUKS open; the pulled tree is wiped first,<br/>so the device holds the only copy)"]
d7["Remove the stack, wipe the live NFS export<br/>(marker proven gone), recover it from the local<br/>root, restore NFS coherence"]
d8["Delete the marker server-side, recover the docker<br/>volume through its NFS-mounted mountpoint and<br/>re-verify on the server; recover the host secrets"]
d9["Redeploy the stack, assert the marker<br/>is back on the live volume"]
d1 --> t_d2 --> d3 --> t_d4 --> t_d5 --> d6 --> d7 --> d8 --> d9
end
purge --> prov --> t_pre --> conv --> reach --> bkphost --> t_term --> drill
drill -.->|next variant round| purge
end
subgraph chaos["Chaos phase (against the last round's stack)"]
seed["Seed a marker file on the app's NFS volume"]
drain["Drain the worker running the app,<br/>forcing a reschedule to another node"]
assertion["Assert the marker and reachability<br/>survived the reschedule"]
seed --> drain --> assertion
end
teardown["Tear down: collect diagnostics + Playwright<br/>reports on failure, kill nodes, remove the lab network"]
boot --> matrix --> chaos --> teardown
classDef trigger fill:#f7e28b,stroke:#b8860b,color:#000
class t_pre,t_d2,t_d4,t_d5 trigger
The nightly timers (volume 01:00, nfs 01:30, secrets 01:45, device 02:00, remote pull 00:30) run independently of every step above and never fire during a lab run.
Scripts per step: bring-up 01_bootstrap.sh - provision 02_provision_inventory.sh - converge 03_wait_converge.sh - reachability 04_verify_reachable.sh - drill backup/base.sh - marker 05_seed_content.sh - drain 06_drain_worker.sh - assert 07_assert_state.sh - purge ../utils/clean/purge_stacks.sh.