mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-10 04:36:40 +00:00
kevinveenbirkenbachandClaude Opus 5
c9d8e85567
The role README claimed the backup machinery "operates transparently against the mount paths" and demoted the source-of-truth rule to advice in a prose box - both written before 2efc73d3b made that rule the code's behaviour. It now states what is enforced: this role skips export-backed volumes, svc-bkp-nfs-2-local is their single capture, and manager-pinned roles, roles forcing another compose_mode_force and nfs: false opt-outs stay covered here. It also names the limit the code actually has - the exclusion checks that some host is in the svc-bkp-nfs-2-local inventory group, not that the role ever ran successfully - because a README promising more than the code delivers is worse than one promising less. Its Schema section carried the role's timer flow; it now carries the mechanism the change set is about: meta/volumes.yml -> swarm_nfs_backed() as the single predicate -> its two consumers (the compose rewrite and the backup exclusion) -> the inventory-group gate -> the export host's single capture -> the restore path with its mountpoint probe -> the drill's [3b] guard against over-exclusion. The two swarm READMEs drew the drill with eight steps and pre-773a25ab6 semantics. Both now draw the nine that exist, including what makes the proof a proof: the pulled tree is wiped before the device recover, so the device holds the only copy left; the live export is wiped with the marker proven gone before it is restored; and [9] deletes the marker server-side before restoring the volume through its bound backing store. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
69 lines
4.3 KiB
Markdown
69 lines
4.3 KiB
Markdown
# Swarm test routine 🔁
|
|
|
|
Sequenced steps of the swarm lab run. Bring-up and the chaos phase are
|
|
workflow steps; the per-round loop in between is driven by
|
|
`utils.tests.swarm.matrix` (one round per variant of the
|
|
primary app). Highlighted boxes (⚡) are the moments a backup-executing
|
|
systemd unit actually starts. See [../README.md](../README.md) for
|
|
topology, naming SPOTs and helpers; the DR drill's own steps are
|
|
documented in [backup/README.md](backup/README.md).
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
boot["Bring up the simulated cluster:<br/>build + start the 5 node containers, wait for systemd,<br/>wire lab DNS + IPs, install the project on every node"]
|
|
|
|
subgraph matrix["Variant matrix - one round per variant of the primary app"]
|
|
purge["Remove the prior round's stacks<br/>(rounds after the first)"]
|
|
prov["Provision the round's inventory and deploy the app stack"]
|
|
t_pre["⚡ Service-loader preload (MODE_BACKUP):<br/>volume + secrets units instant-start on the manager,<br/>nfs unit on the export host - pre-state snapshot<br/>before the deploy mutates anything"]
|
|
conv["Wait until every stack service converges"]
|
|
reach["Probe the app is reachable in-cluster"]
|
|
bkphost["Deploy the backup host<br/>(round 1 only: pull + device roles on bkp-01)"]
|
|
t_term["Terminator at that play's end:<br/>the lab deploys with --skip-backup, so MODE_BACKUP is false<br/>and remote-2-local stays out of the final run list -<br/>the drill is the only caller that starts the pull"]
|
|
|
|
subgraph drill["Disaster-recovery drill (round 1 only)"]
|
|
d1["Seed markers on the live NFS volume<br/>and in the manager secrets"]
|
|
t_d2["⚡ Trigger the deployed backup units again:<br/>volume + secrets on the manager,<br/>nfs on the export host - markers get captured"]
|
|
d3["Locate the backup generation holding the marker;<br/>assert the volume repo is fresh and non-empty<br/>(its NFS-backed volumes are excluded by design)"]
|
|
t_d4["⚡ Install the ssh pull identity, then start<br/>the remote-2-local unit: bkp-01 pulls every provider"]
|
|
t_d5["⚡ Space gate (pulled tree + watchdog floor,<br/>du breakdown on failure), then plug a LUKS loop<br/>device as simulated USB: the .mount unit fires<br/>the local-2-device unit"]
|
|
d6["Recover device -> local backup root<br/>(full LUKS open; the pulled tree is wiped first,<br/>so the device holds the only copy)"]
|
|
d7["Remove the stack, wipe the live NFS export<br/>(marker proven gone), recover it from the local<br/>root, restore NFS coherence"]
|
|
d8["Delete the marker server-side, recover the docker<br/>volume through its NFS-mounted mountpoint and<br/>re-verify on the server; recover the host secrets"]
|
|
d9["Redeploy the stack, assert the marker<br/>is back on the live volume"]
|
|
d1 --> t_d2 --> d3 --> t_d4 --> t_d5 --> d6 --> d7 --> d8 --> d9
|
|
end
|
|
|
|
purge --> prov --> t_pre --> conv --> reach --> bkphost --> t_term --> drill
|
|
drill -.->|next variant round| purge
|
|
end
|
|
|
|
subgraph chaos["Chaos phase (against the last round's stack)"]
|
|
seed["Seed a marker file on the app's NFS volume"]
|
|
drain["Drain the worker running the app,<br/>forcing a reschedule to another node"]
|
|
assertion["Assert the marker and reachability<br/>survived the reschedule"]
|
|
seed --> drain --> assertion
|
|
end
|
|
|
|
teardown["Tear down: collect diagnostics + Playwright<br/>reports on failure, kill nodes, remove the lab network"]
|
|
|
|
boot --> matrix --> chaos --> teardown
|
|
|
|
classDef trigger fill:#f7e28b,stroke:#b8860b,color:#000
|
|
class t_pre,t_d2,t_d4,t_d5 trigger
|
|
```
|
|
|
|
The nightly timers (volume 01:00, nfs 01:30, secrets 01:45, device
|
|
02:00, remote pull 00:30) run independently of every step above and
|
|
never fire during a lab run.
|
|
|
|
Scripts per step: bring-up [01_bootstrap.sh](01_bootstrap.sh) -
|
|
provision [02_provision_inventory.sh](02_provision_inventory.sh) -
|
|
converge [03_wait_converge.sh](03_wait_converge.sh) -
|
|
reachability [04_verify_reachable.sh](04_verify_reachable.sh) -
|
|
drill [backup/base.sh](backup/base.sh) -
|
|
marker [05_seed_content.sh](05_seed_content.sh) -
|
|
drain [06_drain_worker.sh](06_drain_worker.sh) -
|
|
assert [07_assert_state.sh](07_assert_state.sh) -
|
|
purge [../utils/clean/purge_stacks.sh](../utils/clean/purge_stacks.sh).
|