mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-18 00:24:15 +00:00
Three steps of the DR drill, one theme: every step now proves what it claims. [9] restored into the volume's mountpoint while the stack was down - a bare directory, because docker mounts a volume declaring its own backing store only while a container holds it. The payload landed on the node disk and the real mount shadowed it on the next deploy: success reported, nothing delivered, and roughly a gigabyte added to the very device the drill later runs out of. It now deletes the marker server-side, binds the backing store at the mountpoint, runs the recover CLI through it, and asserts the marker on the NFS server's own disk. That also exercises the positive branch of the guard from 773a25ab6 - its refusal path is unit-tested, its pass-through was tested nowhere. When the manager exposes no backing store the branch skips with a reason: docker volume inspect is node-local, and with replicas: 1 the volume object usually lives elsewhere. [3b] replaces a proof that used to happen by accident. The double capture 2efc73d3b removed also showed, incidentally, that the volume unit really writes - the drill found its marker in the volume repo too. The new step checks it directly: the newest backup-docker-to-local generation must be at least as new as DRILL_START_TS (taken on the manager, the same clock that names generations) and hold a non-empty file under a volume subtree. Its failure message names the suspicion it exists for: an exclusion that swallowed every volume. With no volume unit deployed it skips, tracked by VOLUME_TRIGGERED alongside the existing SECRETS_TRIGGERED. [5] prints du -sm per machine and repo when the space precondition fails, so the next red run names what fills the pulled tree instead of leaving it to arithmetic. Four adversarial review rounds found five defects in this block and confirmed each fix: an NFS mount reconstructed from driver options the codebase never emits (it is a bind), the recover CLI's second positional passed a volume name where it takes a host, a missing volume object aborting the drill instead of skipping, and an umount ordered before the return-code check, which under set -e would have killed the drill on a busy mount with no diagnostic and left the bind behind. None of this has run in a swarm yet. Every failure path is loud and names its cause. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DR drill: backup + restore
Marker files seeded into the live NFS volume and into the manager's host
secrets travel the backup chain forward and are recovered back onto the
live instance; the drill passes only when every marker a deployed unit
captured survives the whole loop. base.sh runs the nine steps below; the
numbers in both diagrams are its [n/9] log markers.
Step sequence (who does what)
sequenceDiagram
autonumber
participant N as nfs-server
participant M as manager
participant B as backup host
participant U as LUKS "USB" (loop img on B)
Note over N,M: [1/9] seed markers
N->>N: marker → live NFS volume dir
M->>M: marker → host secrets dir
Note over N,M: [2/9] trigger deployed backup units
M->>M: svc-bkp-volume-2-local + secrets-2-local (when in the app closure)
N->>N: svc-bkp-nfs-2-local
Note over N,M: [3/9] locate the generation holding the marker
N-->>M: find marker in files/*/<volume>/ (volume-scoped)
Note over B: [4/9] pull via remote-2-local unit (rsync over ssh)
B->>N: pull nfs generations
B->>M: pull volume + secrets generations (only when M is a provider)
Note over B,U: [5/9] plug LUKS device, sync via local-2-device unit
B->>U: hard-linked snapshot
Note over M: [6/9] full disaster teardown
alt stack workload
M->>M: docker stack rm + wait
else node-local workload
M->>M: stop compose project on every node
end
Note over B,U: [7/9] recover device → local root
U-->>B: luksOpen + recover CLI → /var/tmp restore root
B->>B: drop pulled tree + loop image (free disk)
Note over N: [8/9] recover local root → live NFS export
B-->>N: tar the volume subtree → recover CLI
N->>N: [8b] restart ganesha, remount clients, coherence probe
Note over M: [9/9] recover docker volume + host secrets
B-->>M: volume generation → recover CLI
B-->>M: secrets generation → recover CLI
Note over M,N: matrix update pass redeploys, then verify_recovered_marker.sh asserts the live marker
Data flow (what is proven)
flowchart TB
subgraph forward["💾 Backup (solid: the deployed systemd units end to end)"]
direction TB
vol["📂 [1] Live NFS volume + docker volume<br/>marker seeded"]:::live
sec["🔑 [1] Host secrets on the manager<br/>(secrets, CA, ACME, node identity) marker seeded"]:::live
localbkp["🗄️ [2-3] Local backups on the NFS server<br/>(+ manager when its closure has the backup roles)<br/>volume / nfs / secrets generation snapshots"]:::store
pulled["🗄️ [4] Backup host<br/>pulled generations (remote-2-local unit, rsync over ssh, pull-only)"]:::store
usb["🔒 [5] Encrypted USB (LUKS)<br/>hard-linked snapshot (local-2-device unit)"]:::device
vol -->|backup units| localbkp
sec -->|secrets unit| localbkp
localbkp -->|remote pull| pulled -->|device sync| usb
end
teardown["🛑 [6] Full disaster:<br/>stack rm (stack workload) or<br/>compose stop on every node (node-local workload),<br/>then export wiped"]:::dead
subgraph recovery["♻️ Recovery (dashed: the recover CLI, newest generation only)"]
direction TB
restored["📁 [7] Restore root on the backup host<br/>newest snapshot after luksOpen (pulled tree + loop img freed)"]:::store
export["📂 [8] Live NFS export<br/>volume subtree rebuilt, ganesha + client mounts refreshed"]:::live
volume["📦 [9] Docker volume<br/>rebuilt on the manager"]:::live
secrets["🔑 [9] Host secrets<br/>rebuilt on the manager (marker cleared first)"]:::live
verified["✅ Update pass redeploys<br/>markers verified on the live instance"]:::ok
usb -.->|decrypt + recover| restored
restored -.->|restore export| export
restored -.->|restore volume| volume
restored -.->|restore secrets| secrets
export -.-> verified
volume -.-> verified
secrets -.-> verified
end
forward --> teardown --> recovery
classDef live fill:#f1f8f6,stroke:#d5e7e2,color:#455a64;
classDef store fill:#f3f7fb,stroke:#d6e2ee,color:#455a64;
classDef device fill:#fbf3f6,stroke:#eed9e2,color:#455a64;
classDef dead fill:#fdf2f2,stroke:#eddada,color:#455a64;
classDef ok fill:#f3f9f4,stroke:#d9ecdc,color:#455a64;
style forward fill:#ffffff,stroke:#e6e6e6;
style recovery fill:#ffffff,stroke:#e6e6e6;
Applicability
- The drill runs whenever the app declares an NFS-flagged volume
(
PRIMARY_NFS_VOLUME); apps without one skip it, since there is nothing to prove a restore against. - The manager legs of steps 2, 4 and 9 run only when the app's include
closure contains
svc-bkp-volume-2-local/svc-bkp-secrets-2-local.utils/tests/swarm/backup_repos.pyderives both the manager's inventory groups and theremote-2-localprovider list from that closure; apps without those roles prove the chain through the nfs repository. - The workload form only changes step 6: stack workloads get
docker stack rm,workload: node-localroles get their compose project stopped on every node. Every other step operates on volumes, the NFS export and the deployed backup units, independent of the orchestrator. - Runs once per app, in the variant-0 CI job, between the matrix's first deploy and its update pass.