Files
computer-playbook/scripts/tests/deploy/swarm/routine/backup
kevinveenbirkenbachandClaude Opus 5 4fbde3b188 test(swarm-drill): prove the volume leg instead of staging a shadow restore
Three steps of the DR drill, one theme: every step now proves what it claims.

[9] restored into the volume's mountpoint while the stack was down - a bare directory,
because docker mounts a volume declaring its own backing store only while a container
holds it. The payload landed on the node disk and the real mount shadowed it on the next
deploy: success reported, nothing delivered, and roughly a gigabyte added to the very
device the drill later runs out of. It now deletes the marker server-side, binds the
backing store at the mountpoint, runs the recover CLI through it, and asserts the marker
on the NFS server's own disk. That also exercises the positive branch of the guard from
773a25ab6 - its refusal path is unit-tested, its pass-through was tested nowhere. When
the manager exposes no backing store the branch skips with a reason: docker volume
inspect is node-local, and with replicas: 1 the volume object usually lives elsewhere.

[3b] replaces a proof that used to happen by accident. The double capture 2efc73d3b
removed also showed, incidentally, that the volume unit really writes - the drill found
its marker in the volume repo too. The new step checks it directly: the newest
backup-docker-to-local generation must be at least as new as DRILL_START_TS (taken on
the manager, the same clock that names generations) and hold a non-empty file under a
volume subtree. Its failure message names the suspicion it exists for: an exclusion that
swallowed every volume. With no volume unit deployed it skips, tracked by
VOLUME_TRIGGERED alongside the existing SECRETS_TRIGGERED.

[5] prints du -sm per machine and repo when the space precondition fails, so the next red
run names what fills the pulled tree instead of leaving it to arithmetic.

Four adversarial review rounds found five defects in this block and confirmed each fix:
an NFS mount reconstructed from driver options the codebase never emits (it is a bind),
the recover CLI's second positional passed a volume name where it takes a host, a missing
volume object aborting the drill instead of skipping, and an umount ordered before the
return-code check, which under set -e would have killed the drill on a busy mount with no
diagnostic and left the bind behind.

None of this has run in a swarm yet. Every failure path is loud and names its cause.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-04 17:58:47 +02:00
..

DR drill: backup + restore

Marker files seeded into the live NFS volume and into the manager's host secrets travel the backup chain forward and are recovered back onto the live instance; the drill passes only when every marker a deployed unit captured survives the whole loop. base.sh runs the nine steps below; the numbers in both diagrams are its [n/9] log markers.

Step sequence (who does what)

sequenceDiagram
    autonumber
    participant N as nfs-server
    participant M as manager
    participant B as backup host
    participant U as LUKS "USB" (loop img on B)

    Note over N,M: [1/9] seed markers
    N->>N: marker → live NFS volume dir
    M->>M: marker → host secrets dir

    Note over N,M: [2/9] trigger deployed backup units
    M->>M: svc-bkp-volume-2-local + secrets-2-local (when in the app closure)
    N->>N: svc-bkp-nfs-2-local

    Note over N,M: [3/9] locate the generation holding the marker
    N-->>M: find marker in files/*/<volume>/ (volume-scoped)

    Note over B: [4/9] pull via remote-2-local unit (rsync over ssh)
    B->>N: pull nfs generations
    B->>M: pull volume + secrets generations (only when M is a provider)

    Note over B,U: [5/9] plug LUKS device, sync via local-2-device unit
    B->>U: hard-linked snapshot

    Note over M: [6/9] full disaster teardown
    alt stack workload
        M->>M: docker stack rm + wait
    else node-local workload
        M->>M: stop compose project on every node
    end

    Note over B,U: [7/9] recover device → local root
    U-->>B: luksOpen + recover CLI → /var/tmp restore root
    B->>B: drop pulled tree + loop image (free disk)

    Note over N: [8/9] recover local root → live NFS export
    B-->>N: tar the volume subtree → recover CLI
    N->>N: [8b] restart ganesha, remount clients, coherence probe

    Note over M: [9/9] recover docker volume + host secrets
    B-->>M: volume generation → recover CLI
    B-->>M: secrets generation → recover CLI

    Note over M,N: matrix update pass redeploys, then verify_recovered_marker.sh asserts the live marker

Data flow (what is proven)

flowchart TB
    subgraph forward["💾 Backup (solid: the deployed systemd units end to end)"]
        direction TB
        vol["📂 [1] Live NFS volume + docker volume<br/>marker seeded"]:::live
        sec["🔑 [1] Host secrets on the manager<br/>(secrets, CA, ACME, node identity) marker seeded"]:::live
        localbkp["🗄️ [2-3] Local backups on the NFS server<br/>(+ manager when its closure has the backup roles)<br/>volume / nfs / secrets generation snapshots"]:::store
        pulled["🗄️ [4] Backup host<br/>pulled generations (remote-2-local unit, rsync over ssh, pull-only)"]:::store
        usb["🔒 [5] Encrypted USB (LUKS)<br/>hard-linked snapshot (local-2-device unit)"]:::device
        vol -->|backup units| localbkp
        sec -->|secrets unit| localbkp
        localbkp -->|remote pull| pulled -->|device sync| usb
    end

    teardown["🛑 [6] Full disaster:<br/>stack rm (stack workload) or<br/>compose stop on every node (node-local workload),<br/>then export wiped"]:::dead

    subgraph recovery["♻️ Recovery (dashed: the recover CLI, newest generation only)"]
        direction TB
        restored["📁 [7] Restore root on the backup host<br/>newest snapshot after luksOpen (pulled tree + loop img freed)"]:::store
        export["📂 [8] Live NFS export<br/>volume subtree rebuilt, ganesha + client mounts refreshed"]:::live
        volume["📦 [9] Docker volume<br/>rebuilt on the manager"]:::live
        secrets["🔑 [9] Host secrets<br/>rebuilt on the manager (marker cleared first)"]:::live
        verified["✅ Update pass redeploys<br/>markers verified on the live instance"]:::ok
        usb -.->|decrypt + recover| restored
        restored -.->|restore export| export
        restored -.->|restore volume| volume
        restored -.->|restore secrets| secrets
        export -.-> verified
        volume -.-> verified
        secrets -.-> verified
    end

    forward --> teardown --> recovery

    classDef live fill:#f1f8f6,stroke:#d5e7e2,color:#455a64;
    classDef store fill:#f3f7fb,stroke:#d6e2ee,color:#455a64;
    classDef device fill:#fbf3f6,stroke:#eed9e2,color:#455a64;
    classDef dead fill:#fdf2f2,stroke:#eddada,color:#455a64;
    classDef ok fill:#f3f9f4,stroke:#d9ecdc,color:#455a64;
    style forward fill:#ffffff,stroke:#e6e6e6;
    style recovery fill:#ffffff,stroke:#e6e6e6;

Applicability

  • The drill runs whenever the app declares an NFS-flagged volume (PRIMARY_NFS_VOLUME); apps without one skip it, since there is nothing to prove a restore against.
  • The manager legs of steps 2, 4 and 9 run only when the app's include closure contains svc-bkp-volume-2-local / svc-bkp-secrets-2-local. utils/tests/swarm/backup_repos.py derives both the manager's inventory groups and the remote-2-local provider list from that closure; apps without those roles prove the chain through the nfs repository.
  • The workload form only changes step 6: stack workloads get docker stack rm, workload: node-local roles get their compose project stopped on every node. Every other step operates on volumes, the NFS export and the deployed backup units, independent of the orchestrator.
  • Runs once per app, in the variant-0 CI job, between the matrix's first deploy and its update pass.