# Swarm deploy test harness

Simulates a full Docker Swarm cluster on a single CI runner (or host via
`make swarm-zombie` / `make roundtrip`) and deploys one application
through it end to end: provision an inventory, deploy every variant
round with the in-deploy Playwright e2e, run the backup+restore DR drill
between rounds, then prove state survives a worker reschedule.

The cluster consists of privileged systemd containers on a dedicated lab
bridge (`192.168.244.0/24`, MTU 1400): three swarm nodes (1 manager +
2 workers), a non-swarm NFSv4 server serving the shared volume storage,
and a non-swarm backup host for the DR drill. The node image
([`compose/swarm/Dockerfile`](../../../../compose/swarm/Dockerfile))
bakes python3 + dnsmasq and masks the zfs boot units the parent image's
userland enables; the containers and the lab network are declared
in [`compose/swarm/compose.yml`](../../../../compose/swarm/compose.yml) (project
`${SWARM_NAME}`, backup host behind the `drill` profile) and started by
one `docker compose up` in `routine/01_bootstrap.sh`. Lab DNS is provisioned
by `compose/swarm/playbook.yml` over `ansible_connection: docker` (Ansible never
uses ssh here); swarm init/join itself happens inside the deploy via the
`svc-swarm-node` role, not in any bring-up script.

## Flow

```mermaid
flowchart TB
    subgraph infra["🥾 Cluster bring-up (per matrix app)"]
        direction TB
        topo["🗺️ utils/topology/base + export<br/>SPOT node names + NFS paths → GITHUB_ENV"]:::infra
        cluster["🐝 01_bootstrap.sh<br/>host side: pre-clean, compose build + up<br/>all 5 nodes + lab network, sudo .deb build"]:::infra
        boot["🧱 compose/swarm/playbook.yml (over docker connection)<br/>systemd wait, NFS-export wipe,<br/>lab DNS, repo + .deb install on every node"]:::infra
        topo --> cluster --> boot
    end

    subgraph deploy["🚀 Matrix deploy (per variant round)"]
        direction TB
        prov["📋 02_provision_inventory<br/>per-round inventory + extend_inventory + write_extras"]:::deploy
        dep["📦 deploy round N: swarm init/join (svc-swarm-node)<br/>+ docker stack deploy + Playwright e2e"]:::deploy
        conv["✅ 03_wait_converge + 04_verify_reachable"]:::deploy
        purge["🧽 purge_stacks (between rounds)"]:::deploy

        subgraph drill["🔐 routine/backup/base.sh (DR drill, once, after round 1)"]
            direction TB
            s1["1️⃣ seed markers into the live NFS volume<br/>and the manager secrets"]:::drill
            s2["2️⃣ trigger backup units<br/>volume + secrets @ manager, nfs-2-local @ NFS server"]:::drill
            s3["3️⃣ locate the generation holding the marker<br/>3️⃣b assert the volume repo is fresh + non-empty<br/>(its NFS-backed volumes are excluded by design)"]:::drill
            s4["4️⃣ pull to the backup host<br/>remote-2-local over the user-backup ssh wrapper"]:::drill
            s5["5️⃣ space gate: pulled tree + watchdog floor,<br/>du breakdown on failure; then LUKS loop 'USB'<br/>via the local-2-device unit"]:::drill
            s6["6️⃣ full disaster: stack rm"]:::drill
            s7["7️⃣ wipe the pulled tree, recover device → local root<br/>(luksOpen; the device holds the only copy left)"]:::drill
            s8["8️⃣ wipe the live export (marker proven gone),<br/>recover root → export, restore NFS coherence"]:::drill
            s9["9️⃣ delete the marker server-side, recover the volume<br/>through its NFS-mounted mountpoint, re-verify on the<br/>server; recover the host secrets"]:::drill
            s1 --> s2 --> s3 --> s4 --> s5 --> s6 --> s7 --> s8 --> s9
        end

        prov --> dep --> conv --> drill --> purge
    end

    subgraph chaos["💪 Reschedule chaos (after the matrix)"]
        direction TB
        seed["🌱 05_seed_content<br/>marker on the NFS volume"]:::chaos
        drain["🩸 06_drain_worker<br/>drain the app's worker, force reschedule"]:::chaos
        assert["🔎 07_assert_state<br/>marker survived + app reachable on new node"]:::chaos
        seed --> drain --> assert
    end

    subgraph teardown["🧹 Always"]
        direction TB
        diag["📊 collect: diagnostics / playwright reports +<br/>workflow-level recursive container.py rescue snapshot"]:::teardown
        clean["🗑️ utils/clean/teardown (kill nodes + lab network)"]:::teardown
        diag --> clean
    end

    subgraph lab["🌐 Lab bridge 192.168.244.0/24 (MTU 1400)"]
        direction LR
        nMgr["👑 swarm-mgr-01 (manager)<br/>stacks, volume-2-local, user-backup, sshd"]:::mgr
        nW1["⚙️ swarm-wrk-01 (worker)"]:::wrk
        nW2["⚙️ swarm-wrk-02 (worker)"]:::wrk
        nNfs["📦 nfs-server (non-swarm)<br/>NFS export, nfs-2-local, sshd"]:::nfsnode
        nBkp["💽 backup-host (non-swarm)<br/>deployed remote-2-local + local-2-device units,<br/>LUKS device"]:::bkpnode
    end

    infra --> deploy --> chaos --> teardown

    cluster -->|▶️ compose up all 5| nMgr & nW1 & nW2 & nNfs & nBkp
    boot -->|🌐 hosts + dnsmasq| nMgr & nW1 & nW2 & nNfs
    boot -->|📥 repo + .deb install| nMgr & nW1 & nW2 & nNfs & nBkp
    dep -->|🚀 swarm init/join + docker stack deploy| nMgr
    s2 -->|⏱️ trigger volume + nfs units| nMgr & nNfs
    s4 -->|⬇️ pull backup@source| nBkp
    s7 -->|♻️ wipe + recover export| nNfs
    s8 -->|🚀 redeploy| nMgr
    seed -->|🌱 marker on export| nNfs
    drain -->|🩸 drain a worker| nMgr
    clean -->|🗑️ kill + remove| nMgr & nW1 & nW2 & nNfs & nBkp

    nMgr -. 📨 schedules app tasks .-> nW1 & nW2
    nMgr -. 💾 NFS-backed volumes .-> nNfs
    nW1 -. 🔗 NFS mount .-> nNfs
    nW2 -. 🔗 NFS mount .-> nNfs
    nBkp -. 🔑 pull over ssh .-> nMgr & nNfs

    classDef infra fill:#f3f7fb,stroke:#d6e2ee,color:#455a64;
    classDef deploy fill:#f3f9f4,stroke:#d9ecdc,color:#455a64;
    classDef drill fill:#fbf6f0,stroke:#eee1cf,color:#455a64;
    classDef chaos fill:#fbf4f4,stroke:#eed9d9,color:#455a64;
    classDef teardown fill:#f7f8f9,stroke:#e3e6e9,color:#455a64;
    classDef mgr fill:#f6f3fb,stroke:#e0d7ee,color:#455a64;
    classDef wrk fill:#f8f4fa,stroke:#e6dcec,color:#455a64;
    classDef nfsnode fill:#f1f8f6,stroke:#d5e7e2,color:#455a64;
    classDef bkpnode fill:#fbf3f6,stroke:#eed9e2,color:#455a64;
    style infra fill:#ffffff,stroke:#e6e6e6;
    style deploy fill:#ffffff,stroke:#e6e6e6;
    style drill fill:#ffffff,stroke:#e6e6e6;
    style chaos fill:#ffffff,stroke:#e6e6e6;
    style teardown fill:#ffffff,stroke:#e6e6e6;
    style lab fill:#ffffff,stroke:#e6e6e6;
```

The drill proves the `svc-bkp-*` chain forward through the DEPLOYED
systemd units (nfs on the export host, volume + secrets on the manager when
the app's include closure pulls those roles in, remote pull + device sync on
the backup host) and every
`recover.py` back (device -> local root -> NFS export, docker volume and
host secrets into the live system paths), with marker files that must
survive the whole loop: the matrix update pass boots the recovered stack
and `verify_recovered_marker.sh` asserts the marker on the live volume. It
reuses the round-1 stack instead of spinning a dedicated cluster, and skips
cleanly when the app declares no NFS-flagged volume. The backup host is started by `routine/01_bootstrap.sh` (drill
profile) and receives its two roles via `extend_inventory`; the pull
trust (backup keypair) and the device mount/target/source come from
`utils/tests/swarm/write/extras.py`, while
`remote-2-local.backup_providers` is derived per round by
`utils/tests/swarm/backup_repos.py` from the same placements
`extend_inventory` turns into inventory groups, and reaches the deploy
through the provisioner's host_vars merge.

## Scripts

The sequenced flow lives in `routine/`, the naming SPOT in `utils/topology/`,
shared helpers in `utils/`, and
the cluster declaration (image, containers, network, DNS play) in
`compose/swarm/` + `compose/swarm/compose.yml`.

| Stage | Script | Purpose |
|---|---|---|
| loop | `scripts/tests/deploy/distros.sh` | SPOT: run the drill once per distro in random order under the shared time budget, then table every outcome (✅ passed / ❌ failed / 🟦 skipped) into the job summary |
| loop | `routine/00_one.sh` | the whole drill for one distro; collect + teardown run from its `EXIT` trap |
| bring-up | `utils/topology/base.sh` | SPOT: node names + NFS export/state paths (sourced, not run) |
| bring-up | `utils/topology/export.sh` | write the topology SPOT into `$GITHUB_ENV` |
| bring-up | `compose/swarm/compose.yml` + `compose/swarm/Dockerfile` | declare the 5 node containers, node image + lab network (compose SPOT) |
| bring-up | `routine/01_bootstrap.sh` | host side: pre-clean, `compose build` + one `compose up`, sudo `.deb` build, then the play |
| bring-up | `compose/swarm/playbook.yml` | node side over docker connection: systemd wait, NFS-export wipe, lab DNS, repo + `.deb` install |
| deploy | `routine/02_provision_inventory.sh` | provision the per-round inventory |
| deploy | `routine/03_wait_converge.sh` | wait for every stack service to converge |
| deploy | `routine/04_verify_reachable.sh` | probe the app is reachable in-cluster |
| deploy | `routine/backup/base.sh` (+ per-host routines in `routine/backup/`) | backup+restore DR drill between rounds |
| deploy | `utils/clean/purge_stacks.sh` | remove prior-round stacks between rounds |
| chaos | `routine/05_seed_content.sh` | seed a marker on the NFS volume |
| chaos | `routine/06_drain_worker.sh` | drain the app's worker + force reschedule |
| chaos | `routine/07_assert_state.sh` | assert the marker + reachability survived |
| teardown | `utils/collect/diagnostics.sh` | write stack/service diagnostics on failure, one file per topic under the rescue dir |
| teardown | `../utils/rescue_index.sh` | list the collected rescue tree so a reader knows what the artifact holds |
| teardown | `utils/collect/playwright_reports.sh` | pull Playwright reports from the manager into `/tmp/playwright-artifacts/<distro>/<app>` |
| teardown | `utils/collect/topology_summary.sh` | render the node list + service placement into `$GITHUB_STEP_SUMMARY` |
| teardown | `utils/clean/teardown.sh` | kill the nodes + remove the lab network |
| helper | `utils/_context.sh` | per-app facts (entity, service, NFS volume, probes) |
| helper | `utils/unmount/nfs_mounts.sh` | best-effort NFS unmount before node removal |
| helper | `utils/unmount/host_state.sh` | release the host mount of the shared state so the next cycle cannot inherit it |
| recovery | `utils/clean/all.sh` | reap leftover clusters across every cluster id |
| recovery | `utils/clean/stale_nfs.sh` | recover stale in-namespace NFS mounts |

The matrix orchestrator
(`utils/tests/swarm/matrix.py`) drives the deploy stage
per variant round; `routine/00_one.sh` drives the surrounding stages for one
distro, and the workflow `.github/workflows/call-test-deploy.yml` drives the
distro loop. Run one app locally with
`make swarm-zombie app=<id>` (keeps the cluster for inspection) or the
whole matrix via `make roundtrip`.