kevinveenbirkenbach/computer-playbook
1
0 Fork 0
mirror of https://github.com/kevinveenbirkenbach/computer-playbook.git synced 2026-11-06 02:54:26 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
fix/dev-python/python-upper-bound
computer-playbook/scripts/tests/deploy/swarm/README.md
T
kevinveenbirkenbachandClaude Opus 5 db66e01020 feat(ci): collapse the four deploy workflows into one chunk chain 
call-test-deploy-{compose,host,swarm,single-node}.yml differed only in
which mode they pinned and which artifact prefix they wrote; compose and
host already ran the identical drill. They become one call-test-deploy.yml
whose matrix rows carry the mode, so the swarm-specific steps are guarded
on it and everything else is shared.

The orchestrator loses its eight deploy jobs and the sequencing job for
three serial chunk blocks. GitHub Actions cannot generate a variable
number of jobs, so the blocks are written out and INFINITO_CI_MAX_CHUNKS
tells the budget how many exist; blocks whose slice is empty skip
themselves. chunk_gate stops the chain at the first red chunk,
resume_from_chunk re-enters after a fix instead of re-running the green
chunks, and modes collapses to a single mode choice.

apps.sh now emits matrix entries for one chunk rather than a token list
for one mode, and no longer sorts them: the discovery order is what the
chunker slices on. split_runner_apps.sh reads its runner quota from
INFINITO_CI_CONCURRENCY instead of carrying a second copy of the 20.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-15 06:09:04 +02:00

11 KiB
Raw Permalink Blame History

Swarm deploy test harness

Simulates a full Docker Swarm cluster on a single CI runner (or host via make swarm-zombie / make roundtrip) and deploys one application through it end to end: provision an inventory, deploy every variant round with the in-deploy Playwright e2e, run the backup+restore DR drill between rounds, then prove state survives a worker reschedule.

The cluster consists of privileged systemd containers on a dedicated lab bridge (192.168.244.0/24, MTU 1400): three swarm nodes (1 manager + 2 workers), a non-swarm NFSv4 server serving the shared volume storage, and a non-swarm backup host for the DR drill. The node image (compose/swarm/Dockerfile) bakes python3 + dnsmasq and masks the zfs boot units the parent image's userland enables; the containers and the lab network are declared in compose/swarm/compose.yml (project ${SWARM_NAME}, backup host behind the drill profile) and started by one docker compose up in routine/01_bootstrap.sh. Lab DNS is provisioned by compose/swarm/playbook.yml over ansible_connection: docker (Ansible never uses ssh here); swarm init/join itself happens inside the deploy via the svc-swarm-node role, not in any bring-up script.

Flow

flowchart TB
    subgraph infra["🥾 Cluster bring-up (per matrix app)"]
        direction TB
        topo["🗺️ utils/topology/base + export<br/>SPOT node names + NFS paths → GITHUB_ENV"]:::infra
        cluster["🐝 01_bootstrap.sh<br/>host side: pre-clean, compose build + up<br/>all 5 nodes + lab network, sudo .deb build"]:::infra
        boot["🧱 compose/swarm/playbook.yml (over docker connection)<br/>systemd wait, NFS-export wipe,<br/>lab DNS, repo + .deb install on every node"]:::infra
        topo --> cluster --> boot
    end

    subgraph deploy["🚀 Matrix deploy (per variant round)"]
        direction TB
        prov["📋 02_provision_inventory<br/>per-round inventory + extend_inventory + write_extras"]:::deploy
        dep["📦 deploy round N: swarm init/join (svc-swarm-node)<br/>+ docker stack deploy + Playwright e2e"]:::deploy
        conv["✅ 03_wait_converge + 04_verify_reachable"]:::deploy
        purge["🧽 purge_stacks (between rounds)"]:::deploy

        subgraph drill["🔐 routine/backup/base.sh (DR drill, once, after round 1)"]
            direction TB
            s1["1️⃣ seed markers into the live NFS volume<br/>and the manager secrets"]:::drill
            s2["2️⃣ trigger backup units<br/>volume + secrets @ manager, nfs-2-local @ NFS server"]:::drill
            s3["3️⃣ locate the generation holding the marker<br/>3️⃣b assert the volume repo is fresh + non-empty<br/>(its NFS-backed volumes are excluded by design)"]:::drill
            s4["4️⃣ pull to the backup host<br/>remote-2-local over the user-backup ssh wrapper"]:::drill
            s5["5️⃣ space gate: pulled tree + watchdog floor,<br/>du breakdown on failure; then LUKS loop 'USB'<br/>via the local-2-device unit"]:::drill
            s6["6️⃣ full disaster: stack rm"]:::drill
            s7["7️⃣ wipe the pulled tree, recover device → local root<br/>(luksOpen; the device holds the only copy left)"]:::drill
            s8["8️⃣ wipe the live export (marker proven gone),<br/>recover root → export, restore NFS coherence"]:::drill
            s9["9️⃣ delete the marker server-side, recover the volume<br/>through its NFS-mounted mountpoint, re-verify on the<br/>server; recover the host secrets"]:::drill
            s1 --> s2 --> s3 --> s4 --> s5 --> s6 --> s7 --> s8 --> s9
        end

        prov --> dep --> conv --> drill --> purge
    end

    subgraph chaos["💪 Reschedule chaos (after the matrix)"]
        direction TB
        seed["🌱 05_seed_content<br/>marker on the NFS volume"]:::chaos
        drain["🩸 06_drain_worker<br/>drain the app's worker, force reschedule"]:::chaos
        assert["🔎 07_assert_state<br/>marker survived + app reachable on new node"]:::chaos
        seed --> drain --> assert
    end

    subgraph teardown["🧹 Always"]
        direction TB
        diag["📊 collect: diagnostics / playwright reports +<br/>workflow-level recursive container.py rescue snapshot"]:::teardown
        clean["🗑️ utils/clean/teardown (kill nodes + lab network)"]:::teardown
        diag --> clean
    end

    subgraph lab["🌐 Lab bridge 192.168.244.0/24 (MTU 1400)"]
        direction LR
        nMgr["👑 swarm-mgr-01 (manager)<br/>stacks, volume-2-local, user-backup, sshd"]:::mgr
        nW1["⚙️ swarm-wrk-01 (worker)"]:::wrk
        nW2["⚙️ swarm-wrk-02 (worker)"]:::wrk
        nNfs["📦 nfs-server (non-swarm)<br/>NFS export, nfs-2-local, sshd"]:::nfsnode
        nBkp["💽 backup-host (non-swarm)<br/>deployed remote-2-local + local-2-device units,<br/>LUKS device"]:::bkpnode
    end

    infra --> deploy --> chaos --> teardown

    cluster -->|▶️ compose up all 5| nMgr & nW1 & nW2 & nNfs & nBkp
    boot -->|🌐 hosts + dnsmasq| nMgr & nW1 & nW2 & nNfs
    boot -->|📥 repo + .deb install| nMgr & nW1 & nW2 & nNfs & nBkp
    dep -->|🚀 swarm init/join + docker stack deploy| nMgr
    s2 -->|⏱️ trigger volume + nfs units| nMgr & nNfs
    s4 -->|⬇️ pull backup@source| nBkp
    s7 -->|♻️ wipe + recover export| nNfs
    s8 -->|🚀 redeploy| nMgr
    seed -->|🌱 marker on export| nNfs
    drain -->|🩸 drain a worker| nMgr
    clean -->|🗑️ kill + remove| nMgr & nW1 & nW2 & nNfs & nBkp

    nMgr -. 📨 schedules app tasks .-> nW1 & nW2
    nMgr -. 💾 NFS-backed volumes .-> nNfs
    nW1 -. 🔗 NFS mount .-> nNfs
    nW2 -. 🔗 NFS mount .-> nNfs
    nBkp -. 🔑 pull over ssh .-> nMgr & nNfs

    classDef infra fill:#f3f7fb,stroke:#d6e2ee,color:#455a64;
    classDef deploy fill:#f3f9f4,stroke:#d9ecdc,color:#455a64;
    classDef drill fill:#fbf6f0,stroke:#eee1cf,color:#455a64;
    classDef chaos fill:#fbf4f4,stroke:#eed9d9,color:#455a64;
    classDef teardown fill:#f7f8f9,stroke:#e3e6e9,color:#455a64;
    classDef mgr fill:#f6f3fb,stroke:#e0d7ee,color:#455a64;
    classDef wrk fill:#f8f4fa,stroke:#e6dcec,color:#455a64;
    classDef nfsnode fill:#f1f8f6,stroke:#d5e7e2,color:#455a64;
    classDef bkpnode fill:#fbf3f6,stroke:#eed9e2,color:#455a64;
    style infra fill:#ffffff,stroke:#e6e6e6;
    style deploy fill:#ffffff,stroke:#e6e6e6;
    style drill fill:#ffffff,stroke:#e6e6e6;
    style chaos fill:#ffffff,stroke:#e6e6e6;
    style teardown fill:#ffffff,stroke:#e6e6e6;
    style lab fill:#ffffff,stroke:#e6e6e6;

The drill proves the svc-bkp-* chain forward through the DEPLOYED systemd units (nfs on the export host, volume + secrets on the manager when the app's include closure pulls those roles in, remote pull + device sync on the backup host) and every recover.py back (device -> local root -> NFS export, docker volume and host secrets into the live system paths), with marker files that must survive the whole loop: the matrix update pass boots the recovered stack and verify_recovered_marker.sh asserts the marker on the live volume. It reuses the round-1 stack instead of spinning a dedicated cluster, and skips cleanly when the app declares no NFS-flagged volume. The backup host is started by routine/01_bootstrap.sh (drill profile) and receives its two roles via extend_inventory; the pull trust (backup keypair) and the device mount/target/source come from utils/tests/swarm/write/extras.py, while remote-2-local.backup_providers is derived per round by utils/tests/swarm/backup_repos.py from the same placements extend_inventory turns into inventory groups, and reaches the deploy through the provisioner's host_vars merge.

Scripts

The sequenced flow lives in routine/, the naming SPOT in utils/topology/, shared helpers in utils/, and the cluster declaration (image, containers, network, DNS play) in compose/swarm/ + compose/swarm/compose.yml.

Stage Script Purpose
loop scripts/tests/deploy/distros.sh SPOT: run the drill once per distro in random order under the shared time budget, then table every outcome (✅ passed / ❌ failed / 🟦 skipped) into the job summary
loop routine/00_one.sh the whole drill for one distro; collect + teardown run from its EXIT trap
bring-up utils/topology/base.sh SPOT: node names + NFS export/state paths (sourced, not run)
bring-up utils/topology/export.sh write the topology SPOT into $GITHUB_ENV
bring-up compose/swarm/compose.yml + compose/swarm/Dockerfile declare the 5 node containers, node image + lab network (compose SPOT)
bring-up routine/01_bootstrap.sh host side: pre-clean, compose build + one compose up, sudo .deb build, then the play
bring-up compose/swarm/playbook.yml node side over docker connection: systemd wait, NFS-export wipe, lab DNS, repo + .deb install
deploy routine/02_provision_inventory.sh provision the per-round inventory
deploy routine/03_wait_converge.sh wait for every stack service to converge
deploy routine/04_verify_reachable.sh probe the app is reachable in-cluster
deploy routine/backup/base.sh (+ per-host routines in routine/backup/) backup+restore DR drill between rounds
deploy utils/clean/purge_stacks.sh remove prior-round stacks between rounds
chaos routine/05_seed_content.sh seed a marker on the NFS volume
chaos routine/06_drain_worker.sh drain the app's worker + force reschedule
chaos routine/07_assert_state.sh assert the marker + reachability survived
teardown utils/collect/diagnostics.sh write stack/service diagnostics on failure, one file per topic under the rescue dir
teardown ../utils/rescue_index.sh list the collected rescue tree so a reader knows what the artifact holds
teardown utils/collect/playwright_reports.sh pull Playwright reports from the manager into /tmp/playwright-artifacts/<distro>/<app>
teardown utils/collect/topology_summary.sh render the node list + service placement into $GITHUB_STEP_SUMMARY
teardown utils/clean/teardown.sh kill the nodes + remove the lab network
helper utils/_context.sh per-app facts (entity, service, NFS volume, probes)
helper utils/unmount/nfs_mounts.sh best-effort NFS unmount before node removal
helper utils/unmount/host_state.sh release the host mount of the shared state so the next cycle cannot inherit it
recovery utils/clean/all.sh reap leftover clusters across every cluster id
recovery utils/clean/stale_nfs.sh recover stale in-namespace NFS mounts

The matrix orchestrator (utils/tests/swarm/matrix.py) drives the deploy stage per variant round; routine/00_one.sh drives the surrounding stages for one distro, and the workflow .github/workflows/call-test-deploy.yml drives the distro loop. Run one app locally with make swarm-zombie app=<id> (keeps the cluster for inspection) or the whole matrix via make roundtrip.

Reference in new issue View Git Blame Copy Permalink