mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 10:59:59 +00:00
kevinveenbirkenbachandClaude Opus 5
db66e01020
call-test-deploy-{compose,host,swarm,single-node}.yml differed only in
which mode they pinned and which artifact prefix they wrote; compose and
host already ran the identical drill. They become one call-test-deploy.yml
whose matrix rows carry the mode, so the swarm-specific steps are guarded
on it and everything else is shared.
The orchestrator loses its eight deploy jobs and the sequencing job for
three serial chunk blocks. GitHub Actions cannot generate a variable
number of jobs, so the blocks are written out and INFINITO_CI_MAX_CHUNKS
tells the budget how many exist; blocks whose slice is empty skip
themselves. chunk_gate stops the chain at the first red chunk,
resume_from_chunk re-enters after a fix instead of re-running the green
chunks, and modes collapses to a single mode choice.
apps.sh now emits matrix entries for one chunk rather than a token list
for one mode, and no longer sorts them: the discovery order is what the
chunker slices on. split_runner_apps.sh reads its runner quota from
INFINITO_CI_CONCURRENCY instead of carrying a second copy of the 20.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
180 lines
11 KiB
Markdown
180 lines
11 KiB
Markdown
# Swarm deploy test harness
|
|
|
|
Simulates a full Docker Swarm cluster on a single CI runner (or host via
|
|
`make swarm-zombie` / `make roundtrip`) and deploys one application
|
|
through it end to end: provision an inventory, deploy every variant
|
|
round with the in-deploy Playwright e2e, run the backup+restore DR drill
|
|
between rounds, then prove state survives a worker reschedule.
|
|
|
|
The cluster consists of privileged systemd containers on a dedicated lab
|
|
bridge (`192.168.244.0/24`, MTU 1400): three swarm nodes (1 manager +
|
|
2 workers), a non-swarm NFSv4 server serving the shared volume storage,
|
|
and a non-swarm backup host for the DR drill. The node image
|
|
([`compose/swarm/Dockerfile`](../../../../compose/swarm/Dockerfile))
|
|
bakes python3 + dnsmasq and masks the zfs boot units the parent image's
|
|
userland enables; the containers and the lab network are declared
|
|
in [`compose/swarm/compose.yml`](../../../../compose/swarm/compose.yml) (project
|
|
`${SWARM_NAME}`, backup host behind the `drill` profile) and started by
|
|
one `docker compose up` in `routine/01_bootstrap.sh`. Lab DNS is provisioned
|
|
by `compose/swarm/playbook.yml` over `ansible_connection: docker` (Ansible never
|
|
uses ssh here); swarm init/join itself happens inside the deploy via the
|
|
`svc-swarm-node` role, not in any bring-up script.
|
|
|
|
## Flow
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
subgraph infra["🥾 Cluster bring-up (per matrix app)"]
|
|
direction TB
|
|
topo["🗺️ utils/topology/base + export<br/>SPOT node names + NFS paths → GITHUB_ENV"]:::infra
|
|
cluster["🐝 01_bootstrap.sh<br/>host side: pre-clean, compose build + up<br/>all 5 nodes + lab network, sudo .deb build"]:::infra
|
|
boot["🧱 compose/swarm/playbook.yml (over docker connection)<br/>systemd wait, NFS-export wipe,<br/>lab DNS, repo + .deb install on every node"]:::infra
|
|
topo --> cluster --> boot
|
|
end
|
|
|
|
subgraph deploy["🚀 Matrix deploy (per variant round)"]
|
|
direction TB
|
|
prov["📋 02_provision_inventory<br/>per-round inventory + extend_inventory + write_extras"]:::deploy
|
|
dep["📦 deploy round N: swarm init/join (svc-swarm-node)<br/>+ docker stack deploy + Playwright e2e"]:::deploy
|
|
conv["✅ 03_wait_converge + 04_verify_reachable"]:::deploy
|
|
purge["🧽 purge_stacks (between rounds)"]:::deploy
|
|
|
|
subgraph drill["🔐 routine/backup/base.sh (DR drill, once, after round 1)"]
|
|
direction TB
|
|
s1["1️⃣ seed markers into the live NFS volume<br/>and the manager secrets"]:::drill
|
|
s2["2️⃣ trigger backup units<br/>volume + secrets @ manager, nfs-2-local @ NFS server"]:::drill
|
|
s3["3️⃣ locate the generation holding the marker<br/>3️⃣b assert the volume repo is fresh + non-empty<br/>(its NFS-backed volumes are excluded by design)"]:::drill
|
|
s4["4️⃣ pull to the backup host<br/>remote-2-local over the user-backup ssh wrapper"]:::drill
|
|
s5["5️⃣ space gate: pulled tree + watchdog floor,<br/>du breakdown on failure; then LUKS loop 'USB'<br/>via the local-2-device unit"]:::drill
|
|
s6["6️⃣ full disaster: stack rm"]:::drill
|
|
s7["7️⃣ wipe the pulled tree, recover device → local root<br/>(luksOpen; the device holds the only copy left)"]:::drill
|
|
s8["8️⃣ wipe the live export (marker proven gone),<br/>recover root → export, restore NFS coherence"]:::drill
|
|
s9["9️⃣ delete the marker server-side, recover the volume<br/>through its NFS-mounted mountpoint, re-verify on the<br/>server; recover the host secrets"]:::drill
|
|
s1 --> s2 --> s3 --> s4 --> s5 --> s6 --> s7 --> s8 --> s9
|
|
end
|
|
|
|
prov --> dep --> conv --> drill --> purge
|
|
end
|
|
|
|
subgraph chaos["💪 Reschedule chaos (after the matrix)"]
|
|
direction TB
|
|
seed["🌱 05_seed_content<br/>marker on the NFS volume"]:::chaos
|
|
drain["🩸 06_drain_worker<br/>drain the app's worker, force reschedule"]:::chaos
|
|
assert["🔎 07_assert_state<br/>marker survived + app reachable on new node"]:::chaos
|
|
seed --> drain --> assert
|
|
end
|
|
|
|
subgraph teardown["🧹 Always"]
|
|
direction TB
|
|
diag["📊 collect: diagnostics / playwright reports +<br/>workflow-level recursive container.py rescue snapshot"]:::teardown
|
|
clean["🗑️ utils/clean/teardown (kill nodes + lab network)"]:::teardown
|
|
diag --> clean
|
|
end
|
|
|
|
subgraph lab["🌐 Lab bridge 192.168.244.0/24 (MTU 1400)"]
|
|
direction LR
|
|
nMgr["👑 swarm-mgr-01 (manager)<br/>stacks, volume-2-local, user-backup, sshd"]:::mgr
|
|
nW1["⚙️ swarm-wrk-01 (worker)"]:::wrk
|
|
nW2["⚙️ swarm-wrk-02 (worker)"]:::wrk
|
|
nNfs["📦 nfs-server (non-swarm)<br/>NFS export, nfs-2-local, sshd"]:::nfsnode
|
|
nBkp["💽 backup-host (non-swarm)<br/>deployed remote-2-local + local-2-device units,<br/>LUKS device"]:::bkpnode
|
|
end
|
|
|
|
infra --> deploy --> chaos --> teardown
|
|
|
|
cluster -->|▶️ compose up all 5| nMgr & nW1 & nW2 & nNfs & nBkp
|
|
boot -->|🌐 hosts + dnsmasq| nMgr & nW1 & nW2 & nNfs
|
|
boot -->|📥 repo + .deb install| nMgr & nW1 & nW2 & nNfs & nBkp
|
|
dep -->|🚀 swarm init/join + docker stack deploy| nMgr
|
|
s2 -->|⏱️ trigger volume + nfs units| nMgr & nNfs
|
|
s4 -->|⬇️ pull backup@source| nBkp
|
|
s7 -->|♻️ wipe + recover export| nNfs
|
|
s8 -->|🚀 redeploy| nMgr
|
|
seed -->|🌱 marker on export| nNfs
|
|
drain -->|🩸 drain a worker| nMgr
|
|
clean -->|🗑️ kill + remove| nMgr & nW1 & nW2 & nNfs & nBkp
|
|
|
|
nMgr -. 📨 schedules app tasks .-> nW1 & nW2
|
|
nMgr -. 💾 NFS-backed volumes .-> nNfs
|
|
nW1 -. 🔗 NFS mount .-> nNfs
|
|
nW2 -. 🔗 NFS mount .-> nNfs
|
|
nBkp -. 🔑 pull over ssh .-> nMgr & nNfs
|
|
|
|
classDef infra fill:#f3f7fb,stroke:#d6e2ee,color:#455a64;
|
|
classDef deploy fill:#f3f9f4,stroke:#d9ecdc,color:#455a64;
|
|
classDef drill fill:#fbf6f0,stroke:#eee1cf,color:#455a64;
|
|
classDef chaos fill:#fbf4f4,stroke:#eed9d9,color:#455a64;
|
|
classDef teardown fill:#f7f8f9,stroke:#e3e6e9,color:#455a64;
|
|
classDef mgr fill:#f6f3fb,stroke:#e0d7ee,color:#455a64;
|
|
classDef wrk fill:#f8f4fa,stroke:#e6dcec,color:#455a64;
|
|
classDef nfsnode fill:#f1f8f6,stroke:#d5e7e2,color:#455a64;
|
|
classDef bkpnode fill:#fbf3f6,stroke:#eed9e2,color:#455a64;
|
|
style infra fill:#ffffff,stroke:#e6e6e6;
|
|
style deploy fill:#ffffff,stroke:#e6e6e6;
|
|
style drill fill:#ffffff,stroke:#e6e6e6;
|
|
style chaos fill:#ffffff,stroke:#e6e6e6;
|
|
style teardown fill:#ffffff,stroke:#e6e6e6;
|
|
style lab fill:#ffffff,stroke:#e6e6e6;
|
|
```
|
|
|
|
The drill proves the `svc-bkp-*` chain forward through the DEPLOYED
|
|
systemd units (nfs on the export host, volume + secrets on the manager when
|
|
the app's include closure pulls those roles in, remote pull + device sync on
|
|
the backup host) and every
|
|
`recover.py` back (device -> local root -> NFS export, docker volume and
|
|
host secrets into the live system paths), with marker files that must
|
|
survive the whole loop: the matrix update pass boots the recovered stack
|
|
and `verify_recovered_marker.sh` asserts the marker on the live volume. It
|
|
reuses the round-1 stack instead of spinning a dedicated cluster, and skips
|
|
cleanly when the app declares no NFS-flagged volume. The backup host is started by `routine/01_bootstrap.sh` (drill
|
|
profile) and receives its two roles via `extend_inventory`; the pull
|
|
trust (backup keypair) and the device mount/target/source come from
|
|
`utils/tests/swarm/write/extras.py`, while
|
|
`remote-2-local.backup_providers` is derived per round by
|
|
`utils/tests/swarm/backup_repos.py` from the same placements
|
|
`extend_inventory` turns into inventory groups, and reaches the deploy
|
|
through the provisioner's host_vars merge.
|
|
|
|
## Scripts
|
|
|
|
The sequenced flow lives in `routine/`, the naming SPOT in `utils/topology/`,
|
|
shared helpers in `utils/`, and
|
|
the cluster declaration (image, containers, network, DNS play) in
|
|
`compose/swarm/` + `compose/swarm/compose.yml`.
|
|
|
|
| Stage | Script | Purpose |
|
|
|---|---|---|
|
|
| loop | `scripts/tests/deploy/distros.sh` | SPOT: run the drill once per distro in random order under the shared time budget, then table every outcome (✅ passed / ❌ failed / 🟦 skipped) into the job summary |
|
|
| loop | `routine/00_one.sh` | the whole drill for one distro; collect + teardown run from its `EXIT` trap |
|
|
| bring-up | `utils/topology/base.sh` | SPOT: node names + NFS export/state paths (sourced, not run) |
|
|
| bring-up | `utils/topology/export.sh` | write the topology SPOT into `$GITHUB_ENV` |
|
|
| bring-up | `compose/swarm/compose.yml` + `compose/swarm/Dockerfile` | declare the 5 node containers, node image + lab network (compose SPOT) |
|
|
| bring-up | `routine/01_bootstrap.sh` | host side: pre-clean, `compose build` + one `compose up`, sudo `.deb` build, then the play |
|
|
| bring-up | `compose/swarm/playbook.yml` | node side over docker connection: systemd wait, NFS-export wipe, lab DNS, repo + `.deb` install |
|
|
| deploy | `routine/02_provision_inventory.sh` | provision the per-round inventory |
|
|
| deploy | `routine/03_wait_converge.sh` | wait for every stack service to converge |
|
|
| deploy | `routine/04_verify_reachable.sh` | probe the app is reachable in-cluster |
|
|
| deploy | `routine/backup/base.sh` (+ per-host routines in `routine/backup/`) | backup+restore DR drill between rounds |
|
|
| deploy | `utils/clean/purge_stacks.sh` | remove prior-round stacks between rounds |
|
|
| chaos | `routine/05_seed_content.sh` | seed a marker on the NFS volume |
|
|
| chaos | `routine/06_drain_worker.sh` | drain the app's worker + force reschedule |
|
|
| chaos | `routine/07_assert_state.sh` | assert the marker + reachability survived |
|
|
| teardown | `utils/collect/diagnostics.sh` | write stack/service diagnostics on failure, one file per topic under the rescue dir |
|
|
| teardown | `../utils/rescue_index.sh` | list the collected rescue tree so a reader knows what the artifact holds |
|
|
| teardown | `utils/collect/playwright_reports.sh` | pull Playwright reports from the manager into `/tmp/playwright-artifacts/<distro>/<app>` |
|
|
| teardown | `utils/collect/topology_summary.sh` | render the node list + service placement into `$GITHUB_STEP_SUMMARY` |
|
|
| teardown | `utils/clean/teardown.sh` | kill the nodes + remove the lab network |
|
|
| helper | `utils/_context.sh` | per-app facts (entity, service, NFS volume, probes) |
|
|
| helper | `utils/unmount/nfs_mounts.sh` | best-effort NFS unmount before node removal |
|
|
| helper | `utils/unmount/host_state.sh` | release the host mount of the shared state so the next cycle cannot inherit it |
|
|
| recovery | `utils/clean/all.sh` | reap leftover clusters across every cluster id |
|
|
| recovery | `utils/clean/stale_nfs.sh` | recover stale in-namespace NFS mounts |
|
|
|
|
The matrix orchestrator
|
|
(`utils/tests/swarm/matrix.py`) drives the deploy stage
|
|
per variant round; `routine/00_one.sh` drives the surrounding stages for one
|
|
distro, and the workflow `.github/workflows/call-test-deploy.yml` drives the
|
|
distro loop. Run one app locally with
|
|
`make swarm-zombie app=<id>` (keeps the cluster for inspection) or the
|
|
whole matrix via `make roundtrip`.
|