mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 10:59:59 +00:00
The lab subnet is pinned per worktree slot while lab network names carry the cluster id, so a cluster kept from one app blocked the next run of another with 'invalid pool request: Pool overlaps'. The pre-run teardown only cleared the same cluster id, which never matched. Matching on the IPAM subnet instead clears exactly the blocker and leaves a sibling worktree's cluster alone; swarm-clean stays the host-wide tool. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
6.2 KiB
6.2 KiB
Roundtrip Loop
Use this page for validating one or more roles across every deploy mode in order (compose, then swarm) as a breadth-first cross-mode parity gate, rather than a focused single-mode debug session. For debugging the compose deploy of one role, see Compose Loop; for the swarm / Act side, see Swarm Loop; for spec-level inner-loop iteration, see Playwright Spec Loop.
When to use
- Use the roundtrip loop to confirm parity: a role (or a set of roles) must come up green in compose AND swarm. It is the end-of-change regression sweep, not the place to debug a fresh failure.
- While you are actively debugging one mode, use the focused loops instead: Compose Loop for the compose deploy, Swarm Loop plus the
swarm-*targets for the swarm deploy. Return to the roundtrip once both modes pass in isolation.
The loop
- Invoke with
make roundtrip apps="<app> [app...]". Each app is taken through the mode sequence in order, and the run stops at the first failure (fail-fast). - With no
apps=, the loop defaults to every application, most-complex first (thecomplexityCLI:infinito meta roles applications complexity --sort "desc weight" --format string). That is a large run (one compose plus one swarm deploy per app), so narrow it withapps=while iterating. - The mode sequence defaults to
compose swarmand is overridable withmodes="compose swarm". The order is always compose first, so the cheaper mode fails fast before the expensive swarm rebuild. Appendk8shere once that mode exists. - Per step the output is streamed to
${TMPDIR:-/tmp}/roundtrip-<app>-<mode>.log. Tell the operator the exacttail -fpath for the running step. - The compose step runs
compose-deploy mode=reinstall apps=<app> full_cycle=true variant=0; the swarm step runsswarm-zombie app=<app>, pinned tovariant:0so a multi-variant app validates one cluster (parity with the compose step, not the full per-variant CI matrix that runs one isolated runner per variant). The cluster is named<app>-swarm-mgr-01etc. via the mandatorySWARM_NAME(the app id is the default cluster id). Each validated swarm cluster is released afterwards unless you passkeep=true. make autoformatandmake testMUST be green before a roundtrip; the loop does not re-run them per app.- A blocked mode is NEVER a reason to silently switch to another. If a mode cannot run (the swarm NFS wedge below, a wedged daemon, a missing host capability), STOP the whole sweep, report the blocker together with the exact recovery command, and wait for it to be cleared. You MUST NOT fall back to a
modes=compose-only (or any reduced) sweep to look productive: it hides the unmet swarm parity and never satisfies the compose-AND-swarm gate. Resume the full sweep only once the blocker is gone.
Inspect before redeploy
The roundtrip is a gate, not a debugger: when a step goes red it stops and leaves the evidence in place. Do NOT just re-run the roundtrip; drop into the matching focused loop.
- Read
${TMPDIR:-/tmp}/roundtrip-<app>-<mode>.logto see which app and mode failed, and the first error. - If the compose step failed, switch to the Compose Loop: reproduce with
make compose-deploy mode=update apps=<app>, inspect viamake compose-exec/make compose-inner-run, apply the real fix in the repository, then re-run the roundtrip. - If the swarm step failed, the cluster is left up (fail-fast skips the release). Inspect it with
make swarm-exec node=<app>-swarm-mgr-01 cmd='...'ormake swarm-shell name=<app>per the Swarm Loop, confirm the fix on the live cluster, then re-run. Release it manually withmake swarm-down name=<app>when done. - Validate every candidate fix on live state BEFORE the next roundtrip; a swarm rebuild costs tens of minutes (same rule as the focused loops). In-cluster edits are validation only, the repo change is the real fix.
Exit
- Once the failing mode is green in its focused loop, you MUST resume the roundtrip over the failed app plus the apps that have not run yet (
apps="<failed-app> <still-untested apps in complexity order>"), and you MUST NOT restart the full sweep from the top. Apps that already went green earlier in the same run stay green; re-running them wastes one swarm rebuild each. - You MUST re-run the full no-
apps=sweep only as the final end-to-end parity confirmation, after the tail has gone green.
Recovery & gotchas
- On a recent Docker engine the swarm step can abort at job setup; run
make act-runner-imageonce and prefix the run withACT_PLATFORM_IMAGE=local/act-runner-fixed:latest(see Workflow Loop). - The swarm-test nfs-server serves shared storage from userspace
nfs-ganesha(theganeshaflavor ofsvc-storage-nfs-server, the default in every runtime), which runs inside the privileged DinD node and never touches the host kernelnfsd, so an interrupted export cannot wedge the host. Thekernelflavor is a per-inventory override for real hosts, wherenfsdis not shared across clusters. If a swarm step ever hangs atReload NFS exportsfrom a stuck kernel-NFS mount, you MUST recover withmake swarm-cleanon the host (needs sudo): it removes what the docker CLI can, then runsumount -f -l /var/lib/infinito+exportfs -ua+systemctl restart docker. The no-sudo sandbox only prints those commands, so the host recovery is mandatory and the sweep MUST stop until it is done; never fall back to compose-only. - Only one swarm cluster fits per worktree: the lab subnet is pinned to the worktree slot (
INFINITO_SWARM_LAB_SUBNETinscripts/system/worktree/up.sh), so a second cluster in the same worktree fails at network creation withinvalid pool request: Pool overlaps.swarm-zombietherefore reclaims whatever holds that subnet before it starts (utils/clean/lab_subnet.sh), so a cluster kept withkeep=truesurvives only until the next swarm run in the same worktree. Inspect a kept cluster before starting the next run. Parallel clusters need separate worktrees (make worktree-up), which carry distinct lab octets and are left untouched by the reclaim;make swarm-cleanis the deliberate exception that clears every worktree's clusters.