mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-09 04:06:42 +00:00
kevinveenbirkenbachandClaude Opus 5
cfe20f0726
The main merge left two step watchers: run.run_watched from main and the newer watchdog.run_watched from here. Their disk and memory floors are byte-identical (6144M, 6% of RAM, abort 75), so run.py now keeps only run_step and takes the watcher from watchdog. mesh.py reaches it through run_step and matrix.py already imported it directly. That alone would have been a regression. The watchdog aborts on io60 alone, which its own docstring says cannot separate a wedged mount from a busy disk, and the six mesh steps had no I/O kill before: a healthy sleep on a host with 126G free and no task in D state died at rc=75 after 10s where the old watcher let it finish. The first sample also lands 10s in, so 50 of the 60 seconds it averages belong to whatever ran before -- an image pull could abort its successor before that step read a byte. The ceiling now takes three things together: a task in D state on the same sample, WEDGE_SAMPLES breaches in a row, and a step old enough to own a whole STALL_WINDOW_S of the average. Both refuted scenarios are regression tests. A real wedge is caught about a minute later than before; the disk floor that covers ENOSPC is an instantaneous read and is untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Swarm test inventory extension
extend_inventory.py expands a provisioned swarm-test inventory so the static
validator and the deploy see every group the run will use. It reads APP_ID,
INV_PATH and SWARM_NAME from the environment, merges the computed groups
into INV_PATH, and writes a sibling backup.yml for the second (backup-host)
play.
Flow
flowchart TD
ENV["env: SWARM_NAME, APP_ID, INV_PATH"] --> MAIN["main()"]
MAIN --> TOPO["_host_topology(app_id)"]
MAIN --> DEP["_placement_dep_groups(app_id)"]
MAIN --> BKP{"backup_repos.repo_placements:\nvolume / secrets role in derive_includes(app_id)?"}
MAIN --> NBKP{"backup_repos.repo_placements:\nsvc-bkp-nfs-2-local in derive_includes(svc-storage-nfs-server)?"}
TOPO --> T1["app_id → manager (+ workers unless manager-placed)"]
TOPO --> T2["svc-swarm-node → manager + workers"]
TOPO --> T3["svc-swarm-manager → manager"]
TOPO --> T4["svc-storage-nfs-client → manager + workers"]
TOPO --> T5["svc-storage-nfs-server → nfs-server"]
DEP --> D1["derive_includes(app_id)\nrole with placement=manager → manager"]
BKP -- yes --> B1["svc-bkp-volume-2-local,\nsvc-bkp-secrets-2-local → manager"]
BKP -- no --> SKIP["skip app-induced backup legs"]
NBKP -- yes --> NFS["svc-bkp-nfs-2-local → nfs-server"]
NBKP -- no --> NSKIP["skip nfs-backup leg"]
T1 & T2 & T3 & T4 & T5 & D1 & B1 & NFS --> MERGE["merge groups into all.children"]
MERGE --> OUT1["write INV_PATH (devices.yml)"]
MAIN --> OUT2["write backup.yml:\nsvc-bkp-remote-2-local,\nsvc-bkp-local-2-device → backup host"]
Groups
| Group | Host | Condition |
|---|---|---|
APP_ID |
manager, workers | workers dropped when the role is placement: manager |
svc-swarm-node |
manager, workers | always |
svc-swarm-manager |
manager | always |
svc-storage-nfs-client |
manager, workers | always |
svc-storage-nfs-server |
nfs-server | always |
dep roles from derive_includes |
manager | role declares placement: manager |
svc-bkp-volume-2-local |
manager | derive_includes(APP_ID) pulls it in via the container_backup consumer |
svc-bkp-secrets-2-local |
manager | derive_includes(APP_ID) pulls it in |
svc-bkp-nfs-2-local |
nfs-server | derive_includes(svc-storage-nfs-server) pulls it in via the nfs_backup consumer |
svc-bkp-remote-2-local, svc-bkp-local-2-device |
backup host | always, written to backup.yml |
Node names come from default.env (INFINITO_SWARM_*_NAME) with the
SWARM_NAME prefix. INFINITO_APP_VARIANTS feeds derive_includes to select
the active variant; in-process callers pass the map as
derive_includes(app_id, variants=...).
The three svc-bkp-* repository rows come from backup_repos.repo_placements.
utils/tests/swarm/matrix.py calls backup_repos.backup_provider_ips, which
resolves the same two closures and returns the nodes as IPs, to build
svc-bkp-remote-2-local.services.remote-2-local.backup_providers.