On the meshed arm the layout joins the backup overrides through
merge_with_defaults and lands in every host's host_vars; the direct arm
carries no mesh value at all.
Three runs of the transport benchmark, per module call:
become off become on
docker 247ms 246ms
ssh over mesh 215ms 1259ms
The connection is not the cost - ssh through the tunnel beats docker exec by
13% - and the tunnel is 4%. What the meshed arm pays is the sudo prompt: free
over the container connection, which runs as root, and 1046ms per escalated
call over ssh as the administrator. That is the 414ms per task the three-arm
comparison measured across a whole deploy.
A lab-only sudoers drop-in removes it. It is applied from the mesh prologue
while the inventory still reaches the nodes as root, so it is in place before
anything logs in as the administrator, and it touches no role: production
keeps 'administrator ALL=(ALL) ALL' and its password prompt.
The cost is that CI no longer exercises password sudo over the mesh. It never
exercised it on the direct arm either, which connects as root, and a wrong
password fails loudly rather than silently.
Run 37931817959 timed the four cells and two of them are real: a module
call costs 245ms over docker and 212ms over ssh through the tunnel, with or
without become on the docker side. The connection plugin is not the cost.
The escalated ssh cell is not real. The benchmark ran without the extras, so
ansible_become_password was absent and all 200 calls failed on 'Incorrect sudo
password' - 3.2s each, which prices a failing sudo rather than a working one.
It now passes the same two -e files the deploy passes.
Its exit code also reached the round, so a diagnostic turned three green
deploys red. transport_bench now discards its own rc: whether a deploy passes
is not a benchmark's business.
The three-arm comparison on web-app-gitea#0 put the meshed arm at 414ms
per task over the direct arm, and the tunnel at 4% of that. What it could not
split is the rest: the arms differ both in connection plugin (docker against
ssh) and in escalation (root with no prompt against administrator with a
password sudo on every task).
A fixed set of module calls, timed either side of the transport switch with
become off and on, gives four numbers that separate the two. It costs about a
minute where a fourth deploy arm would have cost three hours, and it needs no
root login and no sudoers change.
INFINITO_SWARM_TRANSPORT_BENCH guards it, so a normal round times nothing.
The meshed round was its own shape: a mesh-only first pass, one
application deploy, and a second only when a drill needed one. Comparing it
with the direct arm meant adjusting for a missing pass and a stripped browser
suite, and every timing answer carried a caveat.
Both arms now run deploy, converge, drill, redeploy, converge, verify. What
the mesh needs moves into a prologue that installs no role: tunnel up on every
node, keys rotated, controller joined, transport switched. Both application
passes then run over the tunnel and both print their PASS banner, so a meshed
row and a direct row differ only in how they were reached.
The credential reset stays in the prologue rather than moving to the direct
arm's slot between the passes. Its mirror copies the manager's host_vars over
every other host, keeping only applications.svc-net-wireguard, so after the
switch every worker would inherit the manager's mesh address and three
connections would land on one node.
The drill probe now gates both arms. A backup host deployed for an application
with no NFS-flagged volume is several minutes spent on a drill that skips
itself, and that was true on the direct arm before this branch existed.
Narrowing the meshed drill to its backup half bought budget by giving
up the recovery legs over the mesh. The budget overrun is worth measuring
before it is designed around, so the pre-drill pass and the full teardown and
recover come back. Reverts a27b093c4.
The pre-drill pass cost a whole deploy so the drill would have a stack
to tear down and a later pass to bring it back. n8n variant 0 spent 11970s in
that pass alone and hit the 20400s budget before the real deploy started.
The meshed round now deploys once, as the rest of the branch does, and runs
the backup half of the drill after it: backup host plus DRILL_STAGE=backup,
around 550s against roughly 12000s. Teardown, the recover CLI and the bring-up
onto the recovered export are transport-agnostic and the direct arm proves
them on every application the axis rotates there. What the meshed half adds is
the backup host joining the data mesh and running its units over the tunnel.
Step 3b, which caught the Nextcloud volume-repo failure, is inside the half
that is kept.
The drill tears the stack down and restores the data behind it, leaving
the bring-up to whatever deploys next. A meshed round deploys once, so after
a drill nothing brought the stack back onto the recovered export -- and the
round still reported green, because the marker is asserted against the
export rather than against a running stack.
That is observed, not predicted: a meshed BookWyrm round passed with
backup + restore DR drill took 81s (rc=0)
==> DR drill PASSED: ... verified on the NFS export after the update pass
and nothing after it. The cluster finished the round torn down.
A meshed round now deploys before the drill as well, and only when the drill
says it is coming. That first pass carries SWARM_DR_PREPASS, which skips the
end-to-end suites because its stack is about to be thrown away; the pass
after the drill is the full one, so the suites run once and against the
recovered stack. A round with nothing to restore still deploys exactly once.
The arguments every deploy of a round shares move into one mapping, and the
environment the drill reads into one helper, which is what keeps the file
inside the size gate after the addition.
One conflict, in the MCP exclusion list of requirement 035. Both sides
edited the same line: main added svc-net-tor-smtp, this branch replaced the
three svc-net-wireguard-{core,plain,firewalled} entries with the single
svc-net-wireguard it deploys. Resolved to the union of both — 24 entries —
which is what the merged tree actually holds.
Everything else merged clean and was checked for overlap rather than
assumed: no reference anywhere still names the three replaced roles;
matrix.py kept main's PRIMARY_APPS alongside the reordered round, and the
bootstrap pass does not need it, since only one template reads it and
group_vars defaults it empty; and main's ansible.cfg addition sits in
[defaults], so the ssh_args the mesh tool derives its own connection
settings from are untouched.
The round replaces its first deploy with the mesh bootstrap and keeps the
deploy after the seam as the only one that installs the applications: the
same roles at 2.98s a task instead of 0.74s, some five thousand tasks of it,
which is most of why a meshed variant 0 ran past the 20400s budget.
The backup node and the DR drill move behind that pass, because there is
nothing to back up before it; they are held together in one step so the two
call sites do not drift. Credentials still reset before the deploy that has
to carry them.
A direct-transport round is untouched and still deploys twice.
Only the first round deploys the backup host, so a later round meets a
container that has never been provisioned: the mesh role is the first thing
to reach it, and installing the tooling fails because nothing has populated
its package index. Every meshed variant above zero died there.
The converge now takes the sibling inventory only when the round brought
that node up, which is the only round whose data plane has a second spoke
to carry.
Reverts the apt refresh added for the same symptom. It reported the index
fresh on exactly the host that had none, so it never addressed this and the
rounds that do deploy the backup node were already converging without it.
The second write rotates, so the converge, the controller join and the second
pass all run against keys that changed mid-deploy. Also trims the reset
narration.
The mirror wiped each host's mesh block, so the write that follows minted a
fresh keypair for everyone and the converge re-keyed tunnels the deploy was
already running services over. The NFS state mount rides the data mesh and is
soft in CI, so a re-handshake dropped it; the second pass then found local
data under /var/lib/infinito and refused to mount over it.
Production never mirrors host_vars and mounts hard, so this only removes a
scenario the harness invented for itself.
The round already had two passes. The first brings the mesh up over the
container connection; the inventory is then switched to the mesh addresses and
the second pass runs over the tunnel, so no extra round and no extra CI time.
A full deploy of every role over the mesh is a stronger proof than a narrow
bootstrap round would have been.
Brings in 225 commits, including the merged web-app-openbao work and the
LMStudio loopback fix.
Seven conflicts:
- The legacy svc-net-wireguard-core/plain/firewalled roles this branch deleted
in e706170a0 were modified upstream. The deletion stands, which is the whole
premise of the branch.
- The MCP audit takes upstream's list plus svc-net-firewall, with the three
legacy wireguard entries collapsed into the single svc-net-wireguard: 23.
- web-app-openbao/README.md takes main's reviewed version wholesale.
- matrix.py takes both sides: upstream's app_id argument to _reset_credentials
and this branch's _write_mesh call after it.
The merge pushed matrix.py to 511 lines against the 500 cap, so the step
runner moved to utils/tests/swarm/run.py, which is self-contained and leaves
matrix.py at 427. Its test moved with it.
The gate mirrors the manager's host_vars onto every node, so the per-host mesh
each member needs is gone by the time the update pass deploys. Running the
writer again restores it, which is also what 'the tool runs before every
deploy' has to mean once anything rewrites the inventory mid-round.
Placed after extend_inventory, which is what creates the group memberships the
meshes resolve from and the sibling backup.yml the data mesh spans, and before
the deploy, which is the first step that needs a tunnel.
Both inventory files go in one invocation. A mesh written from only the cluster
inventory would silently omit the backup host, and nothing would notice until
the backup play could not reach the hub.
e1d513a3c0 replaced include=round_include with variant_scope in the payload resolve and left the loop variable without a reader, which ruff B007 rejects: 'Loop control variable round_include not used within loop body'. grep confirms the unpacking at matrix.py:388 is now its only occurrence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The swarm matrix resolved variant payloads over the planner's round_include while provisioning the closure derive_includes returns. The two agree until a variant switches its couplings off: the planner then resolves no prerequisites, round_include collapses to the primary, and every deployed dependency silently keeps its base config. For svc-ai-litellm round 1 that meant web-app-keycloak ran with the base realm.totp_enabled true, so CONFIGURE_TOTP became a default required action and every SSO persona sat in a 300s predicate timeout until the job hit its wall clock at 20086s.
payload.py states the intent this restores: the role pulling a dependency in is what decides how lean the dependency runs. variant_scope keeps the decision next to the closure it filters and excludes the three lab infrastructure roles, whose payload would replace group-membership expressions such as {{ 'svc-swarm-node' in group_names }} with an unconditional true in every round.
Measured: round 1 goes from 1 to 29 payload apps and keycloak from absent to totp_enabled false, while the deployed closure stays 32. Repo-wide 62 of 280 rounds gain a real setting, dominated by postgres exposed in 48 and mariadb in 22; no round loses one, and no gained payload enables a service whose provider is not deployed. With disable set the scope shrinks instead, which is correct because a pruned provider is not deployed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The matrix gate rotates credentials between the two deploy passes so PASS 2
has to carry fresh values. It rotated every application block in the host_vars
file, and on a matrix inventory that is not the round. Provision applies its
--include as a strict intersection (filters.py:57-62), so it generates
credentials for at most derive_includes, while mirror_overrides.py adds a block
per mirror artefact carrying only image and version. Rotating those costs one
credential subprocess each for values provision never wrote.
matrix.py now passes --include derive_includes(app_id, variants=round_variants),
the same call 02_provision_inventory.sh:11 feeds provision, from the same
round_variants the provision step got one call earlier. The rotation set is
therefore a superset of what provision generated: every database_password still
rotates, because utils/manager/inventory.py:236-242 writes it under an id from
that same filtered list. Measured for web-app-wordpress: 35 ids.
The scope stops at the credentials. reset_credentials takes credential_ids
beside application_ids because the two halves shared one list, and narrowing it
narrowed _drop_user_passwords too: that resolves through
required_user_policies(roles_dir, application_ids), so 60 of 66 declared users
would have stopped rotating, root, backup and every mcp-* bot among them. The
user half keeps the full document list and is byte-identical to before.
The test fixture declared the same users in both roles, which made the
regression test pass against the regression. web-app-keycloak now declares a
user web-app-a does not, so the assertion has something to bite on.
The CLI aborts when --include matches no block instead of rotating nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The step banner announced a start and nothing else, so a phase duration had to be read off the next banner, and a phase with no output of its own was invisible. In the budget-killed Nextcloud#0 of run 33828309745 the credential rotation spent 17 minutes with nothing but the 30-second sampler between its banner and the next one, and reconstructing the split between provisioning, both passes, the DR drill and the rotation meant subtracting timestamps by hand.
The watch loop moves unchanged into _run_watched. _run now brackets it and prints the elapsed time and the exit code, so this holds for every phase, not only the rotation.
The test stubs _run_watched and pins the three things that matter: the exit code passes through, the opening banner is printed, and the closing line carries duration and rc. It is the first test over this module, whose absence was named earlier as the reason a wrong change to the pass sequence would turn nothing red.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The extras writer carved `applications` out of the deploy-facing twin and said
why in its own docstring: extra-vars replace rather than merge, and would strip
every generated credential. `users` carries the same hazard and stayed in, so
every swarm deploy ran with `-e @swarm-nfs-extras.deploy.yml` replacing a
nine-entry users map with the three from inventories/development/default.yml.
Everything not in those three then fell to its role default. For zammad-api-bot
that default is `{{ 42 | strong_password }}`, and utils/manager/value_generator.py
opens by warning what that means: it is a template, not a value, re-rendered at
every use site, so the task that creates an account and the task that later
authenticates as it read two different secrets. Playwright sent a 42-character
password that no deploy had ever written, and Zammad answered
Seed POST /api/v1/tickets failed: 401 Invalid BasicAuth credentials
on both swarm jobs of Zammad#1 while both compose jobs passed. Compose never
sees an `-e @file`; its driver emits only scalar pairs.
`users` now leaves the twin the way `applications` already does. The two deltas
the block actually carried, the swarm-test pubkeys for administrator and backup,
travel the merging channel instead: baked into base_overrides, handed to the
provisioner through --vars, and deep-merged into host_vars. ensure_swarm_keypairs
is split out so the keys exist before provisioning rather than at extras time.
Two things this restores beyond the 401. The rotation gate rotates
users.<name>.password, and swarm was overwriting the result with the defaults
straight afterwards, so user rotation was never actually exercised there. And
user-administrator/meta/users.yml ties the administrator account password to
ansible_become_password; the extras had been pinning it to default.yml's sample
literal instead, which sudo on these nodes does need to match.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The watchdog reads free space every 30s but surfaces it only in the abort
banner, so a job that dies without tripping the floor leaves no disk series at
all - which is exactly the silent-ENOSPC shape the function's own docstring
names. Reuse the reading the guard already takes and put it in the heartbeat,
so the next occurrence is decidable from the job log alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PASS 2 read the exact values PASS 1 wrote, so no deploy ever had to propagate a secret that moved. Every role that only writes a credential at creation time therefore stayed green without the update path being exercised once.
Both orchestrators now call `inventory credentials reset` after the recover drill and before the async pass: compose through the container in the new .rotate module, swarm through the matrix orchestrator with --mirror, because its inventory keeps one host_vars copy per node. --backup keeps the pre-rotation file for the post-mortem, since after a rotation nothing records what PASS 1 deployed with. `administrator` stays exempt in both: its password is ansible_become_password, and rotating it would lock the deploy out of its own host.
Expect roles that never re-apply a changed secret to fail PASS 2 from here on; that is the gate working. Both database engines already converge their root password on every run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When the host runs out of memory the Actions runner loses its heartbeat. GitHub then reports only 'lost communication': no step completes, so every if-always upload step is skipped and the rescue diagnostics, the inventory artifact and the Playwright results are all lost. Job 92710765494 died exactly that way.
The 30s poll in _run already did this for disk. It now also samples MemAvailable, MemTotal and the memory PSI stall, prints one line per poll, and aborts below 6 percent of RAM. The abort returns 75, which propagates to the EXIT trap in 00_one.sh, so the existing post-mortem apparatus runs normally instead of dying with the runner.
The sample line is the main point: artifacts provably do not survive a lost runner, streamed log lines largely do, and the numbers discriminate memory thrash from a kernel wedge on the next occurrence. The floor is relative to MemTotal so a dev box does not inherit a runner-sized threshold; measured margin across four real drills is 9-15x. PSI is sampled but not used as a trigger, since no drill has ever measured it.
mem_total_mb and mem_stall_pct join the existing host probes in utils/env/runtime.py and share the /proc/meminfo parser. The probes read only /proc, never the host-side NFS mount, which would block in D-state under the very condition they exist to catch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
bdb3b29a9 passed the watchdog floor to the drill as INFINITO_DISK_FLOOR_MB and
broke test-lint:
AssertionError: INFINITO_* keys referenced in code without a registry entry
INFINITO_DISK_FLOOR_MB
The prefix is reserved for the deployment env registry, where every key needs a
default in default.env or a handler that sets it. This value is neither: it is a
handoff from matrix.py to the script matrix.py starts, and it only exists for
the duration of that call. DRILL_EXTRAS, one line above, already goes the same
way under an unprefixed name.
So the key becomes DISK_FLOOR_MB and its declaration moves up beside
DRILL_EXTRAS in the `: "${VAR:?...}"` form the file uses for matrix handoffs,
leaving the INFINITO_* block below to the values that really come from .env.
Registering it in default.env would have satisfied the lint by writing 6144 down
a second time, which is the duplication bdb3b29a9 set out to remove; a nocheck
would have hidden the naming mistake rather than fixed it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill's space check required FREE >= PULLED, but the run is killed from
outside by matrix.py's watchdog as soon as the runner drops below 6G free. The
check knew nothing about that floor, so it waved through a drill that could not
finish: 19008M free against a 13961M pulled tree passes FREE >= PULLED, and the
sync then leaves 5047M, which is under the floor.
That is what happened to the arch swarm drill in run 30830377785. The device
sync reported the marker at 19:16:38.34, the watchdog fired at 19:16:39.73 on
its next 30s tick, and the step died with rc 75 thirteen minutes into work that
was already doomed when the precondition ran.
The requirement is PULLED + FLOOR, not FREE >= PULLED and not 2 * PULLED. FREE
is measured after the pull, so the first copy is already deducted; exactly one
further copy is ever added on top of it - the device copy during the sync, and
later the restore root, by which time base.sh has deleted the pulled tree. The
two extra copies are sequential, never simultaneous.
matrix.py owns the floor now and exports it as INFINITO_DISK_FLOOR_MB. It was
previously a bare `6 * 2**30` the shell had no way to see, which is why the two
sides could disagree about what "enough disk" means.
This makes the failure honest and immediate; it does not create disk. The arch
runner is short by about 1.1G, and closing that needs a capacity decision -
bigger runner, smaller test dataset, or reclaiming build cache before the drill.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docker info reports DockerRootDir as the DAEMON sees it, and
host_storage_constrained fed that string straight into shutil.disk_usage,
which statvfs()es it in the CALLING process's mount namespace. The two
coincide only when the caller is the daemon's host. In the nested act
container of the workspace jobs they do not, so os.statvfs('/var/lib/docker')
raised FileNotFoundError, matrix.py died, and because 00_one.sh invokes it
bare under set -euo pipefail the whole swarm drill went down with it. All
five workspace jobs of run 30560683608 failed this way, byte-identically,
across five base images and five nested distros.
docker_root_free_bytes() now takes the vantage point as a required keyword:
it measures the daemon's data root when that path resolves here, and the
caller's stated local path when it does not. matrix.py passes "/", which is
the same vantage it already trusts for its ENOSPC abort 270 lines earlier;
overlayfs passes statfs through to the upper filesystem, so a container of a
daemon measures that daemon's data-root filesystem through its own rootfs.
Nothing is caught and nothing is guessed. An unreachable daemon still raises
RuntimeError out of docker_data_root(), which is the guard this module exists
for, and local_vantage carries no default because only the caller knows where
it is running.
The compose-side twin loses its || true and its hardcoded /var/lib/docker
substitution, so an unresolvable root no longer measures a guessed filesystem.
Its compose.exec locality was already correct and is unchanged.
Verified against the pre-fix module under the CI condition: with the data root
stubbed to a path absent from this namespace, the old code raises
FileNotFoundError and the new code measures the local vantage; with the daemon
stubbed unreachable, the new code still raises RuntimeError.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The flag was decided by a fixed free-space threshold (100 GiB by default), which fired on roomy hosts and stayed silent on a 145 GiB CI runner that a single deploy round could fill. It now compares free space on Docker's data root against what the deploy declares it needs: the summed min_storage of the selected applications and their transitive dependencies.
The need is variant-aware. utils/storage/constrained.py overlays each app's meta/variants.yml entry before walking the service registry, so a round that trims services down is measured at its own size instead of the base configuration's.
Both entry paths pass their per-round variant map: cli/administration/deploy/development/init.py decides inside the round loop (compose and host share this via ci/one.sh -> dedicated.sh), and utils/tests/swarm/matrix.py feeds the decision into the swarm host_vars payload. --threshold-gib is gone; --force-storage-constrained still overrides.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill configured svc-bkp-remote-2-local with a fixed manager plus NFS provider list for every app, while the manager only runs a backup repository when the app's include closure pulls svc-bkp-volume-2-local or svc-bkp-secrets-2-local in. For an app whose container_backup consumer is statically off, such as svc-dns-unbound, user-backup never ran on the manager, so the pull was rejected with 'Permission denied (publickey,password)' and the fail-if-any wrapper took the whole unit down at drill step 4/9. Four roles reach that step today: svc-dns-unbound, svc-db-redis, svc-registry-cache and svc-registry-docker.
backup_repos.py expresses the placement once and matrix derives the provider list from it per round. derive_includes accepts the round's variant map because INFINITO_APP_VARIANTS only reaches the subprocess callers. The extras file keeps the device paths the drill reads back and loses the provider list nothing read from it, and the drill banners stop claiming backup legs that were never installed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Back to the pre-swarm-nfs-fix baseline:
- 5c2c8f531 + 985ec7eab: the swarm writer-quiesce DR drill. Deep-dive
attribution of CI run 29715516546 found it the dominant failure source
(17 of 23 jobs): [9b/9] handed-off generation mismatch, [6/9] quiesce
stack-rm network race, [4/9] remote-2-local pull stall, compose discourse
restore duplicate key.
- d6f980574: openproject swarm ingress to web:8080. Carried a leaking
proxy_internal_port lookup (no per-app default) that broke the seaweedfs
dependency front-proxy ('Key web not found'); the bypass-the-Apache-proxy
approach was never verified against downloads/X-Sendfile.
- 1dd153bad: xwiki replicas:1 + mem 3g. Did not resolve the xwiki superadmin
failure (still red in run 29715516546), so it is unverified.
Re-approach each from a clean baseline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Defer Swarm SQL replay to the disposable DR drill so shared writers such as Discourse are stopped before the destructive restore. Verify the selected generation with before/after database probes while preserving the Compose restore path.
The swarm deploy job now renders the same summary page as compose:
role runtimes, a swarm structure section (topology roles plus live node
list and service placement from the manager) and the Playwright table.
The matrix deploy tees its ansible output and prints the compose
matrix-deploy round/PASS markers, so the runtime parser segments swarm
runs per variant round too.
The runtime table unifies tasks and roles with a Type column (tasks
first), appends the recap total row and flags a mismatch between the
parsed sum and the recap total; rows are read from ROLES RECAP sections
only, which removes the double-counted TASKS RECAP task rows. Each role
row carries one column per host: executed, skipped or failed, with
ignored failures downgraded. The Playwright variant column drops the
redundant variant- prefix. Verified against the logs of runs
29656268771 (swarm) and 29620158441 (compose): every segment's parsed
sum matches the recap total within 0.05s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The DR drill ran in every variant job because its gate checked the
job-local plan index, which is 0 in each of the single-variant CI jobs;
gate it on the absolute round index so the destructive drill and the
backup host deploy run once per app, in the variant-0 job. Guard the
matrix steps with a disk watchdog that aborts visibly below 6G free
instead of letting the Actions Worker die silently on ENOSPC with no
artifacts. Cap node journald (256M) and DiD container logs (10m x 2),
which were unbounded across five nodes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.
Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The drill (backup/base.sh) no longer runs its own step-10 redeploy; it recovers device -> nfs export -> volume -> secrets and writes a sentinel. The matrix update pass (which runs next anyway) boots the recovered stack, and verify_recovered_marker.sh asserts the live marker after it. Saves one full stack redeploy per drilled app; the marker-gone check before recovery stays in 05_wipe_export.sh.
Also derive recover_drill.sh REPO from the script location instead of an INFINITO_REPO_ROOT shell-side default (lint: no INFINITO_* / absolute-path defaults in .sh).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Node image now builds FROM the distro infinito image (python, ansible, CLI baked in) instead of installing a per-node .deb, so there is no per-node bootstrap or timeout; the swarm CI job depends on the distribution build. Move compose/swarm.yml to compose/swarm/compose.yml and relocate the swarm matrix orchestrator from cli.administration.deploy.swarm to utils.tests.swarm (test code out of the shipped CLI).
Back the NFS export root with a disk-backed named volume instead of a RUNNER_TEMP bind: local-act RUNNER_TEMP resolves onto tmpfs and ganesha FSAL_VFS cannot export a tmpfs, so it loaded zero exports and the controller mount got ENOENT. Add optional compose/swarm/cache.override.yml wiring swarm nodes to the package-cache frontend when present, leaving CI without a cache unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
