The small GitHub runner shape (AWS, 72G root) ran out of disk in swarm and compose rows (#1329: WordPress#2 of run 41261531965, Nextcloud#6 and #7 of run 41262286962) while an empty instance-store disk nvme1n1 sat unused. mount_spare_disk.sh formats and mounts it at /mnt, but only on a GitHub-hosted runner outside act, and only a whole unpartitioned disk without any signature, mount or swap use of at least 75G; every decline leaves a notice. relocate_docker_dataroot.sh then bind-mounts /var/lib/containerd from the spare disk and moves the Docker data-root to /mnt/docker-host (not /mnt/docker, which is the nested daemon's volume that dedicated.sh wipes). It refuses before stopping anything when images, volumes, build cache, foreign containerd namespaces or shims exist or daemon.json is unreadable, and rolls back the bind mount and daemon.json on any later failure or cancel. Swap goes on / once containerd lives on /mnt, dump_state.sh logs disk size, model and free space per storage root, and the swarm watchdog samples / and /var/lib/containerd every 10 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On the meshed arm the layout joins the backup overrides through
merge_with_defaults and lands in every host's host_vars; the direct arm
carries no mesh value at all.
INFINITO_SWARM_MESHES names a layout file of applications overrides. The
controller mesh, the transport mesh and the stale-link scan read it through
the CLI's own spec reader.
The prologue hands the chosen declarations to the mesh writer, and the
controller mesh and the transport switch follow the first declared mesh
instead of a hardcoded 'swarm'.
Locally the controller is the act container on the host's network, so its
interface outlives the round. A link left by other declarations still routes
the pool, and a development round after a three-mesh one died on
'RTNETLINK answers: File exists'. The controller now drops WireGuard links
that hold a pool address but name no declared mesh before it joins; nothing
outside the pool is touched.
The extras looked up the 'swarm' and 'data' meshes by name and fell back to
the underlay when they were absent, so any other declarations ran swarm and
NFS outside the tunnel without failing.
The guarantees were asserted against the declared swarm and data pair,
so they proved two meshes work rather than that the planner works. A
deployment declares its own, and the review asks for any number of them with
the nodes arranged however the client wants.
The same invariants now run over generated topologies: one to eight meshes,
three arrangements each, hosts dealt out so some belong to several meshes and
some to none. 2247 subtests, four seconds, and the planner needed no change to
pass them.
Three runs of the transport benchmark, per module call:
become off become on
docker 247ms 246ms
ssh over mesh 215ms 1259ms
The connection is not the cost - ssh through the tunnel beats docker exec by
13% - and the tunnel is 4%. What the meshed arm pays is the sudo prompt: free
over the container connection, which runs as root, and 1046ms per escalated
call over ssh as the administrator. That is the 414ms per task the three-arm
comparison measured across a whole deploy.
A lab-only sudoers drop-in removes it. It is applied from the mesh prologue
while the inventory still reaches the nodes as root, so it is in place before
anything logs in as the administrator, and it touches no role: production
keeps 'administrator ALL=(ALL) ALL' and its password prompt.
The cost is that CI no longer exercises password sudo over the mesh. It never
exercised it on the direct arm either, which connects as root, and a wrong
password fails loudly rather than silently.
Run 37931817959 timed the four cells and two of them are real: a module
call costs 245ms over docker and 212ms over ssh through the tunnel, with or
without become on the docker side. The connection plugin is not the cost.
The escalated ssh cell is not real. The benchmark ran without the extras, so
ansible_become_password was absent and all 200 calls failed on 'Incorrect sudo
password' - 3.2s each, which prices a failing sudo rather than a working one.
It now passes the same two -e files the deploy passes.
Its exit code also reached the round, so a diagnostic turned three green
deploys red. transport_bench now discards its own rc: whether a deploy passes
is not a benchmark's business.
A second playbook-* file at the root makes 'playbook' a shared
delimiter-bounded prefix, and the structure lint reads that as a folder in
disguise: '2 files share the prefix playbook -> move them into ./playbook/'.
It failed test-lint, which gates the deploy chunks, so run 37929774651
produced no benchmark.
The alternative was renaming playbook-mesh.yml into a new playbook/ folder,
which is a structural change to the branch's own feature made for the sake of
a diagnostic. Renaming the diagnostic is the smaller move.
The local run missed it because the file was still untracked, and the rule
walks git ls-files.
The three-arm comparison on web-app-gitea#0 put the meshed arm at 414ms
per task over the direct arm, and the tunnel at 4% of that. What it could not
split is the rest: the arms differ both in connection plugin (docker against
ssh) and in escalation (root with no prompt against administrator with a
password sudo on every task).
A fixed set of module calls, timed either side of the transport switch with
become off and on, gives four numbers that separate the two. It costs about a
minute where a fourth deploy arm would have cost three hours, and it needs no
root login and no sudoers change.
INFINITO_SWARM_TRANSPORT_BENCH guards it, so a normal round times nothing.
The arm existed to answer one question: how much of the meshed arm's
cost is the connection plugin and how much is the tunnel. Runs 37592061960
(docker), 37753506661 (SSH on the underlay) and 37621556682 (SSH over
WireGuard) answered it on web-app-gitea#0, debian/ext4:
pass 1 5964s / 8511s / 8832s
pass 2 5319s / 7472s / 7705s
Leaving ansible_connection: docker costs +4700s, the tunnel +554s - 89.5%
against 10.5%. The answer is recorded; the scaffolding does not need to ship.
Reverts d792519725, 1b28f87b37, 89b8a8c8d1, 0470491e1a, 529c5ace43,
dd3d10a9a2, c20dd0e5f5, 458d8956e1, 63ade1e03b, d281555945 and 4adfb6e3d2.
95cfc8b7b7 is already superseded by bbfd82f89, which sets the profiling limit
upstream.
The meshed round was its own shape: a mesh-only first pass, one
application deploy, and a second only when a drill needed one. Comparing it
with the direct arm meant adjusting for a missing pass and a stripped browser
suite, and every timing answer carried a caveat.
Both arms now run deploy, converge, drill, redeploy, converge, verify. What
the mesh needs moves into a prologue that installs no role: tunnel up on every
node, keys rotated, controller joined, transport switched. Both application
passes then run over the tunnel and both print their PASS banner, so a meshed
row and a direct row differ only in how they were reached.
The credential reset stays in the prologue rather than moving to the direct
arm's slot between the passes. Its mirror copies the manager's host_vars over
every other host, keeping only applications.svc-net-wireguard, so after the
switch every worker would inherit the manager's mesh address and three
connections would land on one node.
The drill probe now gates both arms. A backup host deployed for an application
with no NFS-flagged volume is several minutes spent on a drill that skips
itself, and that was true on the direct arm before this branch existed.
Narrowing the meshed drill to its backup half bought budget by giving
up the recovery legs over the mesh. The budget overrun is worth measuring
before it is designed around, so the pre-drill pass and the full teardown and
recover come back. Reverts a27b093c4.
The pre-drill pass cost a whole deploy so the drill would have a stack
to tear down and a later pass to bring it back. n8n variant 0 spent 11970s in
that pass alone and hit the 20400s budget before the real deploy started.
The meshed round now deploys once, as the rest of the branch does, and runs
the backup half of the drill after it: backup host plus DRILL_STAGE=backup,
around 550s against roughly 12000s. Teardown, the recover CLI and the bring-up
onto the recovered export are transport-agnostic and the direct arm proves
them on every application the axis rotates there. What the meshed half adds is
the backup host joining the data mesh and running its units over the tunnel.
Step 3b, which caught the Nextcloud volume-repo failure, is inside the half
that is kept.
The drill tears the stack down and restores the data behind it, leaving
the bring-up to whatever deploys next. A meshed round deploys once, so after
a drill nothing brought the stack back onto the recovered export -- and the
round still reported green, because the marker is asserted against the
export rather than against a running stack.
That is observed, not predicted: a meshed BookWyrm round passed with
backup + restore DR drill took 81s (rc=0)
==> DR drill PASSED: ... verified on the NFS export after the update pass
and nothing after it. The cluster finished the round torn down.
A meshed round now deploys before the drill as well, and only when the drill
says it is coming. That first pass carries SWARM_DR_PREPASS, which skips the
end-to-end suites because its stack is about to be thrown away; the pass
after the drill is the full one, so the suites run once and against the
recovered stack. A round with nothing to restore still deploys exactly once.
The arguments every deploy of a round shares move into one mapping, and the
environment the drill reads into one helper, which is what keeps the file
inside the size gate after the addition.
One conflict, in the MCP exclusion list of requirement 035. Both sides
edited the same line: main added svc-net-tor-smtp, this branch replaced the
three svc-net-wireguard-{core,plain,firewalled} entries with the single
svc-net-wireguard it deploys. Resolved to the union of both — 24 entries —
which is what the merged tree actually holds.
Everything else merged clean and was checked for overlap rather than
assumed: no reference anywhere still names the three replaced roles;
matrix.py kept main's PRIMARY_APPS alongside the reordered round, and the
bootstrap pass does not need it, since only one template reads it and
group_vars defaults it empty; and main's ansible.cfg addition sits in
[defaults], so the ssh_args the mesh tool derives its own connection
settings from are untouched.
The round replaces its first deploy with the mesh bootstrap and keeps the
deploy after the seam as the only one that installs the applications: the
same roles at 2.98s a task instead of 0.74s, some five thousand tasks of it,
which is most of why a meshed variant 0 ran past the 20400s budget.
The backup node and the DR drill move behind that pass, because there is
nothing to back up before it; they are held together in one step so the two
call sites do not drift. Credentials still reset before the deploy that has
to carry them.
A direct-transport round is untouched and still deploys twice.
A meshed round deployed every role twice, once over the container
connection and once again over the tunnel, which left the most expensive
transport carrying nothing but idempotence checks. The first pass now brings
up the mesh and nothing else, so the pass after the seam is the only one
that deploys the applications.
The converge drops the backup inventory with it. That node is deployed after
the rotation now, so it renders the current mesh on its first run and has
nothing to converge from.
Only the first round deploys the backup host, so a later round meets a
container that has never been provisioned: the mesh role is the first thing
to reach it, and installing the tooling fails because nothing has populated
its package index. Every meshed variant above zero died there.
The converge now takes the sibling inventory only when the round brought
that node up, which is the only round whose data plane has a second spoke
to carry.
Reverts the apt refresh added for the same symptom. It reported the index
fresh on exactly the host that had none, so it never addressed this and the
rounds that do deploy the backup node were already converging without it.
The converge ran over devices.yml alone, so the backup host kept the
identity it was deployed with while the hub had already been told the new
one. Nothing caught it: the backup node is not a spoke of any play that runs
after the seam, so its half of the data plane went quiet without failing.
The play forces the axis on, so reaching it through the sibling inventory is
independent of the backup deploy's own mesh state.
The second write rotates, so the converge, the controller join and the second
pass all run against keys that changed mid-deploy. Also trims the reset
narration.
write_mesh takes rotate, so the round mints a fresh keypair for every member
between the passes and the converge that follows carries them. The rotation is
asked for rather than inherited from a mirror that happened to wipe the keys,
which is what the requirement's two rotation criteria describe.
The mirror wiped each host's mesh block, so the write that follows minted a
fresh keypair for everyone and the converge re-keyed tunnels the deploy was
already running services over. The NFS state mount rides the data mesh and is
soft in CI, so a re-handshake dropped it; the second pass then found local
data under /var/lib/infinito and refused to mount over it.
Production never mirrors host_vars and mounts hard, so this only removes a
scenario the harness invented for itself.
It holds a mesh address and routes the pool, but it is the machine running
the play. Pointed at itself over ssh, every task a role delegates to it dials
its own tunnel address instead of staying local -- which is where the second
pass died, on the controller-side NFS mount, after running 189 tasks on the
manager and 176 on each worker over the mesh without a failure.
user-administrator creates the account from users.administrator.password,
which the development inventory states outright. The value ansible sends for
become is generated per host and never reaches the account, so every sudo
over the mesh was refused. Nothing noticed before because the container
connection escalates to root without a password.
The extras already carry that inventory's users, so the same value now rides
the deploy-facing twin as ansible_become_password.
The nodes are reached by container name over the docker connection, which
only the lab network resolves. The controller is not on it, so it is handed
the addresses default.env already pins for the five hosts.
The rotation gate mirrors one host's host_vars over every other, so the mesh
write that follows mints a fresh keypair for all but the mirror source. The
nodes kept running the keys of the first pass, so the hub turned the
controller away and its handshake gate failed the deploy.
Re-render every node before the controller joins, so the hub authorises the
set the inventory now holds.
Nodes resolve each other through the extra_hosts compose hands them; the
controller is not a container and got no such map, so wg setconf could not
resolve the hub and the swarm interface failed to come up. Pass the same
name-to-address pairs the compose file builds from the topology.
The round already had two passes. The first brings the mesh up over the
container connection; the inventory is then switched to the mesh addresses and
the second pass runs over the tunnel, so no extra round and no extra CI time.
A full deploy of every role over the mesh is a stronger proof than a narrow
bootstrap round would have been.
It runs the role like any member, but it drives the deploy rather than being
deployed to, so it keeps a local connection while the nodes keep the container
one.
kevinveenbirkenbachandClaude Opus 5.5
evangelostsak