One conflict, in the category catalogue: this branch added a virt category for
the sandboxed container runtimes, the other did not.
Keeping our side alone would have closed the conflict and broken the build. The
same merge introduces a test requiring every category to declare a hanzi label,
unique among its siblings, and virt predates that rule and carries none. It now
declares 沙箱, checked against its ten siblings for a collision.
The local exec helpers covered the development container and a throwaway
sidecar next to it, but nothing reached into an application the stack had
already deployed. Swarm had swarm-app-exec for that; compose had no
counterpart, so the only remaining route was a nested docker exec, which the
contributor rules forbid.
The helper goes through container.sh's positional form rather than its cmd
environment variable. container.sh runs that variable through `sh -lc`, so an
already-composed command would be split a second time; passing argv through
untouched leaves exactly one shell level, inside the target container.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
web-app-openwebui 0,1,2 was killed by the 350-minute action timeout with no
failed task anywhere in its 1.33M log lines. The sweep admitted fedora at
remaining=10372s after debian took 10028s, then never looked again: the
budget was only checked between distros. Its internal deadline passed 594
seconds before GitHub killed the job - the run knew it was over time for ten
minutes and had no way to act. Worst-case runtime was deadline plus the full
duration of the last admitted distro, unbounded.
The trigger was a single stalled task (803s for a no-op that takes 1.3s
warm, and a fully cold transaction 8 minutes later cost 24s), but that only
decided which job hit the wall. The wall itself is structural.
remaining is already computed at the top of the loop, so wrapping the call
in timeout adds no new state. rc 124 becomes 'skipped' rather than 'failed',
because a job that stops at its budget is the documented intent, not a
defect - test_distro_budget_fits_timeout.py already asserts this invariant,
which until now was simply untrue.
Two conditions narrow it, and both matter:
dur >= remaining - 5 keeps an inner timeout that fires early as a failure.
Verified: timeout returns 124 only for its own kill and passes inner exit
codes through unchanged (0 for success, 3 for an inner exit 3).
passed > 0 is the one that protects the test result. Without it, a job whose
very first distro overruns would report 'skipped' and exit 0 - green while
having verified nothing. That would have been a silent false pass across all
243 jobs of a run. The guard was executed over the full truth table,
including passed=0.
-k 60 is deliberate: one.sh execs into dedicated.sh and its ansible and
docker children can outlive SIGTERM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
07_compose_minimal.sh claimed to disable 'all variable (group-conditional)
services' and then named seven by hand. tor was not among them, so a
deploy meant to be minimal pulled svc-net-tor, which hard-depends on
web-svc-mirror, whose group-conditional prometheus flag the closure
resolver treats as true — dragging the whole web stack in. Steps 12 and
13 passed no disable at all, which is why a mariadb swarm test deployed
keycloak, mailu, matomo, prometheus and dashboard.
The set is now derived from the contract the repo already enforces
(tests/integration/roles/meta/services/test_dynamic_flags.py): a key is
group-conditional when enabled OR shared is a Jinja expression. Checking
only enabled would miss mariadb, postgres, redis, memcached, typesense.
Exclusions are load-bearing, not cosmetic: dashboard because disabling
the primary app aborts init; javascript because it aliases cdn, which
web-app-dashboard declares a hard dependency with no runtime fallback;
mariadb and postgres because steps 12 and 13 test them.
variable_services assigns to a local first — disable="$(cmd)" does not
set the exit status of a simple command, so a failing CLI would have
yielded an empty disable set and steps 12/13 would have run the full
deploy and reported green.
roundtrip could not take disable at all; it now reaches both the compose
and the swarm leg.
Not verified by a deploy: 33 of the keys are new to step 07, and the
variant axis of svc-db-mariadb/svc-db-postgres collapses because tor is
its only dimension.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
web-app-openwebui 0,1,2 was killed by the 350-minute action timeout with no
failed task anywhere in its 1.33M log lines. The sweep admitted fedora at
remaining=10372s after debian took 10028s, then never looked again: the
budget was only checked between distros. Its internal deadline passed 594
seconds before GitHub killed the job - the run knew it was over time for ten
minutes and had no way to act. Worst-case runtime was deadline plus the full
duration of the last admitted distro, unbounded.
The trigger was a single stalled task (803s for a no-op that takes 1.3s
warm, and a fully cold transaction 8 minutes later cost 24s), but that only
decided which job hit the wall. The wall itself is structural.
remaining is already computed at the top of the loop, so wrapping the call
in timeout adds no new state. rc 124 becomes 'skipped' rather than 'failed',
because a job that stops at its budget is the documented intent, not a
defect - test_distro_budget_fits_timeout.py already asserts this invariant,
which until now was simply untrue.
Two conditions narrow it, and both matter:
dur >= remaining - 5 keeps an inner timeout that fires early as a failure.
Verified: timeout returns 124 only for its own kill and passes inner exit
codes through unchanged (0 for success, 3 for an inner exit 3).
passed > 0 is the one that protects the test result. Without it, a job whose
very first distro overruns would report 'skipped' and exit 0 - green while
having verified nothing. That would have been a silent false pass across all
243 jobs of a run. The guard was executed over the full truth table,
including passed=0.
-k 60 is deliberate: one.sh execs into dedicated.sh and its ansible and
docker children can outlive SIGTERM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In job 91983553540 all five lab nodes derived the same pool name,
infinito_ci_unknown, and the run died on `cannot destroy
'infinito_ci_unknown': pool is busy`. The name came from `hostname`, which
returns non-zero inside the node containers, so the `|| echo unknown` fallback
answered for every one of them. One node won the race and created the pool; the
others found it, took it for a leftover, and tried to tear down a pool a
sibling was actively using.
234c1158c already fixed this collision class three days ago by deriving the
name from `hostname` instead of a constant, and its message named the zfs
exclusion in resolve.sh as the only thing keeping the collision out of sight.
That fix was never exercised, because the exclusion stayed up until ed1121053
removed it - and it does not work, because the call it rests on fails in the
containers it was written for. Guessing the identity is the defect; the
playbook has known it all along as inventory_hostname, and ci/one.sh knows its
own deploy mode. Both now state it, as a required fourth argument with no
default, the way SIZE is already stated. verdict() stops guessing too: every
row it wrote into the job summary said `unknown`, which is why the summary was
never worth reading.
The second half is independent of the naming. reclaim_stale_pool ran
`zpool destroy -f` as a bare command under `set -euo pipefail`, the only
failure path in this script that bypassed decline(). A pool that refuses to be
destroyed is by definition not a leftover - something holds it - so that case
now declines through the usual contract: a drawn pick falls back and reports,
a stated one fails loudly as required-but-unavailable. A reclaim can no longer
be more destructive than the state it exists to clean up.
Verified over the real invocation path rather than a model of it: an
ansible-playbook run using the same `script:` task and docker connection as
compose/swarm/playbook.yml, against three containers in one play, has each node
address its own pool - infinito_ci_race-mgr-01, -wrk-01, -nfs-01 - with
failed=0 throughout. A stubbed-pool harness covers the paths that need a kernel
module this machine does not have: a sibling holding the pool declines with rc
0 instead of ending the run, the same case with a stated pick exits 1 through
the contract, a genuine leftover is still destroyed, and a missing identity is
a hard error. The earlier replacement harness (ext4 recognised, btrfs
replacing it, no loop leaked) still passes against the stricter signature.
Pool names now carry dashes, which zfs_namecheck.c accepts; at 42 characters
they sit far inside the 240 limit and match no reserved prefix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role README claimed the backup machinery "operates transparently against the mount
paths" and demoted the source-of-truth rule to advice in a prose box - both written
before 2efc73d3b made that rule the code's behaviour. It now states what is enforced:
this role skips export-backed volumes, svc-bkp-nfs-2-local is their single capture, and
manager-pinned roles, roles forcing another compose_mode_force and nfs: false opt-outs
stay covered here. It also names the limit the code actually has - the exclusion checks
that some host is in the svc-bkp-nfs-2-local inventory group, not that the role ever ran
successfully - because a README promising more than the code delivers is worse than one
promising less.
Its Schema section carried the role's timer flow; it now carries the mechanism the
change set is about: meta/volumes.yml -> swarm_nfs_backed() as the single predicate ->
its two consumers (the compose rewrite and the backup exclusion) -> the inventory-group
gate -> the export host's single capture -> the restore path with its mountpoint probe
-> the drill's [3b] guard against over-exclusion.
The two swarm READMEs drew the drill with eight steps and pre-773a25ab6 semantics. Both
now draw the nine that exist, including what makes the proof a proof: the pulled tree is
wiped before the device recover, so the device holds the only copy left; the live export
is wiped with the marker proven gone before it is restored; and [9] deletes the marker
server-side before restoring the volume through its bound backing store.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three steps of the DR drill, one theme: every step now proves what it claims.
[9] restored into the volume's mountpoint while the stack was down - a bare directory,
because docker mounts a volume declaring its own backing store only while a container
holds it. The payload landed on the node disk and the real mount shadowed it on the next
deploy: success reported, nothing delivered, and roughly a gigabyte added to the very
device the drill later runs out of. It now deletes the marker server-side, binds the
backing store at the mountpoint, runs the recover CLI through it, and asserts the marker
on the NFS server's own disk. That also exercises the positive branch of the guard from
773a25ab6 - its refusal path is unit-tested, its pass-through was tested nowhere. When
the manager exposes no backing store the branch skips with a reason: docker volume
inspect is node-local, and with replicas: 1 the volume object usually lives elsewhere.
[3b] replaces a proof that used to happen by accident. The double capture 2efc73d3b
removed also showed, incidentally, that the volume unit really writes - the drill found
its marker in the volume repo too. The new step checks it directly: the newest
backup-docker-to-local generation must be at least as new as DRILL_START_TS (taken on
the manager, the same clock that names generations) and hold a non-empty file under a
volume subtree. Its failure message names the suspicion it exists for: an exclusion that
swallowed every volume. With no volume unit deployed it skips, tracked by
VOLUME_TRIGGERED alongside the existing SECRETS_TRIGGERED.
[5] prints du -sm per machine and repo when the space precondition fails, so the next red
run names what fills the pulled tree instead of leaving it to arithmetic.
Four adversarial review rounds found five defects in this block and confirmed each fix:
an NFS mount reconstructed from driver options the codebase never emits (it is a bind),
the recover CLI's second positional passed a volume name where it takes a host, a missing
volume object aborting the drill instead of skipping, and an umount ordered before the
return-code check, which under set -e would have killed the drill on a busy mount with no
diagnostic and left the bind behind.
None of this has run in a swarm yet. Every failure path is loud and names its cause.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pick was made once per matrix entry, in a workflow step ahead of the drill,
and every distro of that entry then ran on it. Coverage followed: one entry
exercised one filesystem, and the swarm never reached zfs at all, because
resolve.sh's userland_for() kept it out of any entry containing arch, centos or
fedora - which is every automatic sweep. The draw now happens inside the loop in
distros.sh, once per distro, so a five-distro entry covers zfs with 87% and all
three kinds with 62% probability instead of one kind with certainty.
The workflow input changes meaning with it. It was a single kind, offered as a
choice in the manual entry; it is now a space-separated allow-list. Empty draws
from all three, and naming kinds narrows the draw to them and marks the pick
required, so a host that cannot deliver one of them fails rather than quietly
running on something else. A name that is not a filesystem this project puts a
data root on is rejected outright instead of reaching the applying step.
For the draw to mean anything the images have to carry the userlands, so
scripts/install/zfs.sh becomes scripts/install/filesystem.sh and installs ext4
and btrfs alongside zfs. That is not symmetric work: btrfs-progs does not exist
in the EL family at all, because RHEL dropped btrfs, and a plain
`dnf install btrfs-progs` there fails the whole transaction and takes e2fsprogs
with it. EPEL carries it, so the EL family goes through EPEL from the start -
split by distro rather than by a failed first attempt, since Fedora has no
epel-release to fall back to and routing a transient failure there would turn it
into a fatal one. verify() now demands all four binaries, so a packaging
regression fails the build instead of leaving a hole the drill discovers later.
Measured on all five bases: debian trixie, ubuntu 26.04, arch, fedora 44 and
centos 10 each end with mkfs.ext4, mkfs.btrfs, zfs and zpool present and ldd
fully resolved.
Two defects only became reachable once a single job could ask for different
filesystems in turn:
* docker_dataroot.sh answered "already prepared" and returned for any mount it
found, so from the second distro on, every draw would have been silently
ignored on a host that survives the sweep - the runner in compose and host
mode. It now compares what is mounted against what was asked for and
dismantles the old root when they differ. Everything docker cached there
goes with it; that is what covering more than one filesystem per entry
costs.
* reclaim_stale_pool ran only when the new pick was zfs, so a zfs to btrfs
transition left the previous distro's pool imported on the shared kernel,
pinning a loop device over an image its dead container had already unlinked.
It now runs on every pick, which needed the /dev/zfs handling lifted out of
the zfs branch into zfs_device_ready().
The restore_docker trap is armed before anything can stop docker, rather than
after, so a failure while dismantling a data root cannot leave a host without a
daemon. resolve.sh splits its allow-list into an array before testing it, so a
whitespace-only input is the empty list it looks like rather than a narrowed one
that yields an empty pick marked required. pick.sh clears both keys when it
declines to draw: they are declared in default.env, so a generated .env carries
whatever was exported when it was made, and a caller gating its apply on the
value would otherwise repoint the developer's own docker in the very case the
skip exists to prevent.
Verified by execution rather than by reading: a privileged container ran
docker_dataroot.sh with real loop devices and mkfs through ext4 -> ext4 -> btrfs
-> ext4, confirming the recognition, the replacement in both directions and that
no loop leaks across it; a second harness with a recording zpool stub confirmed
that a btrfs pick does reclaim a stale pool, that no pool means no destroy, and
that a host without /dev/zfs still completes an ext4 pick without touching zpool.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dockerd's zfs graphdriver resolves `zfs` on PATH and opens /dev/zfs before it
accepts a data root (moby daemon/graphdriver/zfs/zfs.go:54,59), so the userland
has to sit in the image, not on the host. Only the apt images carried it, which
is why resolve.sh's userland_for() kept zfs out of every entry containing arch,
centos or fedora - that is every automatic swarm sweep, so the zfs data root and
the zfs branch of the backup launcher were never once exercised. Run
30781723102 shows it: "zfs left out: not every distro of this entry carries the
userland".
scripts/install/zfs.sh joins the three existing install scripts in the image
build and takes a different route per distro, each one run in a real container
of the matching base image before landing here:
debian trixie 2.3.2 zfsutils-linux, contrib enabled first
ubuntu 26.04 2.4.1 zfsutils-linux
arch 2.4.3 the AUR PKGBUILD, which configures --with-config=user
fedora 44 2.4.3 zfsonlinux repo plus a dummy zfs-kmod provider
centos 10 2.2.10 the same, from the epel path
The dummy is what keeps the RPM route usable: the `zfs` package carries an
ungated `Requires: %{name}-kmod`, so a plain `dnf install zfs` resolves through
zfs-dkms and installs 389 extra packages and 1.44 GB of kernel and toolchain to
build a module against a kernel the container never boots. Providing zfs-kmod
from an empty package keeps the same install at 60 packages with no kernel
package at all, and the transaction is checked afterwards so a regression fails
the build instead of bloating the image. The repository is removed again once
the userland is in, because the dummy pins one version and an upgrade would
otherwise re-resolve the kmod at deploy time.
Version drift against the host module is not a concern: zpool create enables a
feature only when the loaded module reports it through /sys/module/zfs/features
.pool (cmd/zpool/zpool_main.c, module/zcommon/zfeature_common.c), and libzfs
compares no versions at all (lib/libzfs/os/linux/libzfs_util_os.c:123-134), so a
newer userland degrades rather than breaking.
With every image carrying it, userland_for() and every_distro_carries() have
nothing left to decide and go. Two defects that only became reachable once zfs
could be drawn are fixed in the same move:
* /dev/zfs is a misc device, so the old lookup in /proc/devices could never
find its major and the node was never created. A node container's /dev is
populated at creation and udevd is masked, so nothing else fills the gap
either: the first bring-up of every job would have declined, blaming a
module that modprobe had just loaded. The minor now comes from /proc/misc.
* Pools live in the kernel the lab nodes share with the runner, while the
teardown only removes containers, and the node hostnames repeat across the
distro sweep. From the second distro on, zpool create hit the previous
iteration's pool. It is destroyed and its loop detached first.
The node image drops its own apt-only zfs branch and instead masks the six boot
units the userland enables by preset, which on a node can only fail: the pool is
created after boot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Docker mounts a volume that declares its own backing store (Options set: nfs,
cifs, bind) only while a container holds it. recover.py resolved the volume's
Mountpoint and rsynced into it regardless, so with the consuming stack stopped -
the state its own docstring demands - the payload landed on the node's disk
under the bare mountpoint directory, and the next stack start mounted the real
backing store over it: a restore that reports success, delivers nothing, and
leaves an invisible, unreclaimable orphan behind.
That orphan had weight. In run 30859490335 the swarm drill's [9/9] volume leg
restored ~1 GiB of nextcloud_data onto the manager's 30 GiB loop device in
exactly the minute the device crossed SeaweedFS's 2 GiB minFreeSpace floor -
the last straw on a device already >=93% consumed (the sizing is ea14535f4) -
after which every authenticated Nextcloud request died on an unwritable object
store.
recover.py now probes the mountpoint with `mountpoint -q` (over ssh for
--docker-host) whenever the volume declares Options, and dies naming the
mechanism and the way out instead of writing a single byte. Plain local volumes
without Options keep the old path untouched - their _data directory on the node
disk is the legitimate target.
The drill skips its [9/9] volume leg for such volumes with the reason in the
log: its data restore already happens in [8/9] straight into the NFS export,
which is the restore path the role's README prescribes for NFS-backed volumes
on swarm. The leg never proved anything for them - it only filled the disk.
Three new unit tests pin the contract: unmounted refused, mounted accepted,
remote probed via ssh with the exact command line.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every lab node got an identical 30 GiB loop device because docker_dataroot.sh
defaulted SIZE to 30G and no caller ever stated it. The swarm manager hosts the
in-cluster registry, ~60 containers, and - by deliberate placement, since /data
is NFS-unsafe - the SeaweedFS object store, whose sizing filter claims 14 GiB
of slots plus a 2 GiB minFreeSpace floor. The idle nfs-server node got the same
30 GiB and used 1% of it.
In run 30859490335 the manager's device stood at 30G/30G (98%), SeaweedFS
crossed its floor at 02:04:45 and marked all 14 volumes unwritable, permanently.
Nextcloud in variant 0 stores objects in that SeaweedFS via OBJECTSTORE_S3_*,
so every authenticated page render performed an object write that failed as a
10-attempt assign ladder - 15 requests, 15 bursts of exactly 10, burst end
matching the nginx 500 to the second. That is the "🐝 web-app-nextcloud 0"
failure: 5 addon specs, 3 attempts each, all dead on the post-login page.
The default is gone: SIZE is now a required argument and a missing value dies
naming the call site. The swarm playbook states it per node - 60G for the
manager, 30G for the rest. 60G covers the observed demand (>=31 GiB: a full
device plus the denied writes) with headroom; the image is sparse, so the
raised ceiling costs only what is actually written, and the runner-side
watchdog at 6 GiB free remains the backstop. The compose and host workflows
state their unchanged 30G explicitly.
Not addressed here: what fills the first 28 GiB (registry + image layers +
volumes composition is unmeasured - the rescue artifact holds exactly one df
snapshot), and the drill-side shadow restore that supplied the last ~1 GiB;
both are separate changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
bdb3b29a9 passed the watchdog floor to the drill as INFINITO_DISK_FLOOR_MB and
broke test-lint:
AssertionError: INFINITO_* keys referenced in code without a registry entry
INFINITO_DISK_FLOOR_MB
The prefix is reserved for the deployment env registry, where every key needs a
default in default.env or a handler that sets it. This value is neither: it is a
handoff from matrix.py to the script matrix.py starts, and it only exists for
the duration of that call. DRILL_EXTRAS, one line above, already goes the same
way under an unprefixed name.
So the key becomes DISK_FLOOR_MB and its declaration moves up beside
DRILL_EXTRAS in the `: "${VAR:?...}"` form the file uses for matrix handoffs,
leaving the INFINITO_* block below to the values that really come from .env.
Registering it in default.env would have satisfied the lint by writing 6144 down
a second time, which is the duplication bdb3b29a9 set out to remove; a nocheck
would have hidden the naming mistake rather than fixed it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hostname(1) is not installed in every node image. In run 30830377785 the arch
drill printed
trigger_units.sh: line 22: hostname: command not found
starting svc-bkp-local-2-device.11.6.0.infinito.nexus.service on
so the node name came out empty on all three call sites. The cosmetic one is the
progress line; the damaging one is the journal dump path, which becomes
"${unit}..journal.txt" and is identical on every node. Four hosts writing the
same filename into the shared dumps directory means the artifact keeps whichever
one wrote last, and a unit failure gets investigated against another host's
journal.
uname is coreutils and present wherever the drill runs. Reading it once also
drops three subshells from the loop.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill's space check required FREE >= PULLED, but the run is killed from
outside by matrix.py's watchdog as soon as the runner drops below 6G free. The
check knew nothing about that floor, so it waved through a drill that could not
finish: 19008M free against a 13961M pulled tree passes FREE >= PULLED, and the
sync then leaves 5047M, which is under the floor.
That is what happened to the arch swarm drill in run 30830377785. The device
sync reported the marker at 19:16:38.34, the watchdog fired at 19:16:39.73 on
its next 30s tick, and the step died with rc 75 thirteen minutes into work that
was already doomed when the precondition ran.
The requirement is PULLED + FLOOR, not FREE >= PULLED and not 2 * PULLED. FREE
is measured after the pull, so the first copy is already deducted; exactly one
further copy is ever added on top of it - the device copy during the sync, and
later the restore root, by which time base.sh has deleted the pulled tree. The
two extra copies are sequential, never simultaneous.
matrix.py owns the floor now and exports it as INFINITO_DISK_FLOOR_MB. It was
previously a bare `6 * 2**30` the shell had no way to see, which is why the two
sides could disagree about what "enough disk" means.
This makes the failure honest and immediate; it does not create disk. The arch
runner is short by about 1.1G, and closing that needs a capacity decision -
bigger runner, smaller test dataset, or reclaiming build cache before the drill.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A job whose nodes all declined the filesystem tested none of it, and the table
row saying so reads like an ordinary result. That is how the debian arm lost its
zfs coverage without anyone noticing: docker_dataroot.sh exists to exercise the
snapshot mode of svc-bkp-volume-2-local, which engages only on btrfs and zfs,
and on that arm it has been engaging on neither.
The summary now carries an explicit warning when no node applied the requested
filesystem and at least one declined it. Partial coverage is coverage and stays
unflagged; a job with no verdicts at all still exits early.
Deliberately a finding, not a failure. Stating a filesystem instead of `auto`
would set REQUIRED=true in resolve.sh and turn a decline into a hard error, but
that would paint the debian arm red for a cause that is understood and not yet
fixed - the node image has no zfsutils candidate without the package cache. The
goal was never to enforce the pick, only to stop losing it silently.
Exercised against three real job logs from run 30781723102: the debian arm
(5 nodes, all declined) produces the warning; the ubuntu arm (10 nodes, zfs
applied) and an ext4 job do not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard compared free space against USB_SIZE_MB = PULLED_MB*2+256, which is
the nominal size of an image 02_luks_device.sh creates with truncate - sparse,
so nothing ever allocates it. It also ran after that script, i.e. once the image
was already created, formatted and mounted, so the image's own footprint was
already subtracted from the df it read. It charged the pulled tree a second time
while that tree was counted as used, not free. In run 30781723102 it demanded
28108M against 16582M and refused a drill that fits: the peak is 14555M
additional, derived from rescue-diagnostics-swarm-web-app-nextcloud-0, and the
same run's debian job contains a full 9/9 completion of the same drill.
Rather than predicting the image's ext4 overhead - two earlier attempts at that
constant were wrong in both directions - both thresholds are now read:
df on DEV_MOUNT reports what the encrypted filesystem actually offers after
mkfs, and df on DIR_BACKUPS reports what the host still has for the copy the
sync writes into the sparse file and, later, the restore root. Both are compared
against PULLED_MB, a measured quantity. USB_SIZE_MB keeps sizing the image.
Every abort now tears the device down (umount, luksClose, rm -f); a refused
drill previously left the mapping and the image behind.
Exercised against the recorded numbers and six edge cases: 26624/16582/13926
passes where the old guard refused, an undersized device or host is refused, and
an empty or non-numeric df reading is reported as unreadable.
Both thresholds sit exactly at PULLED_MB, so second-order allocation inside the
image (extent trees, directory blocks) is uncovered - tens of MiB against a
13.9 GiB tree. The drill has not been run with this guard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The script stops the daemon at :157-158 and starts it again only on the success
path at :220. decline() exits 0 and set -e exits non-zero, and both walk past
that start, so any other outcome leaves the host without docker. One of the
three callers is test-deploy-host.yml:171, where the script runs on the GitHub
runner itself rather than in a lab node container.
An EXIT trap rather than a restart inside decline(), because decline is not the
only affected exit: arm_autoclear's exit 1 and every set -e abort at mkdir,
truncate, zfs create, mkfs, mount, btrfs subvolume create and the daemon.json
writer sit in the same window. The trap is armed only after the stop, and
DOCKER_STOPPED=false after a successful start makes it a no-op on the green
path. A failed restart is reported rather than swallowed, so it cannot hide the
failure that led there.
This hazard predates 02dc003bb: the loop-device decline added in the commit
before it has the same property. 02dc003bb widened it by putting a second
decline behind the stop.
Verified with a stubbed-PATH harness driving the real script: after a failed
zpool create the call log contains `systemctl start docker`, and it comes after
`losetup -d`, so the teardown still precedes the restart. REQUIRED=true still
exits 1, the success path is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three places where a data-root verdict was produced and then lost.
report.sh matched `requested=[a-z]+`, which never matches `ext4`. Against the
real deploy log of a run that recorded five ext4 verdicts, the old pattern
yields 0 matches and the summary table was silently empty of them; the fixed
pattern aggregates the same five into one row.
docker_dataroot.sh wrote the reason only to GITHUB_STEP_SUMMARY, which is unset
inside the lab node containers - so on exactly the machines that decline, the
reason went nowhere. It now rides on the machine-readable report line too. This
is what leaves the mariadb drill's decline reason unrecoverable from run
30781723102.
resolve.sh's exclusions went to stderr only, i.e. into the deploy log rather
than the job summary.
The reason is emitted as a list below the table, not as a column: reasons embed
per-node data (the pool name carries the hostname, plus the loop device), so
folding them into the `sort | uniq -c` key would split one five-node row into
five one-node rows exactly in the failure case the reason exists to explain.
`reason=[^']*` keeps ansible's closing quote out of the value, and `|` is
replaced so a reason cannot break the markdown table.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The script promises at :12-13 that a host which cannot deliver the stated
filesystem declines and reports when REQUIRED is false. The zfs branch did not:
`zpool create` stood unguarded, so a random pick that the host could not satisfy
exited 1. In run 30781723102 that killed the swarm lab bring-up on all five
nodes of the nested svc-db-postgres drill (compose/swarm/playbook.yml:84) and
cascaded through swarm-zombie to a red Workspace (debian:bookworm) job.
The pre-flight could not have caught it: it only tested that /dev/zfs exists as
a character device, and on those nodes it did while the module was unusable.
Guarding the real operation instead removes the predictor entirely - whatever
makes zfs undeliverable now routes through decline, and REQUIRED=true still
fails hard there. The teardown is required because a loop device is already
attached to the image at that point and decline exits 0.
This does NOT fix why the module is unusable on debian nodes. That is
pre-existing and was already silent: the mariadb drill in the same job declined
zfs on all five nodes and stayed green, while ubuntu applied zfs on ten. The
change makes the two paths consistent; it does not restore the lost coverage.
To make the loss diagnosable rather than invisible, modprobe's output is no
longer discarded into `|| true` but reported and attached to the decline reason,
and verdict() now records the reason in the step summary, folded to one line so
the markdown list item survives a multi-line stderr.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three conflicts, each resolved by combining both sides rather than picking one.
compose.yml passes both env vars through to the container: swarm-nfs adds
INFINITO_SWARM_MGR_IP, this branch adds INFINITO_TOR_SOCKS_PORT, and neither
substitutes for the other.
daemon.json.j2 takes the data-root block from swarm-nfs and keeps HEAD's dns
key. The incoming side writes networks.internet.dns directly; lookup(
'container_dns') already returns that value as its clearnet entry and prepends
the docker0 bridge on a Tor node in compose mode, so the incoming form is a
strict subset and dropping it loses nothing.
The WordPress CA-trust addon spec gains skipUnlessServiceEnabled("sso") from
swarm-nfs alongside HEAD's gotoOnion and resolveTimeout, and asserts on the
Plugin Name the mu-plugin header declares - the mustuse screen renders that,
not the filename - with resolveTimeout applied to the literal the incoming
side left bare.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The routine flowchart marked a terminator-driven remote-2-local start on bkp-01, failing for lack of the ssh trust the drill installs later. Both halves are wrong: the lab deploys the app stack and the backup host with --skip-backup, so MODE_BACKUP is false, svc-bkp-remote-2-local never sets system_service_force_flush_final, and the unit never enters system_service_run_list.
The drill is the only caller that starts the pull. The unit journal of the failed run holds exactly one invocation, at 20:58:30 - one line after 01_ssh_trust.sh installed the key, and that first rsync moved 5.8 GB. A terminator-started pull would have run at ~20:55:4x, before the key existed.
The box also loses its trigger colouring, since nothing fires there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
losetup -f prints the device with a trailing (lost) marker when its node is missing, and both claim sites fed the whole string to mknod. The recovery path that materialises /dev/loop8 and beyond therefore never worked, which is why the pool appeared to stop at the eight preallocated nodes even though max_loop is 0. Strip the marker before the node is built.
Nothing released the data-root loop a node claims, so each distro round leaked four of them: across 140 job logs leg 1 always applies 5 and declines 0, leg 2 applies 3 and declines 2, leg 3 declines 4. The declining nodes fall back to overlayfs, which disables the btrfs snapshot mode the script exists to exercise, and the DR drill then finds no device at all.
Detach right after the mount instead. The mount holds the opener, so the kernel arms autoclear rather than detaching, and the loop plus its sparse image are reclaimed when the node container takes its mount namespace down. Verified in a privileged container: the mount survives the detach, AUTOCLEAR reads 1, the filesystem stays writable, btrfs subvolume create still works, and the loop is released on umount.
Drop the teardown block added earlier: it filtered on (lost), which marks a missing device node rather than a missing backing file, so it selected exactly the devices losetup -d cannot detach, ran unprivileged, and printed success unconditionally. Restore the (deleted) filter in the drill. Require mountpoint up front so a missing tool declines instead of killing the node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role runs ganesha at -N NIV_EVENT, and roughly eighteen init steps between the CLIENT commits and the listener log below that level, so the silence in run 30699990912 marked the edge of visibility rather than the fault. Raise INIT and DISPATCH under MODE_DEBUG: nfs_Init_svc(), which binds 2049, logs under COMPONENT_DISPATCH, so INIT alone would have missed exactly the step in question.
Add a userspace backtrace beside the kernel stack dump, which only ever says futex_do_wait. eu-stack names the function holding the lock; elfutils goes into the lab image rather than the role, since a production NFS server should not carry a debugger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
grep exits 1 when it matches nothing, and under set -euo pipefail the command substitution killed the script before the '[ -n VERDICTS ] || exit 0' line beneath it could express that empty is fine -- which the file header already promises. Three jobs of run 30699990912 were red for this alone: web-app-matrix 2, web-app-penpot 0 and the nested act drill in test-workspace, all with their deploys intact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every node runs docker_dataroot.sh, which attaches a loop for its data root, and nothing ever detached them. Run 30699990912 shows the consequence: all eight kernel loop devices bound to docker-fs.img in two inode groups, i.e. two rounds' worth, leaving none for the DR drill's LUKS image.
Release the orphaned ones after the nodes are gone -- only then is a loop unambiguously ours to reclaim, rather than a concurrent run's. Match (lost) as well as (deleted): a backing file inside a removed container's overlay becomes unfindable rather than merely unlinked, which is why the existing cleanup in 02_luks_device.sh never fired.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The early exit at already-prepared never touched pool properties, so a pool created before 9c0e1efdc would keep acltype=off and setfacl would keep failing on a reused runner. Guard on the pool existing under its current name: e816fc1c9 made that name per-host, so a host carrying an old infinito_ci falls through untouched rather than aborting under set -e.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
compose names the swarm-test volume after its project, so every cluster
left a <cluster>_nfs-export behind that no teardown path touched. At
roughly 5GB each they filled the host disk after ~30 runs, which showed
up as unexplained playwright timeouts once writes started stalling.
Remove it per cluster in teardown, per worktree subnet in lab-reclaim,
and host-wide in swarm-clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ZFS pools live in the kernel's namespace, which the five lab node containers share, and compose/swarm/playbook.yml runs this script on hosts: all. Each created the same 'infinito_ci', so the first would win and the other four would die on 'pool already exists' under set -e. Only resolve.sh dropping zfs for arch/centos/fedora nodes has kept that out of sight; a dispatch limited to debian or ubuntu exposes it.
The mountpoint needs no suffix: mount namespaces are not shared, only the pool name is global.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OpenZFS defaults acltype=off on Linux, so setfacl on anything under the data root returns 'Operation not supported' -- which killed three compose jobs of run 30695207248 the moment the filesystem matrix drew zfs. Set it at pool creation; -O properties are inherited by the docker dataset, so no second edit is needed. xattr=sa is the companion OpenZFS recommends alongside posixacl and it lowers the space cost.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill died with cryptsetup's 'loop device with autoclear flag is required', which names neither the resource nor the reason. Claim a loop for the image and release it again first, creating the /dev node when the container's tmpfs carries none, so cryptsetup finds what it needs and a genuine shortage stops the drill with a message that says so.
The cryptsetup calls themselves are unchanged and keep owning their own autoclear device, so nothing lingers once the mapper closes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill's cryptsetup step died claiming a loop device and the artifact carried nothing to say why: no losetup -a, no /dev listing, and the backup host appears in no other collector loop. Record both views, what the kernel has attached and which nodes the container's /dev carries, so an exhausted pool and a missing device node are told apart at a glance.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The parenthetical claimed the capture names the address and errno of a failed bind. dnsmasq logs that at startup only, and a startup bind failure is fatal, so the half-bound state seen in the artifact cannot originate there. The dynamic bind path is measurably silent on success and untested on failure. The neighbouring markers in this collector carry no description either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The socket table already shows which listening address is missing, but nothing recorded why. A build that cannot resolve dies on the node whose dnsmasq never bound the advertised resolver, and the journal is the only place naming the address and errno. Captured whole rather than tailed, because a failed bind is logged at startup and the neighbouring captures keep only the tail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RUNNER_POOL and NODE_POOL were assertions. With a distro list of just debian
and ubuntu the node pool offered zfs, because the table records that those
distros ship the zfs userland - but the module comes from the runner's kernel,
which carries none. One run in three would then have drawn zfs and either
declined silently, testing ext4 while the summary claimed zfs, or hard-failed on
all five nodes with nothing actually broken.
Ask the kernel instead: /proc/filesystems for what is loaded, modinfo for what
could be. Both answer unprivileged, which resolve.sh needs since it runs without
sudo. The modinfo half carries weight rather than being belt-and-braces - xfs is
absent from /proc/filesystems on this host while modinfo finds the module, so
without it the pool would shrink to whatever happens to be loaded.
The distro table stays above it for the other question: which userland a
container can install. Two layers because they are two things, and only the
kernel one cannot be fixed by a package.
Every exclusion now names its reason on stderr and distinguishes a missing tool
from a missing capability; a filesystem used to drop out of the pool in silence.
For the full five-distro sweep the pool is unchanged: ext4 btrfs, measured
before and after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An opencloud snapshot carried 174 entries, and every line repeated the whole
path - the deepest ran about 150 characters, most of it the same prefix as the
line above. Print the basename at its depth instead, with a glyph per kind:
folder, json, journal or log, plain text.
The separator is swapped for \001 across the sort. Without it a directory and
its children come apart: '/' outranks '.', so a sibling file sorts between the
two and reads as though it lived inside. The swap also makes the order
independent of the locale, which was deciding it before.
The existing cut-short test looked for 'logs/container.log', which no longer
appears, so it now checks the basename; a fourth case pins the structure rather
than the wording - a child is indented deeper than its folder and siblings sit
level.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The distro table pinned centos to ext4, and since the node pool is the
intersection across every distro in a run, the full five-distro sweep always
landed on ext4. btrfs and zfs are what engage the backup snapshot mode, so that
path was never exercised in swarm.
The premise held for a centos system and not for a centos container: Red Hat
dropped btrfs from its own kernel, but a node container runs the host kernel,
and btrfs-progs is merely absent from the base repos - EPEL carries it. Measured
in a centos:stream9 container on this host: mkfs.btrfs, mount, subvolume create,
snapshot -r, and inode 256 at the subvolume root, which is what baudolo checks.
install_packages is the load-bearing part. docker_dataroot.sh ships its own
installer and runs before the ansible package layer, so meta/packages.yml never
reaches it; without the epel fallback there, opening the pool would have let a
centos node be handed btrfs it cannot deliver, decline silently, and leave the
cluster running four btrfs nodes and one ext4 - the very inconsistency the
intersection exists to prevent.
Verified by running the script itself, not extracted parts: it reports
requested=btrfs effective=btrfs on centos.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The data-root task runs on all five swarm nodes at once, and they are
containers on one host kernel sharing its loop devices. losetup -f only names
the next free one; by the time the following losetup attached it, a sibling had
taken it - 'failed to set up loop device: Device or resource busy'. Claim with
--find --show, which picks and attaches in one call, and retry with a short
pause when a sibling wins anyway.
The step runs before anything is pulled, so on some nodes docker is not
installed yet and the closing 'systemctl start docker' died with rc=5, taking
the whole task with it under set -e. The two systemctl stop calls above it had
tolerated that all along. Start only when the unit exists, and report
applied-pending-docker instead of applied so the job summary stops claiming a
data root that no daemon has picked up yet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve conflicts, resolved by intent rather than by side.
Six shared one shape: the incoming side adds the new rescue_index.sh call but
drops this branch's `# nocheck: shell-or-true` markers. The markers carry a
lint contract here, so the incoming structure was taken and the markers put
back — dedicated.sh, guide/one.sh, guide/host_deploy.sh, swarm/routine/00_one.sh
and trigger_units.sh, where the incoming version additionally writes the full
journal to a file instead of only echoing a tail.
web-app-matrix keeps HEAD's _synapse_extra_args accumulator (the superset: CA
mount plus the onion case) but adopts the incoming improvement inside it — the
inlined regex_replace gives way to lookup('subnet_gateway').
handlers/__init__.py drops rescue_diagnostics_dir from the old import group
(the incoming side moved it under .infinito.rescue, keeping both would import
it twice) and keeps tor_socks_port, a HEAD-only handler ORDERED_HANDLERS still
needs.
svc-bkp-volume-2-local takes the incoming side throughout: both branches landed
the same feature, but swarm-nfs evolved it — the knob moved from the global
STORAGE_BACKUP_SNAPSHOT_MODE to services.volume-2-local.snapshot_mode, plus the
subvolume helper, 04_snapshot_subject.yml, meta/packages.yml and baudolo 3.2.2.
Three files that left behind were removed: files/baudolo_snapshot.py and the two
worktree tests at their pre-refactor paths, all byte-identical to their moved
counterparts.
Six follow-on failures the merge produced, fixed at the root:
- STORAGE_BACKUP_SNAPSHOT_MODE lost its last reader with the config-lookup
design and is gone from group_vars.
- utils/github/variant_bundle_limits.py sat next to variant_bundles.py, which
the prefix lint forbids; the pair now lives in utils/github/variant/ as
bundles.py and bundle_limits.py, with the -m entry point, the ruff
per-file-ignore, the design-doc link and the mirroring test path updated.
- The incoming WordPress addon spec navigated with a raw page.goto and carried
three bare timeout literals; both route through gotoOnion and resolveTimeout
now, which is what makes navigation survive onion transport.
- Six unmarked `|| true` got a concrete reason each: docker.socket is absent on
distros that ship docker without it, the loop device is already detached when
the probe failed before attaching, and modprobe zfs may fail because the
/dev/zfs check decides.
make quality green across all four targets. The cold-domains budget test that
had to be waived here passes on its own now — 19d56f07e raises it to 100s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
find and du write what they already collected and only then return nonzero, so
the guards added in ce1c5b5fe were throwing away the evidence the index exists
to report. One vanished entry mid-walk blanked the whole listing: '0 file(s)',
no paths, on a tree that was mostly readable. The rescue tree is indexed while
a failed run is still tearing down, so that is the normal case, not the corner
one.
Stop resetting LISTING, fall back on SIZE only when du produced nothing, and
turn the early exit into a warning so the partial listing still prints.
The existing test could not see this: its stub find emitted nothing, so it only
covered a total failure. The new case stubs a find that prints rows and then
fails, and it fails against ce1c5b5fe.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make lint-action was red repo-wide: a type:choice input may not carry an
unnamed option, and entry-manual.yml used the empty string to mean 'pick one
at random'. The lint gated nothing while it failed on every run.
Call the case 'auto', the spelling the workspace and instructions inputs in
the same file already use, and resolve it in resolve.sh - the one place that
decides random-versus-stated. The pass-through stays verbatim, so the six
downstream declarations keep seeing '' when a caller says nothing and 'auto'
when a human deliberately chose random. Both resolve the same way; the origin
stays distinguishable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rescue artifact is a tarball of whatever the collectors managed to grab, so
reading a failed run meant downloading it to find out which files exist at all.
Run rescue_index.sh right after each collection instead, so the job log names
the tree the artifact contains.
All four sites call it without a fallback, which is what the preceding commit's
exit-status contract is for. host_deploy.sh and one.sh hoist the rescue path
into a variable first - it is now needed twice, once for the collector's
INFINITO_RESCUE_DIAGNOSTICS_DIR and once for the index.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every caller runs this in a cleanup path: three shell teardown handlers and a
GitHub step under bash -e, none of them with a fallback. A nonzero rc therefore
aborts the caller's remaining teardown or adds a second red step on top of the
failure being rescued. Until now that could not happen only because each
internal failure path happened to be caught - a property that dies with the
next edit. A terminal 'trap exit 0' states the guarantee once and makes it
independent of the enumeration.
The failure is not hypothetical: a find that rejects -printf (BusyBox) makes
the old script exit 1, reproduced with a stub.
Collapses three walks of the same tree into one, and counts files with awk
rather than grep -c, which answers 1 on zero matches and takes the assignment
down with it under pipefail. A walk that fails now says so - it used to print
'0 file(s)', which reads exactly like an empty tree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-matrix selection never picked anything but ext4. candidates()
intersected what every distribution in the entry supports, and with the list the
orchestrator passes - arch debian ubuntu fedora centos - that leaves ext4 alone:
centos stream has no btrfs, zfs is debian/ubuntu only. btrfs was reachable only
by stating it in the manual input.
The intersection is right for swarm and wrong for the other two. distros.sh
runs 00_one.sh per distribution and each round builds its own lab, so a swarm
pick must hold in every round's nodes. compose and host carve the data root on
the runner itself, whose OS is fixed; there DISTROS names the application images
and has nothing to do with the filesystem underneath docker.
A fourth argument says which, rather than parsing it out of the label, and
guards with `:?` so a forgotten call site fails loudly. The runner pool omits
zfs: the runner-image build installs neither the zfs userland nor
linux-modules-extra, where the Ubuntu azure kernel keeps zfs.ko, so a drawn zfs
would decline on every run. A stated one still overrides the pool and is marked
required, so `filesystem: zfs` fails loudly instead of quietly falling back.
The test executes the resolver rather than reading it, and pins the relation
that breaks silently: the keys it writes to GITHUB_ENV are the keys the three
workflows read. A typo on either side hands docker_dataroot.sh an empty string,
which reports "no filesystem stated" and exits 0 - the feature switched off
under a green run. Pool contents are policy and are not asserted; the per-distro
expectation is derived from the resolver's own answers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The collector carried a shell-side default for INFINITO_RESCUE_DIAGNOSTICS_DIR,
which the SPOT lint forbids because a second source drifts unnoticed - and this
one already had: the fallback spelled /tmp/rescue-diagnostics, the value of
INFINITO_RESCUE_DIAGNOSTICS_BASE, while the variable it stood in for resolves to
/tmp/infinito-rescue-diagnostics. Had it ever been used, the collector would have
written where no artifact collects.
A `:?` guard replaces it, matching the six the DR drill already carries. The
only caller sets the variable explicitly for both collectors, and _context.sh
sources the env layer regardless, so the guard should never fire.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
