ubuntu-latest is rolling over to ubuntu-26.04, whose sudo is sudo-rs. It
ignores `-E` ("preserving the entire environment is not supported") and
hands the child only the explicit `env` assignments, so the swarm matrix
died at import with `KeyError: 'INFINITO_SRC_DIR'` before any inventory
existed (PR #1009, Nextcloud#0). Jobs that landed on ubuntu-24.04 kept
passing.
Every `sudo -E` now passes the caller's exported names through
`--preserve-env="$(compgen -e | paste -sd, -)"`, which both classic sudo
and sudo-rs honour. A lint rejects `sudo -E` and bare `--preserve-env`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
In job 91983553540 all five lab nodes derived the same pool name,
infinito_ci_unknown, and the run died on `cannot destroy
'infinito_ci_unknown': pool is busy`. The name came from `hostname`, which
returns non-zero inside the node containers, so the `|| echo unknown` fallback
answered for every one of them. One node won the race and created the pool; the
others found it, took it for a leftover, and tried to tear down a pool a
sibling was actively using.
234c1158c already fixed this collision class three days ago by deriving the
name from `hostname` instead of a constant, and its message named the zfs
exclusion in resolve.sh as the only thing keeping the collision out of sight.
That fix was never exercised, because the exclusion stayed up until ed1121053
removed it - and it does not work, because the call it rests on fails in the
containers it was written for. Guessing the identity is the defect; the
playbook has known it all along as inventory_hostname, and ci/one.sh knows its
own deploy mode. Both now state it, as a required fourth argument with no
default, the way SIZE is already stated. verdict() stops guessing too: every
row it wrote into the job summary said `unknown`, which is why the summary was
never worth reading.
The second half is independent of the naming. reclaim_stale_pool ran
`zpool destroy -f` as a bare command under `set -euo pipefail`, the only
failure path in this script that bypassed decline(). A pool that refuses to be
destroyed is by definition not a leftover - something holds it - so that case
now declines through the usual contract: a drawn pick falls back and reports,
a stated one fails loudly as required-but-unavailable. A reclaim can no longer
be more destructive than the state it exists to clean up.
Verified over the real invocation path rather than a model of it: an
ansible-playbook run using the same `script:` task and docker connection as
compose/swarm/playbook.yml, against three containers in one play, has each node
address its own pool - infinito_ci_race-mgr-01, -wrk-01, -nfs-01 - with
failed=0 throughout. A stubbed-pool harness covers the paths that need a kernel
module this machine does not have: a sibling holding the pool declines with rc
0 instead of ending the run, the same case with a stated pick exits 1 through
the contract, a genuine leftover is still destroyed, and a missing identity is
a hard error. The earlier replacement harness (ext4 recognised, btrfs
replacing it, no loop leaked) still passes against the stricter signature.
Pool names now carry dashes, which zfs_namecheck.c accepts; at 42 characters
they sit far inside the 240 limit and match no reserved prefix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pick was made once per matrix entry, in a workflow step ahead of the drill,
and every distro of that entry then ran on it. Coverage followed: one entry
exercised one filesystem, and the swarm never reached zfs at all, because
resolve.sh's userland_for() kept it out of any entry containing arch, centos or
fedora - which is every automatic sweep. The draw now happens inside the loop in
distros.sh, once per distro, so a five-distro entry covers zfs with 87% and all
three kinds with 62% probability instead of one kind with certainty.
The workflow input changes meaning with it. It was a single kind, offered as a
choice in the manual entry; it is now a space-separated allow-list. Empty draws
from all three, and naming kinds narrows the draw to them and marks the pick
required, so a host that cannot deliver one of them fails rather than quietly
running on something else. A name that is not a filesystem this project puts a
data root on is rejected outright instead of reaching the applying step.
For the draw to mean anything the images have to carry the userlands, so
scripts/install/zfs.sh becomes scripts/install/filesystem.sh and installs ext4
and btrfs alongside zfs. That is not symmetric work: btrfs-progs does not exist
in the EL family at all, because RHEL dropped btrfs, and a plain
`dnf install btrfs-progs` there fails the whole transaction and takes e2fsprogs
with it. EPEL carries it, so the EL family goes through EPEL from the start -
split by distro rather than by a failed first attempt, since Fedora has no
epel-release to fall back to and routing a transient failure there would turn it
into a fatal one. verify() now demands all four binaries, so a packaging
regression fails the build instead of leaving a hole the drill discovers later.
Measured on all five bases: debian trixie, ubuntu 26.04, arch, fedora 44 and
centos 10 each end with mkfs.ext4, mkfs.btrfs, zfs and zpool present and ldd
fully resolved.
Two defects only became reachable once a single job could ask for different
filesystems in turn:
* docker_dataroot.sh answered "already prepared" and returned for any mount it
found, so from the second distro on, every draw would have been silently
ignored on a host that survives the sweep - the runner in compose and host
mode. It now compares what is mounted against what was asked for and
dismantles the old root when they differ. Everything docker cached there
goes with it; that is what covering more than one filesystem per entry
costs.
* reclaim_stale_pool ran only when the new pick was zfs, so a zfs to btrfs
transition left the previous distro's pool imported on the shared kernel,
pinning a loop device over an image its dead container had already unlinked.
It now runs on every pick, which needed the /dev/zfs handling lifted out of
the zfs branch into zfs_device_ready().
The restore_docker trap is armed before anything can stop docker, rather than
after, so a failure while dismantling a data root cannot leave a host without a
daemon. resolve.sh splits its allow-list into an array before testing it, so a
whitespace-only input is the empty list it looks like rather than a narrowed one
that yields an empty pick marked required. pick.sh clears both keys when it
declines to draw: they are declared in default.env, so a generated .env carries
whatever was exported when it was made, and a caller gating its apply on the
value would otherwise repoint the developer's own docker in the very case the
skip exists to prevent.
Verified by execution rather than by reading: a privileged container ran
docker_dataroot.sh with real loop devices and mkfs through ext4 -> ext4 -> btrfs
-> ext4, confirming the recognition, the replacement in both directions and that
no loop leaks across it; a second harness with a recording zpool stub confirmed
that a btrfs pick does reclaim a stale pool, that no pool means no destroy, and
that a host without /dev/zfs still completes an ext4 pick without touching zpool.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The serial distro loop (shuffle, time budget, fast-fail skip, exit on the first failing distro) lived inline in scripts/tests/deploy/ci/all.sh and was therefore reachable only from the compose and host deploys. Swarm and guide have to iterate distros with the same semantics, so the loop moves into scripts/tests/deploy/distros.sh and takes the per-distro command as "$@".
ci/all.sh keeps the app-level preamble and execs the loop with the new ci/one.sh, which resolves the CI image and dispatches to the per-app override or dedicated.sh.
Image resolution leaves the loop as well: scripts/meta/resolve/image/ci.sh renders the reference through cli.meta.ci.image_ref instead of respelling ghcr.io/... locally, which meta/distros.yml declares as the single source.
The two CI time knobs are declared in default.env instead of being repeated as literals, so no caller has to invent a budget.
Three paths are dropped as dead: the empty budget that disabled the limit, the failed counter that could only ever print 0, and the branch for an unset deadline.
Both new scripts are pinned by contract tests. Those tests clear BASH_ENV because the test container exports it as the env loader, which would otherwise re-inject every stripped key and make the assertions pass for the wrong reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5.5