A run-wide draw proved one distribution and said nothing about the other four, while reporting green. Both are matrix-row axes now, assigned by the same deterministic rotation mode and tor already use, so one sweep exercises every distribution and every filesystem.
The two read the row's position like an odometer, the distro as the low digit. Turning both on the position directly covers only n of the n x m pairs whenever the pools are the same length, and no sweep unlocks it.
Both are part of a row's identity: glyphs in the job label, columns in the plan table, shards in the artifact name. Both are selectable through %distro and /filesystem, and a retrigger replays the distro off a failed title. It leaves the filesystem out, because a title states the kind the matrix assigned and a deploy may have fallen back from it.
A row's filesystem is a preference unless a human names it, on the run or in a token. Reading the matrix's own narrowing as a demand would fail rows for every transient condition the applying step is built to tolerate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pick was made once per matrix entry, in a workflow step ahead of the drill,
and every distro of that entry then ran on it. Coverage followed: one entry
exercised one filesystem, and the swarm never reached zfs at all, because
resolve.sh's userland_for() kept it out of any entry containing arch, centos or
fedora - which is every automatic sweep. The draw now happens inside the loop in
distros.sh, once per distro, so a five-distro entry covers zfs with 87% and all
three kinds with 62% probability instead of one kind with certainty.
The workflow input changes meaning with it. It was a single kind, offered as a
choice in the manual entry; it is now a space-separated allow-list. Empty draws
from all three, and naming kinds narrows the draw to them and marks the pick
required, so a host that cannot deliver one of them fails rather than quietly
running on something else. A name that is not a filesystem this project puts a
data root on is rejected outright instead of reaching the applying step.
For the draw to mean anything the images have to carry the userlands, so
scripts/install/zfs.sh becomes scripts/install/filesystem.sh and installs ext4
and btrfs alongside zfs. That is not symmetric work: btrfs-progs does not exist
in the EL family at all, because RHEL dropped btrfs, and a plain
`dnf install btrfs-progs` there fails the whole transaction and takes e2fsprogs
with it. EPEL carries it, so the EL family goes through EPEL from the start -
split by distro rather than by a failed first attempt, since Fedora has no
epel-release to fall back to and routing a transient failure there would turn it
into a fatal one. verify() now demands all four binaries, so a packaging
regression fails the build instead of leaving a hole the drill discovers later.
Measured on all five bases: debian trixie, ubuntu 26.04, arch, fedora 44 and
centos 10 each end with mkfs.ext4, mkfs.btrfs, zfs and zpool present and ldd
fully resolved.
Two defects only became reachable once a single job could ask for different
filesystems in turn:
* docker_dataroot.sh answered "already prepared" and returned for any mount it
found, so from the second distro on, every draw would have been silently
ignored on a host that survives the sweep - the runner in compose and host
mode. It now compares what is mounted against what was asked for and
dismantles the old root when they differ. Everything docker cached there
goes with it; that is what covering more than one filesystem per entry
costs.
* reclaim_stale_pool ran only when the new pick was zfs, so a zfs to btrfs
transition left the previous distro's pool imported on the shared kernel,
pinning a loop device over an image its dead container had already unlinked.
It now runs on every pick, which needed the /dev/zfs handling lifted out of
the zfs branch into zfs_device_ready().
The restore_docker trap is armed before anything can stop docker, rather than
after, so a failure while dismantling a data root cannot leave a host without a
daemon. resolve.sh splits its allow-list into an array before testing it, so a
whitespace-only input is the empty list it looks like rather than a narrowed one
that yields an empty pick marked required. pick.sh clears both keys when it
declines to draw: they are declared in default.env, so a generated .env carries
whatever was exported when it was made, and a caller gating its apply on the
value would otherwise repoint the developer's own docker in the very case the
skip exists to prevent.
Verified by execution rather than by reading: a privileged container ran
docker_dataroot.sh with real loop devices and mkfs through ext4 -> ext4 -> btrfs
-> ext4, confirming the recognition, the replacement in both directions and that
no loop leaks across it; a second harness with a recording zpool stub confirmed
that a btrfs pick does reclaim a stale pool, that no pool means no destroy, and
that a host without /dev/zfs still completes an ext4 pick without touching zpool.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dockerd's zfs graphdriver resolves `zfs` on PATH and opens /dev/zfs before it
accepts a data root (moby daemon/graphdriver/zfs/zfs.go:54,59), so the userland
has to sit in the image, not on the host. Only the apt images carried it, which
is why resolve.sh's userland_for() kept zfs out of every entry containing arch,
centos or fedora - that is every automatic swarm sweep, so the zfs data root and
the zfs branch of the backup launcher were never once exercised. Run
30781723102 shows it: "zfs left out: not every distro of this entry carries the
userland".
scripts/install/zfs.sh joins the three existing install scripts in the image
build and takes a different route per distro, each one run in a real container
of the matching base image before landing here:
debian trixie 2.3.2 zfsutils-linux, contrib enabled first
ubuntu 26.04 2.4.1 zfsutils-linux
arch 2.4.3 the AUR PKGBUILD, which configures --with-config=user
fedora 44 2.4.3 zfsonlinux repo plus a dummy zfs-kmod provider
centos 10 2.2.10 the same, from the epel path
The dummy is what keeps the RPM route usable: the `zfs` package carries an
ungated `Requires: %{name}-kmod`, so a plain `dnf install zfs` resolves through
zfs-dkms and installs 389 extra packages and 1.44 GB of kernel and toolchain to
build a module against a kernel the container never boots. Providing zfs-kmod
from an empty package keeps the same install at 60 packages with no kernel
package at all, and the transaction is checked afterwards so a regression fails
the build instead of bloating the image. The repository is removed again once
the userland is in, because the dummy pins one version and an upgrade would
otherwise re-resolve the kmod at deploy time.
Version drift against the host module is not a concern: zpool create enables a
feature only when the loaded module reports it through /sys/module/zfs/features
.pool (cmd/zpool/zpool_main.c, module/zcommon/zfeature_common.c), and libzfs
compares no versions at all (lib/libzfs/os/linux/libzfs_util_os.c:123-134), so a
newer userland degrades rather than breaking.
With every image carrying it, userland_for() and every_distro_carries() have
nothing left to decide and go. Two defects that only became reachable once zfs
could be drawn are fixed in the same move:
* /dev/zfs is a misc device, so the old lookup in /proc/devices could never
find its major and the node was never created. A node container's /dev is
populated at creation and udevd is masked, so nothing else fills the gap
either: the first bring-up of every job would have declined, blaming a
module that modprobe had just loaded. The minor now comes from /proc/misc.
* Pools live in the kernel the lab nodes share with the runner, while the
teardown only removes containers, and the node hostnames repeat across the
distro sweep. From the second distro on, zpool create hit the previous
iteration's pool. It is destroyed and its loop detached first.
The node image drops its own apt-only zfs branch and instead masks the six boot
units the userland enables by preset, which on a node can only fail: the pool is
created after boot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three places where a data-root verdict was produced and then lost.
report.sh matched `requested=[a-z]+`, which never matches `ext4`. Against the
real deploy log of a run that recorded five ext4 verdicts, the old pattern
yields 0 matches and the summary table was silently empty of them; the fixed
pattern aggregates the same five into one row.
docker_dataroot.sh wrote the reason only to GITHUB_STEP_SUMMARY, which is unset
inside the lab node containers - so on exactly the machines that decline, the
reason went nowhere. It now rides on the machine-readable report line too. This
is what leaves the mariadb drill's decline reason unrecoverable from run
30781723102.
resolve.sh's exclusions went to stderr only, i.e. into the deploy log rather
than the job summary.
The reason is emitted as a list below the table, not as a column: reasons embed
per-node data (the pool name carries the hostname, plus the loop device), so
folding them into the `sort | uniq -c` key would split one five-node row into
five one-node rows exactly in the failure case the reason exists to explain.
`reason=[^']*` keeps ansible's closing quote out of the value, and `|` is
replaced so a reason cannot break the markdown table.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RUNNER_POOL and NODE_POOL were assertions. With a distro list of just debian
and ubuntu the node pool offered zfs, because the table records that those
distros ship the zfs userland - but the module comes from the runner's kernel,
which carries none. One run in three would then have drawn zfs and either
declined silently, testing ext4 while the summary claimed zfs, or hard-failed on
all five nodes with nothing actually broken.
Ask the kernel instead: /proc/filesystems for what is loaded, modinfo for what
could be. Both answer unprivileged, which resolve.sh needs since it runs without
sudo. The modinfo half carries weight rather than being belt-and-braces - xfs is
absent from /proc/filesystems on this host while modinfo finds the module, so
without it the pool would shrink to whatever happens to be loaded.
The distro table stays above it for the other question: which userland a
container can install. Two layers because they are two things, and only the
kernel one cannot be fixed by a package.
Every exclusion now names its reason on stderr and distinguishes a missing tool
from a missing capability; a filesystem used to drop out of the pool in silence.
For the full five-distro sweep the pool is unchanged: ext4 btrfs, measured
before and after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The distro table pinned centos to ext4, and since the node pool is the
intersection across every distro in a run, the full five-distro sweep always
landed on ext4. btrfs and zfs are what engage the backup snapshot mode, so that
path was never exercised in swarm.
The premise held for a centos system and not for a centos container: Red Hat
dropped btrfs from its own kernel, but a node container runs the host kernel,
and btrfs-progs is merely absent from the base repos - EPEL carries it. Measured
in a centos:stream9 container on this host: mkfs.btrfs, mount, subvolume create,
snapshot -r, and inode 256 at the subvolume root, which is what baudolo checks.
install_packages is the load-bearing part. docker_dataroot.sh ships its own
installer and runs before the ansible package layer, so meta/packages.yml never
reaches it; without the epel fallback there, opening the pool would have let a
centos node be handed btrfs it cannot deliver, decline silently, and leave the
cluster running four btrfs nodes and one ext4 - the very inconsistency the
intersection exists to prevent.
Verified by running the script itself, not extracted parts: it reports
requested=btrfs effective=btrfs on centos.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make lint-action was red repo-wide: a type:choice input may not carry an
unnamed option, and entry-manual.yml used the empty string to mean 'pick one
at random'. The lint gated nothing while it failed on every run.
Call the case 'auto', the spelling the workspace and instructions inputs in
the same file already use, and resolve it in resolve.sh - the one place that
decides random-versus-stated. The pass-through stays verbatim, so the six
downstream declarations keep seeing '' when a caller says nothing and 'auto'
when a human deliberately chose random. Both resolve the same way; the origin
stays distinguishable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-matrix selection never picked anything but ext4. candidates()
intersected what every distribution in the entry supports, and with the list the
orchestrator passes - arch debian ubuntu fedora centos - that leaves ext4 alone:
centos stream has no btrfs, zfs is debian/ubuntu only. btrfs was reachable only
by stating it in the manual input.
The intersection is right for swarm and wrong for the other two. distros.sh
runs 00_one.sh per distribution and each round builds its own lab, so a swarm
pick must hold in every round's nodes. compose and host carve the data root on
the runner itself, whose OS is fixed; there DISTROS names the application images
and has nothing to do with the filesystem underneath docker.
A fourth argument says which, rather than parsing it out of the label, and
guards with `:?` so a forgotten call site fails loudly. The runner pool omits
zfs: the runner-image build installs neither the zfs userland nor
linux-modules-extra, where the Ubuntu azure kernel keeps zfs.ko, so a drawn zfs
would decline on every run. A stated one still overrides the pool and is marked
required, so `filesystem: zfs` fails loudly instead of quietly falling back.
The test executes the resolver rather than reading it, and pins the relation
that breaks silently: the keys it writes to GITHUB_ENV are the keys the three
workflows read. A typo on either side hands docker_dataroot.sh an empty string,
which reports "no filesystem stated" and exits 0 - the feature switched off
under a green run. Pool contents are policy and are not asserted; the per-distro
expectation is derived from the resolver's own answers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The backup snapshot mode only engages on btrfs or zfs, so on CI - where the
data root is whatever the runner or the node image happens to provide - it
was never exercised at all. Nothing proved the snapshot path beyond unit
tests and a synthetic probe.
`entry-manual` gains a `filesystem` choice that reaches both deploy modes.
Empty means each matrix entry draws its own, so coverage spreads over runs
instead of pinning one path forever; the pick and its origin go to the step
summary, because reproducing a red run means re-stating it verbatim. The
draw is restricted to what every distribution of that entry can deliver -
centos stream has no btrfs at all, and zfs is a third-party repository
outside debian and ubuntu - so a run never dies on a packaging gap. A
stated filesystem skips that restriction: it is an instruction, and a host
that cannot deliver it fails loudly rather than deploying on something
else.
docker_dataroot.sh formats a sparse loop image, mounts it at /mnt/docker-fs
and carves the data root inside it, then points the daemon there through
daemon.json - merged, not overwritten, so the registry-cache entry
survives. It runs on the runner for compose and inside each node for swarm,
before anything is pulled, and installs the filesystem tooling it needs.
Two container facts it has to work around: a privileged container's /dev is
a fresh tmpfs, so neither /dev/loopN nor /dev/zfs exists there and both are
mknod'ed from /proc/devices before use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5