Without the bindings the ansible apt module installs them itself mid-play, and the dist-upgrade that follows deadlocks: the module spins in user code while apt-get blocks writing to a full stderr pipe. It ran that way on all five nodes for three hours.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dockerd's zfs graphdriver resolves `zfs` on PATH and opens /dev/zfs before it
accepts a data root (moby daemon/graphdriver/zfs/zfs.go:54,59), so the userland
has to sit in the image, not on the host. Only the apt images carried it, which
is why resolve.sh's userland_for() kept zfs out of every entry containing arch,
centos or fedora - that is every automatic swarm sweep, so the zfs data root and
the zfs branch of the backup launcher were never once exercised. Run
30781723102 shows it: "zfs left out: not every distro of this entry carries the
userland".
scripts/install/zfs.sh joins the three existing install scripts in the image
build and takes a different route per distro, each one run in a real container
of the matching base image before landing here:
debian trixie 2.3.2 zfsutils-linux, contrib enabled first
ubuntu 26.04 2.4.1 zfsutils-linux
arch 2.4.3 the AUR PKGBUILD, which configures --with-config=user
fedora 44 2.4.3 zfsonlinux repo plus a dummy zfs-kmod provider
centos 10 2.2.10 the same, from the epel path
The dummy is what keeps the RPM route usable: the `zfs` package carries an
ungated `Requires: %{name}-kmod`, so a plain `dnf install zfs` resolves through
zfs-dkms and installs 389 extra packages and 1.44 GB of kernel and toolchain to
build a module against a kernel the container never boots. Providing zfs-kmod
from an empty package keeps the same install at 60 packages with no kernel
package at all, and the transaction is checked afterwards so a regression fails
the build instead of bloating the image. The repository is removed again once
the userland is in, because the dummy pins one version and an upgrade would
otherwise re-resolve the kmod at deploy time.
Version drift against the host module is not a concern: zpool create enables a
feature only when the loaded module reports it through /sys/module/zfs/features
.pool (cmd/zpool/zpool_main.c, module/zcommon/zfeature_common.c), and libzfs
compares no versions at all (lib/libzfs/os/linux/libzfs_util_os.c:123-134), so a
newer userland degrades rather than breaking.
With every image carrying it, userland_for() and every_distro_carries() have
nothing left to decide and go. Two defects that only became reachable once zfs
could be drawn are fixed in the same move:
* /dev/zfs is a misc device, so the old lookup in /proc/devices could never
find its major and the node was never created. A node container's /dev is
populated at creation and udevd is masked, so nothing else fills the gap
either: the first bring-up of every job would have declined, blaming a
module that modprobe had just loaded. The minor now comes from /proc/misc.
* Pools live in the kernel the lab nodes share with the runner, while the
teardown only removes containers, and the node hostnames repeat across the
distro sweep. From the second distro on, zpool create hit the previous
iteration's pool. It is destroyed and its loop detached first.
The node image drops its own apt-only zfs branch and instead masks the six boot
units the userland enables by preset, which on a node can only fail: the pool is
created after boot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docker_dataroot.sh installs zfsutils-linux at runtime, so the zfs pick depends
on the package cache being up. In run 30781723102 that decided the two swarm
drills differently: the cache stack was removed at 06:39:49 and rebuilt at
07:18:25, so the mariadb drill had no zfsutils candidate at all and declined,
while the postgres drill got one and reached zpool create. Installing it at
build time removes that dependency.
Debian keeps zfsutils-linux in contrib, which the base image does not enable, so
the branch adds a contrib source first - only for ID=debian, since Ubuntu ships
it in its default components. A failed update after adding the source removes
the source again rather than leaving a broken repo line behind.
The install is deliberately separate from the mandatory packages and
non-fatal. Node images for centos, fedora and arch have no zfs userland either
and resolve.sh's userland_for() already accounts for that, so a missing zfs
userland is a supported state - whereas folding it into the mandatory install
would let a packaging error break the node image for all five distros. A failure
now prints to stderr and the run declines the pick visibly.
Verified against the real bookworm contrib index (deb.debian.org, HTTP 200):
zfsutils-linux 2.1.11-1+deb12u1 exists there, all four libzfs libraries are
Depends rather than Recommends so --no-install-recommends keeps them, and the
only Recommends are zfs-modules|zfs-dkms and zfs-zed, which we do not want - the
module comes from the host via /lib/modules. Control flow exercised with a
stubbed apt-get across six combinations of distro, package availability and
update failure; every one leaves the build at rc 0.
Not covered here: whether a 2.1 userland can drive the runner's newer host
module. zfs-version.txt and modules.txt from 264556a31 answer that next run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
18abce480 added kmod so the data-root setup can run its one explicit `modprobe
zfs`. It also handed a working loader to the four modprobe@ instances systemd
pulls at every node boot - configfs, drm, efi_pstore and fuse - which until now
failed on the missing binary, as every node's journal in
rescue-diagnostics-workspace-debian-bookworm shows. They would have started
loading host modules from a privileged container, on every node of every swarm
job in all five distros.
Masking the template keeps them exactly as dead as they are today and leaves the
loader available to the caller that needs it, so the behaviour change of
18abce480 is confined to the one site it was meant for. It sits with the
existing masks for systemd-resolved and systemd-udevd.
The image was not built; that the mask covers every instance of the template is
read from systemd's behaviour, not measured. The next run shows it: the units
report masked instead of failed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The node image ships no module loader, so `modprobe zfs` in
scripts/tests/deploy/utils/filesystem/docker_dataroot.sh was a no-op whose error
was additionally discarded by `2>/dev/null || true`. Every node's own systemd
says so at boot, in rescue-diagnostics-workspace-debian-bookworm of run
30781723102:
(modprobe)[27]: modprobe@configfs.service: Unable to locate executable
'/sbin/modprobe': No such file or directory
The module itself is reachable: the nodes run privileged with /lib/modules
bind-mounted read-only, and their df confirms it. What was missing is the
loader, not the module. zfs-load-module.service is installed and enabled in the
image and sits inactive for the same reason.
This does not by itself make the debian lane apply zfs. `The ZFS modules are not
loaded.` in job 91588086713 came from libzfs at `zpool create`, not from the
suppressed modprobe - the kernel ring buffer shows five 30 GiB loop attachments
at 07:37:32, i.e. the script was past the /dev/zfs gate. And a Debian bookworm
node only has a zfsutils-linux candidate while the package cache stack is up:
the cache was removed at 06:39:49 and rebuilt at 07:18:25, which is why the
mariadb drill declined at the userland check while the postgres drill reached
zpool create. That cache dependency is untouched here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role runs ganesha at -N NIV_EVENT, and roughly eighteen init steps between the CLIENT commits and the listener log below that level, so the silence in run 30699990912 marked the edge of visibility rather than the fault. Raise INIT and DISPATCH under MODE_DEBUG: nfs_Init_svc(), which binds 2049, logs under COMPONENT_DISPATCH, so INIT alone would have missed exactly the step in question.
Add a userspace backtrace beside the kernel stack dump, which only ever says futex_do_wait. eu-stack names the function holding the lock; elfutils goes into the lab image rather than the role, since a production NFS server should not carry a debugger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The LUKS step of the DR drill died on `The file /dev/mapper/usbdrill does
not exist and no size was specified.` -- from mkfs, not from cryptsetup,
because luksOpen had exited 0. The node did have a udev, and it had
created the mapper entry: as a symlink to ../dm-1. Modern udev no longer
mknods, it only manages symlinks, and the /dev/dm-N target it points at is
a devtmpfs node that a container's tmpfs /dev never receives. Without a
udev libdevmapper creates the node itself, which is why the debian nodes
passed -- their image ships no udevd at all, while arch cannot install
systemd without one.
Masking udevd in the node image removes the whole class rather than one
call site. DM_DISABLE_UDEV rides on the node containers instead of the
script because the drill opens the volume a second time through the
recover CLI in a separate docker exec, which would not inherit an export.
The block-device check then fails loudly with both prerequisites named,
and closes the mapping first so an abort cannot leak it onto the runner --
device-mapper is not namespaced and every distro round shares the VM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The backup DR drill could not pull from the manager on centos and fedora:
the backup user's ssh died with "Connection closed", exit 255. Pubkey auth
had actually succeeded -- sshd only calls the PAM account phase once
authenticated -- and that phase failed because the RHEL parents ship
/etc/shadow mode 0000. Their setuid-root unix_chkpwd then needs
CAP_DAC_READ_SEARCH, which the host's unix-chkpwd AppArmor profile denies
right through the container: profiles attach by executable path, so the
lab nodes hit it despite already running privileged with
apparmor=unconfined.
chmod u+rw adds only the owner bit, which is what arch ships and what a
setuid helper needs. A hard 0600 would strip the group bit debian and
ubuntu rely on for their setgid-shadow helper and break the distros that
work today.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The node image layers on a pinned pkgmgr-arch parent that can be weeks old, and `pacman -Sy <pkg>` against it is the partial upgrade arch upstream documents as unsupported. Two failure modes: the parent's archlinux-keyring predates any packager key added since, so a package signed by one aborts the RUN with unknown trust; and -Sy without u installs fresh binaries next to the image's older libraries, which pacman cannot catch because it matches SONAMEs, not symbol versions (cryptsetup declares libcrypto.so=3-64 and installs happily onto an older openssl).
Refreshing archlinux-keyring first and then running a full -Syu --needed fixes both. The cache wipe matches what the apt and dnf branches already do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill ran in every variant job because its gate checked the
job-local plan index, which is 0 in each of the single-variant CI jobs;
gate it on the absolute round index so the destructive drill and the
backup host deploy run once per app, in the variant-0 job. Guard the
matrix steps with a disk watchdog that aborts visibly below 6G free
instead of letting the Actions Worker die silently on ENOSPC with no
artifacts. Cap node journald (256M) and DiD container logs (10m x 2),
which were unbounded across five nodes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Node image now builds FROM the distro infinito image (python, ansible, CLI baked in) instead of installing a per-node .deb, so there is no per-node bootstrap or timeout; the swarm CI job depends on the distribution build. Move compose/swarm.yml to compose/swarm/compose.yml and relocate the swarm matrix orchestrator from cli.administration.deploy.swarm to utils.tests.swarm (test code out of the shipped CLI).
Back the NFS export root with a disk-backed named volume instead of a RUNNER_TEMP bind: local-act RUNNER_TEMP resolves onto tmpfs and ganesha FSAL_VFS cannot export a tmpfs, so it loaded zero exports and the controller mount got ENOENT. Add optional compose/swarm/cache.override.yml wiring swarm nodes to the package-cache frontend when present, leaving CI without a cache unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The node image baked one machine-id into every container, so the pull
target directory backup/<machine-id>/ was shared by all providers and
the per-provider rsyncs raced each other with --delete (DR drill step
4/9, run 29186471592 job 86634020663). Emptying /etc/machine-id makes
systemd generate a distinct id on first boot; real hosts always had
their own.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the per-node docker-run bring-up with a single node image
(compose/swarm/Dockerfile bakes python3/dnsmasq/openssh/rsync/cryptsetup)
declared in compose/swarm.yml, one compose up, and one ansible play
(compose/swarm/playbook.yml over ansible_connection: docker) that waits for
systemd, captures IPs, wires lab DNS and installs the .deb per node. Reshape
the harness into topology/ (naming SPOT), routine/ (01_bootstrap + the
sequenced 02-08 steps) and utils/ (clean/, collect/, shared helpers); the
workflow, matrix.py and Makefile follow the new paths. swarm init/join was
already ansible-owned (svc-swarm-node), corrected in the README diagram.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5