feat(images): carry the zfs userland on all distros

dockerd's zfs graphdriver resolves `zfs` on PATH and opens /dev/zfs before it
accepts a data root (moby daemon/graphdriver/zfs/zfs.go:54,59), so the userland
has to sit in the image, not on the host. Only the apt images carried it, which
is why resolve.sh's userland_for() kept zfs out of every entry containing arch,
centos or fedora - that is every automatic swarm sweep, so the zfs data root and
the zfs branch of the backup launcher were never once exercised. Run
30781723102 shows it: "zfs left out: not every distro of this entry carries the
userland".

scripts/install/zfs.sh joins the three existing install scripts in the image
build and takes a different route per distro, each one run in a real container
of the matching base image before landing here:

  debian trixie 2.3.2   zfsutils-linux, contrib enabled first
  ubuntu 26.04 2.4.1    zfsutils-linux
  arch 2.4.3            the AUR PKGBUILD, which configures --with-config=user
  fedora 44 2.4.3       zfsonlinux repo plus a dummy zfs-kmod provider
  centos 10 2.2.10      the same, from the epel path

The dummy is what keeps the RPM route usable: the `zfs` package carries an
ungated `Requires: %{name}-kmod`, so a plain `dnf install zfs` resolves through
zfs-dkms and installs 389 extra packages and 1.44 GB of kernel and toolchain to
build a module against a kernel the container never boots. Providing zfs-kmod
from an empty package keeps the same install at 60 packages with no kernel
package at all, and the transaction is checked afterwards so a regression fails
the build instead of bloating the image. The repository is removed again once
the userland is in, because the dummy pins one version and an upgrade would
otherwise re-resolve the kmod at deploy time.

Version drift against the host module is not a concern: zpool create enables a
feature only when the loaded module reports it through /sys/module/zfs/features
.pool (cmd/zpool/zpool_main.c, module/zcommon/zfeature_common.c), and libzfs
compares no versions at all (lib/libzfs/os/linux/libzfs_util_os.c:123-134), so a
newer userland degrades rather than breaking.

With every image carrying it, userland_for() and every_distro_carries() have
nothing left to decide and go. Two defects that only became reachable once zfs
could be drawn are fixed in the same move:

  * /dev/zfs is a misc device, so the old lookup in /proc/devices could never
    find its major and the node was never created. A node container's /dev is
    populated at creation and udevd is masked, so nothing else fills the gap
    either: the first bring-up of every job would have declined, blaming a
    module that modprobe had just loaded. The minor now comes from /proc/misc.
  * Pools live in the kernel the lab nodes share with the runner, while the
    teardown only removes containers, and the node hostnames repeat across the
    distro sweep. From the second distro on, zpool create hit the previous
    iteration's pool. It is destroyed and its loop detached first.

The node image drops its own apt-only zfs branch and instead masks the six boot
units the userland enables by preset, which on a node can only fail: the pool is
created after boot.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kevinveenbirkenbachandClaude Opus 5 committed 2026-08-04 13:27:34 +02:00
1 parent 65d2aa028e
commit ed1121053b
10 files changed
+285 -50

No files matched your search

+3 -2
View File
@@ -30,10 +30,11 @@ COPY . ${INFINITO_SRC_DIR}
# hadolint ignore=DL3008,DL3033,DL3041,SC1090
RUN set -euo pipefail; \
source <(grep -hE '^INFINITO_(PYTHON|DOCKER_CLI|PACKAGE)_INSTALL_SCRIPT=' "${INFINITO_SRC_DIR}/default.env"); \
source <(grep -hE '^INFINITO_(PYTHON|DOCKER_CLI|PACKAGE|ZFS)_INSTALL_SCRIPT=' "${INFINITO_SRC_DIR}/default.env"); \
/bin/bash "${INFINITO_SRC_DIR}/${INFINITO_PYTHON_INSTALL_SCRIPT:?}"; \
/bin/bash "${INFINITO_SRC_DIR}/${INFINITO_DOCKER_CLI_INSTALL_SCRIPT:?}"; \
/bin/bash "${INFINITO_SRC_DIR}/${INFINITO_PACKAGE_INSTALL_SCRIPT:?}"
/bin/bash "${INFINITO_SRC_DIR}/${INFINITO_PACKAGE_INSTALL_SCRIPT:?}"; \
/bin/bash "${INFINITO_SRC_DIR}/${INFINITO_ZFS_INSTALL_SCRIPT:?}"
RUN set -euo pipefail; \
systemctl mask systemd-firstboot.service first-boot-complete.target || true; \
+7 -7
View File
@@ -17,6 +17,12 @@
# AppArmor profile then denies the setuid helper CAP_DAC_READ_SEARCH and every
# PAM account lookup fails; a hard 0600 instead of u+rw breaks debian's setgid
# helper.
#
# The zfs userland arrives with the parent image and enables six units by
# preset. All of them run before scripts/tests/deploy/utils/filesystem/
# docker_dataroot.sh creates the pool, so on a node they can only fail and
# leave systemd degraded; they are masked here rather than in the parent, which
# a real host also boots.
ARG INFINITO_IMAGE
FROM ${INFINITO_IMAGE}
@@ -24,13 +30,6 @@ RUN set -eux; \
if command -v apt-get >/dev/null 2>&1; then \
apt-get update -qq; \
apt-get install -y -qq --no-install-recommends dnsmasq openssh-server cryptsetup e2fsprogs elfutils kmod; \
. /etc/os-release; \
if [ "${ID:-}" = debian ] && [ -n "${VERSION_CODENAME:-}" ]; then \
echo "deb http://deb.debian.org/debian ${VERSION_CODENAME} contrib" >/etc/apt/sources.list.d/contrib.list; \
apt-get update -qq || rm -f /etc/apt/sources.list.d/contrib.list; \
fi; \
apt-get install -y -qq --no-install-recommends zfsutils-linux || \
echo "zfsutils-linux unavailable; a zfs data-root pick will decline on this image" >&2; \
rm -rf /var/lib/apt/lists/*; \
elif command -v dnf >/dev/null 2>&1; then \
dnf install -y dnsmasq openssh-server cryptsetup e2fsprogs elfutils kmod; \
@@ -45,6 +44,7 @@ RUN set -eux; \
systemctl mask systemd-resolved; \
systemctl mask systemd-udevd systemd-udevd-kernel.socket systemd-udevd-control.socket systemd-udevd-varlink.socket systemd-udev-trigger; \
systemctl mask modprobe@.service; \
systemctl mask zfs-import-cache zfs-import-scan zfs-mount zfs-share zfs-volume-wait zfs-zed; \
systemctl enable ssh 2>/dev/null || systemctl enable sshd; \
chmod u+rw /etc/shadow; \
if [ -e /etc/gshadow ]; then chmod u+rw /etc/gshadow; fi; \
+3
View File
@@ -324,3 +324,6 @@ INFINITO_WHITELIST=
# Master toggle for the worker SPOT chain (INFINITO_WORKER_CPU / _FETCH); falsy values: 0/false/no/off.
INFINITO_WORKER_ENABLED=true
# Repo-relative path of the ZFS userland install script.
INFINITO_ZFS_INSTALL_SCRIPT=scripts/install/zfs.sh
+1
View File
@@ -6,6 +6,7 @@ This directory contains scripts that install and bootstrap host-side tooling use
- 🐧 Install ad-hoc APT packages for CI and workflow jobs via `apt.sh`
- 📦 Build and install distro package metadata via `package.sh` (Arch/Debian/Ubuntu/Fedora)
- 🏖️ Install OS-level Claude Code sandbox dependencies (`bubblewrap`, `socat`) via `sandbox.sh`, covering Debian, Ubuntu, Fedora, CentOS/RHEL/Rocky/Alma, and Arch; also exposed as `make install-agent`
- 🧊 Install the OpenZFS userland via `zfs.sh` (apt on Debian/Ubuntu, the AUR PKGBUILD on Arch, the zfsonlinux repository on Fedora/CentOS); the kernel module comes from the host
- 🐍 Bootstrap virtual environments and editable project dependencies
- 🛠️ Keep host preparation separate from lint, build, and test execution
+213
View File
@@ -0,0 +1,213 @@
#!/usr/bin/env bash
# Bake the OpenZFS userland into the distro image. The kernel module always
# comes from the host - dockerd's zfs graphdriver only needs the `zfs` binary on
# PATH and an openable /dev/zfs, so a pool the host prepared is unusable without
# these packages inside the container.
#
# Three trip-wires this script exists to route around:
#
# * The OpenZFS `zfs` RPM carries an ungated `Requires: %{name}-kmod`, so a
# plain `dnf install zfs` resolves through zfs-dkms and drags in
# kernel-devel, kernel-debug-core and a DKMS build against a kernel the
# container never boots: measured at 389 extra packages and 1.44 GB. A
# dummy package that only *provides* zfs-kmod keeps the same install at 60
# packages with no kernel package at all. Its version must equal the exact
# %{version} dnf resolves for `zfs`; a mismatch silently falls back to DKMS.
# Because that version is pinned, the repository is removed again once the
# userland is in: leaving it enabled means the next `dnf upgrade` finds a
# newer zfs whose kmod dependency the dummy no longer satisfies, and the
# whole DKMS pull returns at deploy time rather than at build time.
# * Debian keeps zfsutils-linux in contrib, which the base image does not
# enable. Without the extra source apt reports no installation candidate.
# * Arch has no zfs userland in core/extra. archzfs publishes a prebuilt
# zfs-utils, but only on a rolling release tag whose asset filename carries
# the version, so any pinned URL 404s on the next upstream bump. Building
# the AUR PKGBUILD (which configures --with-config=user, userland only) is
# the reproducible route and costs about six minutes. That build runs with
# --skippgpcheck rather than --skipinteg: the release tarball's sha256 and
# b2 sums stay verified, and only the detached OpenZFS signature is skipped,
# because checking it means carrying the release key in the builder's
# keyring while the same PKGBUILD already supplies both the sums and the
# list of accepted keys, so the signature adds no trust the sums do not.
set -euo pipefail
RETRIES=3
AUR_BUILDER=pkgbuild
ZFS_RELEASE_FEDORA="3-1 3-0 2-8"
ZFS_RELEASE_EL="3-0 2-8 2-3"
log() { echo "[zfs] $*"; }
# Param: $@ command to run, retried on failure with a widening backoff
retry() {
local attempt=1
until "$@"; do
if [ "${attempt}" -ge "${RETRIES}" ]; then
log "ERROR: '$*' failed after ${attempt} attempts"
return 1
fi
log "attempt ${attempt} of '$1' failed; retrying"
sleep $((attempt * 5))
attempt=$((attempt + 1))
done
}
install_apt() {
export DEBIAN_FRONTEND=noninteractive
local apt_opts=(-o Acquire::Retries="${RETRIES}" -qq)
if [ "${ID:-}" = debian ]; then
log "enabling contrib for ${VERSION_CODENAME:?}, which is where zfsutils-linux lives"
echo "deb http://deb.debian.org/debian ${VERSION_CODENAME} contrib" \
>/etc/apt/sources.list.d/contrib.list
fi
retry apt-get "${apt_opts[@]}" update
retry apt-get "${apt_opts[@]}" install -y --no-install-recommends zfsutils-linux
rm -rf /var/lib/apt/lists/*
}
install_pacman() {
retry pacman -Sy --noconfirm --needed base-devel git
if ! id "${AUR_BUILDER}" >/dev/null 2>&1; then
useradd -m -s /bin/bash "${AUR_BUILDER}"
fi
install -d -m 0755 /etc/sudoers.d
printf '%s ALL=(ALL) NOPASSWD: /usr/bin/pacman\n' "${AUR_BUILDER}" \
>/etc/sudoers.d/99-"${AUR_BUILDER}"-pacman
chmod 0440 /etc/sudoers.d/99-"${AUR_BUILDER}"-pacman
local build_root=/tmp/zfs-utils-build
rm -rf "${build_root}"
install -d -o "${AUR_BUILDER}" -g "${AUR_BUILDER}" "${build_root}"
retry su "${AUR_BUILDER}" -c \
"git clone --depth 1 https://aur.archlinux.org/zfs-utils.git '${build_root}'"
su "${AUR_BUILDER}" -c \
"cd '${build_root}' && makepkg --syncdeps --noconfirm --cleanbuild --skippgpcheck"
local pkg
pkg="$(find "${build_root}" -maxdepth 1 -type f -name 'zfs-utils-*.pkg.tar.*' \
! -name '*-debug-*' | head -n1)"
if [ -z "${pkg}" ]; then
log "ERROR: makepkg produced no zfs-utils package"
return 1
fi
pacman -U --noconfirm "${pkg}"
rm -rf "${build_root}" /var/cache/pacman/pkg
}
# Param: $1 path segment of the zfsonlinux repository, `fedora` or `epel`
# Param: $2 space-separated zfs-release versions to try, newest first
release_install() {
local base="$1" candidates="$2" dist version url
dist="$(rpm --eval '%{dist}')"
for version in ${candidates}; do
url="https://zfsonlinux.org/${base}/zfs-release-${version}${dist}.noarch.rpm"
if dnf install -y "${url}"; then
log "zfs-release: ${url}"
return 0
fi
log "no zfs-release ${version} for ${dist}"
done
log "ERROR: none of the zfs-release versions '${candidates}' exists for ${dist}"
return 1
}
install_dnf() {
case "${ID:-}" in
fedora) retry release_install fedora "${ZFS_RELEASE_FEDORA}" ;;
*) retry release_install epel "${ZFS_RELEASE_EL}" ;;
esac
local zfs_version
zfs_version="$(dnf --quiet repoquery --qf '%{version}\n' zfs | sort -V | tail -n1)"
if [ -z "${zfs_version}" ]; then
log "ERROR: the zfs-release repository offers no zfs package"
return 1
fi
log "zfs userland offered by the repository: ${zfs_version}"
if ! command -v rpmbuild >/dev/null 2>&1; then
retry dnf install -y rpm-build
fi
local topdir=/tmp/zfs-kmod-container
rm -rf "${topdir}"
mkdir -p "${topdir}/SPECS"
cat >"${topdir}/SPECS/zfs-kmod-container.spec" <<SPEC
Name: zfs-kmod-container
Version: ${zfs_version}
Release: 1
Summary: Satisfies the zfs kernel module dependency inside a container
License: CDDL
BuildArch: noarch
Provides: zfs-kmod = ${zfs_version}
%description
The container never loads a zfs module; the host supplies it and the container
reaches it through /dev/zfs. This package exists only so the zfs userland
resolves without pulling a DKMS toolchain and a kernel it will never boot.
%files
SPEC
rpmbuild --define "_topdir ${topdir}" -bb "${topdir}/SPECS/zfs-kmod-container.spec"
dnf install -y "${topdir}"/RPMS/noarch/zfs-kmod-container-*.rpm
retry dnf install -y zfs
if rpm -qa 'kernel-devel*' 'kernel-debug*' 'zfs-dkms' | grep -q .; then
log "ERROR: the transaction pulled kernel or DKMS packages, so the dummy did not take"
rpm -qa 'kernel-devel*' 'kernel-debug*' 'zfs-dkms' >&2
return 1
fi
dnf remove -y zfs-release
rm -rf "${topdir}"
dnf clean all
}
verify() {
local tool
for tool in zfs zpool; do
if ! command -v "${tool}" >/dev/null 2>&1; then
log "ERROR: ${tool} is missing after the install"
return 1
fi
done
if ldd "$(command -v zpool)" | grep -q 'not found'; then
log "ERROR: zpool has unresolved shared libraries"
ldd "$(command -v zpool)" | grep 'not found' >&2
return 1
fi
if command -v dnf >/dev/null 2>&1 && dnf repolist --enabled 2>/dev/null | grep -q '^zfs'; then
log "ERROR: a zfs repository is still enabled, so an upgrade would re-resolve the kmod"
return 1
fi
log "installed $(command -v zfs) and $(command -v zpool); all libraries resolved"
}
main() {
# shellcheck source=/dev/null
. /etc/os-release
if command -v apt-get >/dev/null 2>&1; then
install_apt
elif command -v pacman >/dev/null 2>&1; then
install_pacman
elif command -v dnf >/dev/null 2>&1; then
install_dnf
else
log "ERROR: no supported package manager (apt/pacman/dnf)"
exit 1
fi
verify
}
main "$@"
+2 -1
View File
@@ -11,7 +11,8 @@ bridge (`192.168.244.0/24`, MTU 1400): three swarm nodes (1 manager +
2 workers), a non-swarm NFSv4 server serving the shared volume storage,
and a non-swarm backup host for the DR drill. The node image
([`compose/swarm/Dockerfile`](../../../../compose/swarm/Dockerfile))
bakes python3 + dnsmasq; the containers and the lab network are declared
bakes python3 + dnsmasq and masks the zfs boot units the parent image's
userland enables; the containers and the lab network are declared
in [`compose/swarm/compose.yml`](../../../../compose/swarm/compose.yml) (project
`${SWARM_NAME}`, backup host behind the `drill` profile) and started by
one `docker compose up` in `routine/01_bootstrap.sh`. Lab DNS is provisioned
@@ -23,6 +23,7 @@ IMAGE=/var/tmp/docker-fs.img
POOL_HOST="$(hostname 2>/dev/null || echo unknown)"
POOL="infinito_ci_$(printf '%s' "${POOL_HOST}" | tr -dc 'A-Za-z0-9_.:-')"
DAEMON=/etc/docker/daemon.json
MISC_MAJOR=10
report() { echo "docker-dataroot-filesystem: $*"; }
@@ -63,6 +64,28 @@ arm_autoclear() {
report "WARNING: ${loop} did not arm autoclear (AUTOCLEAR='${armed}')"
}
# Param: $1 pool whose leftovers from an earlier bring-up have to go
#
# Pools live in the kernel, which the lab nodes share with the runner and with
# every later distro of the same sweep, while the container teardown only removes
# containers. A pool of this name is therefore still imported when the next
# iteration starts, and `zpool create` refuses the name rather than reusing it.
# Its vdev is a loop device backed by an image the dead container already
# unlinked, so that space stays claimed until the loop is detached as well.
reclaim_stale_pool() {
local pool="$1" vdev
zpool list -H -o name "${pool}" >/dev/null 2>&1 || return 0
vdev="$(zpool list -vHP "${pool}" | awk 'NR == 2 {print $1}')"
report "destroying ${pool}, left imported on this kernel by an earlier bring-up"
zpool destroy -f "${pool}"
if [ -b "${vdev}" ]; then
losetup -d "${vdev}"
report "detached its vdev ${vdev}"
else
report "WARNING: its vdev '${vdev}' is no block device; that loop stays attached"
fi
}
decline() {
if [ "${REQUIRED}" = true ]; then
report "FAIL: ${FSTYPE} was stated but cannot be delivered: $*"
@@ -135,11 +158,16 @@ zfs)
if ! modprobe_out="$(modprobe zfs 2>&1)"; then
report "modprobe zfs failed: ${modprobe_out}"
fi
zfs_minor="$(awk '$2 == "zfs" {print $1}' /proc/misc)"
[ -n "${zfs_minor}" ] ||
decline "the zfs kernel module is not loaded on the host: ${modprobe_out}"
if [ ! -c /dev/zfs ]; then
zfs_major="$(awk '$2 == "zfs" {print $1}' /proc/devices | head -n1)"
[ -n "${zfs_major}" ] && mknod /dev/zfs c "${zfs_major}" 0
report "the module is loaded but this /dev predates it; creating the node from /proc/misc"
mknod /dev/zfs c "${MISC_MAJOR}" "${zfs_minor}"
fi
[ -c /dev/zfs ] || decline "the zfs kernel module is not loaded on the host: ${modprobe_out}"
[ -c /dev/zfs ] ||
decline "zfs holds misc minor ${zfs_minor} but /dev/zfs could not be created"
reclaim_stale_pool "${POOL}"
;;
esac
@@ -2,18 +2,21 @@
# Resolve which filesystem one matrix entry runs its docker data root on and
# record the decision for the steps that follow.
#
# A random pick draws from what this kernel serves, measured here, narrowed to
# what every distro of the entry carries a userland for. A stated pick is
# honoured even where it is unsupported, and the applying step then fails. The
# choice goes to the step summary, since reproducing a red run means re-stating
# it verbatim.
# A random pick draws from what this kernel serves, measured here. Every distro
# image carries the userland for all three, the zfs one baked by
# scripts/install/zfs.sh, so the pool depends on the kernel alone. A stated pick
# is honoured even where it is unsupported, and the applying step then fails.
# The choice goes to the step summary, since reproducing a red run means
# re-stating it verbatim.
#
# Arguments:
# $1 STATED ext4 | btrfs | zfs; 'auto' or empty for a random pick
# $2 LABEL matrix entry the pick belongs to, e.g. compose/web-app-gitea
# $3 DISTROS space-separated distributions the entry deploys on
# $4 SCOPE node every distro in DISTROS must carry the userland too
# runner DISTROS is ignored; only the kernel decides
# $3 DISTROS space-separated distributions the entry deploys on, recorded
# with the decision so a red run can be restated verbatim
# $4 SCOPE runner | node; names where the pick gets applied. The kernel
# measured here is the runner's either way, because the node
# containers share it.
set -euo pipefail
STATED="${1:-}"
@@ -44,35 +47,12 @@ kernel_serves() {
return 1
}
userland_for() {
case "$1" in
centos | arch | fedora) echo "ext4 btrfs" ;;
*) echo "ext4 btrfs zfs" ;;
esac
}
every_distro_carries() {
local fs="$1" distro
[ "${SCOPE}" = node ] || return 0
for distro in ${DISTROS}; do
case " $(userland_for "${distro}") " in
*" ${fs} "*) ;;
*) return 1 ;;
esac
done
return 0
}
candidates() {
local kept="" fs
for fs in ${POOL}; do
if ! kernel_serves "${fs}"; then
continue
fi
if ! every_distro_carries "${fs}"; then
note "${fs} left out: not every distro of this entry carries the userland"
continue
fi
kept="${kept} ${fs}"
done
kept="${kept# }"
@@ -91,7 +71,7 @@ else
REQUIRED=false
fi
echo "filesystem for ${LABEL}: ${PICKED} (${ORIGIN})"
echo "filesystem for ${LABEL} on '${DISTROS}': ${PICKED} (${ORIGIN})"
if [ -n "${GITHUB_ENV:-}" ]; then
{
@@ -10,7 +10,9 @@ one is not, a stated pick overrides the pool, and a pick is always drawn from a
pool the target can serve.
The pool contents are policy, not contract, and are deliberately not asserted -
the per-distro expectation is derived from the resolver's own answers.
the expectation is derived from the resolver's own answers. What is asserted is
that the pool depends on neither the distro set nor the scope, because every
distro image carries all three userlands.
"""
from __future__ import annotations
@@ -122,10 +124,15 @@ class TestFilesystemResolve(unittest.TestCase):
pool = resolve("", " ".join(ALL_DISTROS), scope).pool
self.assertTrue(set(absent).isdisjoint(pool))
def test_the_node_pool_also_drops_what_a_distro_cannot_install(self) -> None:
runner = set(resolve("", " ".join(ALL_DISTROS), "runner").pool)
node = set(resolve("", " ".join(ALL_DISTROS), "node").pool)
self.assertTrue(node.issubset(runner))
def test_the_distro_set_does_not_narrow_the_pool(self) -> None:
every = resolve("", " ".join(ALL_DISTROS), "node").pool
for distro in ALL_DISTROS:
self.assertEqual(resolve("", distro, "node").pool, every)
def test_both_scopes_draw_from_the_same_pool(self) -> None:
runner = resolve("", " ".join(ALL_DISTROS), "runner").pool
node = resolve("", " ".join(ALL_DISTROS), "node").pool
self.assertEqual(node, runner)
def test_a_missing_scope_is_a_hard_error(self) -> None:
proc = subprocess.run(
+1
View File
@@ -60,6 +60,7 @@ STATIC_KEYS = (
"INFINITO_DOCKER_CLI_INSTALL_SCRIPT",
"INFINITO_PACKAGE_INSTALL_SCRIPT",
"INFINITO_PYTHON_INSTALL_SCRIPT",
"INFINITO_ZFS_INSTALL_SCRIPT",
"INFINITO_ALIAS_REPOSITORY",
"INFINITO_FORK_REPOSITORY_NAME",
"INFINITO_PLAYWRIGHT_KEEP",