scripts/install/filesystem.sh retried `dnf install -y zfs` three times
against the same metadata during the Fedora image build of main run
39684179395 (XMPP#0): every attempt listed pcp-*-7.2.2-1.fc44, every
mirror answered 404, "Failed to download packages" after 3 attempts.
Each retried dnf install now runs with --refresh, so every attempt
resolves against freshly fetched metadata instead of repeating the one
that lists packages no mirror serves anymore.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Both image-build jobs of run 37494541850 failed, and only centos broke; arch,
debian, ubuntu and fedora built. The headline reads like an outage:
Error: Failed to download metadata for repo 'baseos':
Cannot download repomd.xml ... All mirrors were tried
The line above it says otherwise:
Errors during downloading metadata for repository 'baseos':
- Downloading successful, but checksum doesn't match.
Calculated: 76995e25878232aba76bc22834280007be1baf40...
The download succeeds; the repomd.xml served does not match the checksum the
metalink advertises. The decisive detail is that this digest is byte-identical
on attempts 1, 2 and 3. A real re-fetch reaches a different mirror and yields a
different digest, so attempts 2 and 3 fetched nothing: dnf had written the bad
metadata to /var/cache/dnf and retry only slept and re-invoked the same command
against the same file. Three attempts, one download, and the same for all seven
build attempts.
retry_dnf is retry plus a metadata clean before each re-run, so the next
attempt is a fetch and can land on a mirror that is not mid-sync. All ten dnf
call sites use it, not just the one that failed: the zfs builddep and install
paths carry the identical weakness. apt and pacman keep retry, having no
equivalent cache trap.
The success path is unchanged, so the common case behaves identically. On the
failure path the script runs under set -e, which aborts if the clean itself
returns non-zero: measured, and it fails closed and loud rather than continuing
against a stale cache.
This fixes the retry, not the mirror. If the EL mirrors served stale metadata
for the whole window, three real fetches still fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The filesystem install ran in the RUN after 'COPY .', so any commit
rebuilt the zfs userland: the AUR build on Arch and, on arm64, the
source RPM rebuild on Fedora and CentOS, all of it under emulation now
that one job builds every architecture. It now runs in its own layer
right after default.env and scripts/install are copied, which only
those two paths invalidate. apt-mirrors.sh moves with it and still runs
first.
Running before the Python step, filesystem.sh no longer finds a system
that step already upgraded, so its pacman call does the full -Syu
instead of a partial -Sy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zfsonlinux serves no aarch64 metadata for either Fedora or EL. dnf5
skipped the missing repository and fell through to the source rebuild,
but dnf4 on CentOS Stream aborted every transaction that loaded it, so
the arm64 centos image died at the repoquery.
The repoquery now skips an unavailable zfs repository, and the source
path disables it on every call and removes zfs-release right after the
source RPM is downloaded. The pieces that differ between dnf
generations follow the installed one: dnf-plugins-core or dnf5-plugins,
'download --source' (dnf5 keeps it as an alias of --srpm), and 'mark
remove' or 'mark dependency'. EL enables CRB for builddep, where the
devel dependencies live, and the package snapshot moves ahead of the
repoquery so the plugins count as build tooling too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The arm64 Fedora image failed because zfsonlinux publishes Fedora RPMs
for x86_64 only, so repoquery found no zfs to pin the dummy kmod to.
An empty repoquery now downloads the source RPM of the same release,
takes the version from it, rebuilds it and installs the runtime
packages only.
Everything installed from rpm-build onwards is marked as a dependency
and autoremoved afterwards. A plain dnf remove refuses because builddep
pulls the protected systemd, and taking the snapshot after rpm-build
left gcc held by redhat-rpm-config's annobin requirement.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pick was made once per matrix entry, in a workflow step ahead of the drill,
and every distro of that entry then ran on it. Coverage followed: one entry
exercised one filesystem, and the swarm never reached zfs at all, because
resolve.sh's userland_for() kept it out of any entry containing arch, centos or
fedora - which is every automatic sweep. The draw now happens inside the loop in
distros.sh, once per distro, so a five-distro entry covers zfs with 87% and all
three kinds with 62% probability instead of one kind with certainty.
The workflow input changes meaning with it. It was a single kind, offered as a
choice in the manual entry; it is now a space-separated allow-list. Empty draws
from all three, and naming kinds narrows the draw to them and marks the pick
required, so a host that cannot deliver one of them fails rather than quietly
running on something else. A name that is not a filesystem this project puts a
data root on is rejected outright instead of reaching the applying step.
For the draw to mean anything the images have to carry the userlands, so
scripts/install/zfs.sh becomes scripts/install/filesystem.sh and installs ext4
and btrfs alongside zfs. That is not symmetric work: btrfs-progs does not exist
in the EL family at all, because RHEL dropped btrfs, and a plain
`dnf install btrfs-progs` there fails the whole transaction and takes e2fsprogs
with it. EPEL carries it, so the EL family goes through EPEL from the start -
split by distro rather than by a failed first attempt, since Fedora has no
epel-release to fall back to and routing a transient failure there would turn it
into a fatal one. verify() now demands all four binaries, so a packaging
regression fails the build instead of leaving a hole the drill discovers later.
Measured on all five bases: debian trixie, ubuntu 26.04, arch, fedora 44 and
centos 10 each end with mkfs.ext4, mkfs.btrfs, zfs and zpool present and ldd
fully resolved.
Two defects only became reachable once a single job could ask for different
filesystems in turn:
* docker_dataroot.sh answered "already prepared" and returned for any mount it
found, so from the second distro on, every draw would have been silently
ignored on a host that survives the sweep - the runner in compose and host
mode. It now compares what is mounted against what was asked for and
dismantles the old root when they differ. Everything docker cached there
goes with it; that is what covering more than one filesystem per entry
costs.
* reclaim_stale_pool ran only when the new pick was zfs, so a zfs to btrfs
transition left the previous distro's pool imported on the shared kernel,
pinning a loop device over an image its dead container had already unlinked.
It now runs on every pick, which needed the /dev/zfs handling lifted out of
the zfs branch into zfs_device_ready().
The restore_docker trap is armed before anything can stop docker, rather than
after, so a failure while dismantling a data root cannot leave a host without a
daemon. resolve.sh splits its allow-list into an array before testing it, so a
whitespace-only input is the empty list it looks like rather than a narrowed one
that yields an empty pick marked required. pick.sh clears both keys when it
declines to draw: they are declared in default.env, so a generated .env carries
whatever was exported when it was made, and a caller gating its apply on the
value would otherwise repoint the developer's own docker in the very case the
skip exists to prevent.
Verified by execution rather than by reading: a privileged container ran
docker_dataroot.sh with real loop devices and mkfs through ext4 -> ext4 -> btrfs
-> ext4, confirming the recognition, the replacement in both directions and that
no loop leaks across it; a second harness with a recording zpool stub confirmed
that a btrfs pick does reclaim a stale pool, that no pool means no destroy, and
that a host without /dev/zfs still completes an ext4 pick without touching zpool.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbach
