The pick was made once per matrix entry, in a workflow step ahead of the drill,
and every distro of that entry then ran on it. Coverage followed: one entry
exercised one filesystem, and the swarm never reached zfs at all, because
resolve.sh's userland_for() kept it out of any entry containing arch, centos or
fedora - which is every automatic sweep. The draw now happens inside the loop in
distros.sh, once per distro, so a five-distro entry covers zfs with 87% and all
three kinds with 62% probability instead of one kind with certainty.
The workflow input changes meaning with it. It was a single kind, offered as a
choice in the manual entry; it is now a space-separated allow-list. Empty draws
from all three, and naming kinds narrows the draw to them and marks the pick
required, so a host that cannot deliver one of them fails rather than quietly
running on something else. A name that is not a filesystem this project puts a
data root on is rejected outright instead of reaching the applying step.
For the draw to mean anything the images have to carry the userlands, so
scripts/install/zfs.sh becomes scripts/install/filesystem.sh and installs ext4
and btrfs alongside zfs. That is not symmetric work: btrfs-progs does not exist
in the EL family at all, because RHEL dropped btrfs, and a plain
`dnf install btrfs-progs` there fails the whole transaction and takes e2fsprogs
with it. EPEL carries it, so the EL family goes through EPEL from the start -
split by distro rather than by a failed first attempt, since Fedora has no
epel-release to fall back to and routing a transient failure there would turn it
into a fatal one. verify() now demands all four binaries, so a packaging
regression fails the build instead of leaving a hole the drill discovers later.
Measured on all five bases: debian trixie, ubuntu 26.04, arch, fedora 44 and
centos 10 each end with mkfs.ext4, mkfs.btrfs, zfs and zpool present and ldd
fully resolved.
Two defects only became reachable once a single job could ask for different
filesystems in turn:
* docker_dataroot.sh answered "already prepared" and returned for any mount it
found, so from the second distro on, every draw would have been silently
ignored on a host that survives the sweep - the runner in compose and host
mode. It now compares what is mounted against what was asked for and
dismantles the old root when they differ. Everything docker cached there
goes with it; that is what covering more than one filesystem per entry
costs.
* reclaim_stale_pool ran only when the new pick was zfs, so a zfs to btrfs
transition left the previous distro's pool imported on the shared kernel,
pinning a loop device over an image its dead container had already unlinked.
It now runs on every pick, which needed the /dev/zfs handling lifted out of
the zfs branch into zfs_device_ready().
The restore_docker trap is armed before anything can stop docker, rather than
after, so a failure while dismantling a data root cannot leave a host without a
daemon. resolve.sh splits its allow-list into an array before testing it, so a
whitespace-only input is the empty list it looks like rather than a narrowed one
that yields an empty pick marked required. pick.sh clears both keys when it
declines to draw: they are declared in default.env, so a generated .env carries
whatever was exported when it was made, and a caller gating its apply on the
value would otherwise repoint the developer's own docker in the very case the
skip exists to prevent.
Verified by execution rather than by reading: a privileged container ran
docker_dataroot.sh with real loop devices and mkfs through ext4 -> ext4 -> btrfs
-> ext4, confirming the recognition, the replacement in both directions and that
no loop leaks across it; a second harness with a recording zpool stub confirmed
that a btrfs pick does reclaim a stale pool, that no pool means no destroy, and
that a host without /dev/zfs still completes an ext4 pick without touching zpool.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dockerd's zfs graphdriver resolves `zfs` on PATH and opens /dev/zfs before it
accepts a data root (moby daemon/graphdriver/zfs/zfs.go:54,59), so the userland
has to sit in the image, not on the host. Only the apt images carried it, which
is why resolve.sh's userland_for() kept zfs out of every entry containing arch,
centos or fedora - that is every automatic swarm sweep, so the zfs data root and
the zfs branch of the backup launcher were never once exercised. Run
30781723102 shows it: "zfs left out: not every distro of this entry carries the
userland".
scripts/install/zfs.sh joins the three existing install scripts in the image
build and takes a different route per distro, each one run in a real container
of the matching base image before landing here:
debian trixie 2.3.2 zfsutils-linux, contrib enabled first
ubuntu 26.04 2.4.1 zfsutils-linux
arch 2.4.3 the AUR PKGBUILD, which configures --with-config=user
fedora 44 2.4.3 zfsonlinux repo plus a dummy zfs-kmod provider
centos 10 2.2.10 the same, from the epel path
The dummy is what keeps the RPM route usable: the `zfs` package carries an
ungated `Requires: %{name}-kmod`, so a plain `dnf install zfs` resolves through
zfs-dkms and installs 389 extra packages and 1.44 GB of kernel and toolchain to
build a module against a kernel the container never boots. Providing zfs-kmod
from an empty package keeps the same install at 60 packages with no kernel
package at all, and the transaction is checked afterwards so a regression fails
the build instead of bloating the image. The repository is removed again once
the userland is in, because the dummy pins one version and an upgrade would
otherwise re-resolve the kmod at deploy time.
Version drift against the host module is not a concern: zpool create enables a
feature only when the loaded module reports it through /sys/module/zfs/features
.pool (cmd/zpool/zpool_main.c, module/zcommon/zfeature_common.c), and libzfs
compares no versions at all (lib/libzfs/os/linux/libzfs_util_os.c:123-134), so a
newer userland degrades rather than breaking.
With every image carrying it, userland_for() and every_distro_carries() have
nothing left to decide and go. Two defects that only became reachable once zfs
could be drawn are fixed in the same move:
* /dev/zfs is a misc device, so the old lookup in /proc/devices could never
find its major and the node was never created. A node container's /dev is
populated at creation and udevd is masked, so nothing else fills the gap
either: the first bring-up of every job would have declined, blaming a
module that modprobe had just loaded. The minor now comes from /proc/misc.
* Pools live in the kernel the lab nodes share with the runner, while the
teardown only removes containers, and the node hostnames repeat across the
distro sweep. From the second distro on, zpool create hit the previous
iteration's pool. It is destroyed and its loop detached first.
The node image drops its own apt-only zfs branch and instead masks the six boot
units the userland enables by preset, which on a node can only fail: the pool is
created after boot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RUNNER_POOL and NODE_POOL were assertions. With a distro list of just debian
and ubuntu the node pool offered zfs, because the table records that those
distros ship the zfs userland - but the module comes from the runner's kernel,
which carries none. One run in three would then have drawn zfs and either
declined silently, testing ext4 while the summary claimed zfs, or hard-failed on
all five nodes with nothing actually broken.
Ask the kernel instead: /proc/filesystems for what is loaded, modinfo for what
could be. Both answer unprivileged, which resolve.sh needs since it runs without
sudo. The modinfo half carries weight rather than being belt-and-braces - xfs is
absent from /proc/filesystems on this host while modinfo finds the module, so
without it the pool would shrink to whatever happens to be loaded.
The distro table stays above it for the other question: which userland a
container can install. Two layers because they are two things, and only the
kernel one cannot be fixed by a package.
Every exclusion now names its reason on stderr and distinguishes a missing tool
from a missing capability; a filesystem used to drop out of the pool in silence.
For the full five-distro sweep the pool is unchanged: ext4 btrfs, measured
before and after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An opencloud snapshot carried 174 entries, and every line repeated the whole
path - the deepest ran about 150 characters, most of it the same prefix as the
line above. Print the basename at its depth instead, with a glyph per kind:
folder, json, journal or log, plain text.
The separator is swapped for \001 across the sort. Without it a directory and
its children come apart: '/' outranks '.', so a sibling file sorts between the
two and reads as though it lived inside. The swap also makes the order
independent of the locale, which was deciding it before.
The existing cut-short test looked for 'logs/container.log', which no longer
appears, so it now checks the basename; a fourth case pins the structure rather
than the wording - a child is indented deeper than its folder and siblings sit
level.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
mbake wraps the WORDPRESS_ADDONS_REQUIRED jinja chain, ruff collapses a path
that now fits one line and normalises the quotes of a stub literal. No value,
expression or string content changes; each line was read rather than trusted.
These files were committed during this session without autoformat having run
over them, so the next quality gate would have produced exactly this diff.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three violations in test files written earlier today, all surfaced by the first
full quality run rather than by the targeted checks I had been running.
test_addons_vendored_files built its finding list with append in a loop.
test_env_handlers_registered walked the tree with rglob, bypassing the shared
project cache the lint exists to protect on a 250+ role tree.
test_resolve read two keys that resolve.sh writes per matrix entry into
GITHUB_ENV, which can never be .env entries; the second key now derives from the
first so a single line carries the marker - and stays short enough that ruff
format leaves it on the line the lint inspects.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
find and du write what they already collected and only then return nonzero, so
the guards added in ce1c5b5fe were throwing away the evidence the index exists
to report. One vanished entry mid-walk blanked the whole listing: '0 file(s)',
no paths, on a tree that was mostly readable. The rescue tree is indexed while
a failed run is still tearing down, so that is the normal case, not the corner
one.
Stop resetting LISTING, fall back on SIZE only when du produced nothing, and
turn the early exit into a warning so the partial listing still prints.
The existing test could not see this: its stub find emitted nothing, so it only
covered a total failure. The new case stubs a find that prints rows and then
fails, and it fails against ce1c5b5fe.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every caller runs this in a cleanup path: three shell teardown handlers and a
GitHub step under bash -e, none of them with a fallback. A nonzero rc therefore
aborts the caller's remaining teardown or adds a second red step on top of the
failure being rescued. Until now that could not happen only because each
internal failure path happened to be caught - a property that dies with the
next edit. A terminal 'trap exit 0' states the guarantee once and makes it
independent of the enumeration.
The failure is not hypothetical: a find that rejects -printf (BusyBox) makes
the old script exit 1, reproduced with a stub.
Collapses three walks of the same tree into one, and counts files with awk
rather than grep -c, which answers 1 on zero matches and takes the assignment
down with it under pipefail. A walk that fails now says so - it used to print
'0 file(s)', which reads exactly like an empty tree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every unit test for a shell script sat flat in tests/unit/scripts/ with the
path encoded into the filename - test_worktree_lib.py, test_filesystem_resolve.py,
test_swarm_diagnostics_routing.py. Finding the test for a script meant guessing
which words of its path had made it into the name.
Mirror the layout instead, the way tests/unit/utils/ already does:
scripts/system/worktree/lib.sh is now tested by
tests/unit/scripts/system/worktree/test_lib.py. The name prefixes that stood in
for directories disappear with them.
Pure relocation - 0 insertions, 0 deletions. All eight moves are recorded as
renames, so git log --follow still reaches the history.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-matrix selection never picked anything but ext4. candidates()
intersected what every distribution in the entry supports, and with the list the
orchestrator passes - arch debian ubuntu fedora centos - that leaves ext4 alone:
centos stream has no btrfs, zfs is debian/ubuntu only. btrfs was reachable only
by stating it in the manual input.
The intersection is right for swarm and wrong for the other two. distros.sh
runs 00_one.sh per distribution and each round builds its own lab, so a swarm
pick must hold in every round's nodes. compose and host carve the data root on
the runner itself, whose OS is fixed; there DISTROS names the application images
and has nothing to do with the filesystem underneath docker.
A fourth argument says which, rather than parsing it out of the label, and
guards with `:?` so a forgotten call site fails loudly. The runner pool omits
zfs: the runner-image build installs neither the zfs userland nor
linux-modules-extra, where the Ubuntu azure kernel keeps zfs.ko, so a drawn zfs
would decline on every run. A stated one still overrides the pool and is marked
required, so `filesystem: zfs` fails loudly instead of quietly falling back.
The test executes the resolver rather than reading it, and pins the relation
that breaks silently: the keys it writes to GITHUB_ENV are the keys the three
workflows read. A typo on either side hands docker_dataroot.sh an empty string,
which reports "no filesystem stated" and exits 0 - the feature switched off
under a green run. Pool contents are policy and are not asserted; the per-distro
expectation is derived from the resolver's own answers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four checks and one assertion asserted that a literal appears in a file:
`PATTERN="${1:?`, `UNIT_DUMPS="${…:?`, three strings inside two other files,
and a message the collector prints. None of them describes a relation - they
describe the spelling of code that is itself the source of truth, and each one
would have failed a legitimate rewording while proving nothing about behaviour.
What stays checks something the code cannot state alone: that every call site
of the trigger passes the directory (two counts of the same file compared
against each other, which is what found two call sites a grep for the filename
could not see), and that neither file carries its own copy of the rescue path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill's remote pull exits 1 after 65 minutes and nothing says why. The
unit loops over two providers and only counts errors, so which one failed is
unrecoverable; the pull runs `rsync -abP`, whose --progress writes one journal
line per file - 77137 of them on a nextcloud pull - and the drill captured
`journalctl -n 60`. The entire diagnostic was therefore rsync filenames.
The unit now names the host it could not pull from. rsync keeps --partial and
trades --progress for --info=stats2, turning the flood into a closing balance.
trigger_units.sh writes the complete journal of a failed unit into the rescue
dump directory the artifact ships as local-dumps, and prints only the lines
from warning upwards.
That directory arrives as an argument instead of a second copy of the paths
SPOT: the drill reads INFINITO_RESCUE_DIAGNOSTICS_DIR once, which the env
handler derives from group_vars/all/05_paths.yml, and hands it to every call
site. A test pins that contract, and it earned its place immediately - it found
two call sites that a grep for the file name cannot see, and both were the
units this change exists for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Of six CI failures triaged today, three were explainable only from downloaded
artifacts and two not even there: the job log carried "not converged:
docker_docker" or a bare "rc=1", while this collector drowned that same log in
compose trees, env listings and ganesha thread stacks.
diagnostics.sh switches its stdout per section with exec, so every topic lands
in its own file under the rescue directory and the log keeps one line. An
unwritable output directory degrades to stdout rather than killing the
collector - a failed mkdir would otherwise abort the script at the first
redirection, which the caller swallows with `|| true`, leaving no diagnostics
at all and no sign of it. For the same reason the collector says so when it
ends up having captured nothing.
00_one.sh hands the same directory to both collectors. The variable was scoped
to the python one, so the shell collector wrote where no artifact picks it up.
The capture grows by exactly what today's investigations went without: per-node
resolv.conf, daemon.json, listening sockets, NAT rules and a resolve probe
against the three hosts that killed builds, plus df and docker system df.
rescue_index.sh runs last and lists what was collected together with the note
that it is downloadable. The index is the one thing worth printing.
A test pins the routing against a stub docker: one file per section, no bleed
between them, a single log line, and the degrade path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`ruff`'s B905 wants every `zip()` to say whether it tolerates operands of
different length. Here it never sees one: both operands are the `packed`
form of an address from the same network, so both are four bytes for IPv4
and sixteen for IPv6. `strict=True` states that, and turns a silently
truncated - and therefore too permissive - regex into an error should the
invariant ever break.
The three other files carry `ruff format` reflows the same run produced.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tests/lint/filesystem/python/test_no_uncached_file_reads.py requires every read
under tests/, utils/, cli/, plugins/ and library/ to go through read_text from
utils.cache.files, because the same files are read many times across the suite.
3a77165d6 landed with two raw Path.read_text() calls and left that lint red; the
commit only ran the comment lint before landing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The swarm gate asked whether the drill ran by grepping the job log for
"Matrix-deploy <app>: provision/deploy/e2e/verify per round" — a copy of a
GitHub Actions step display name. d9a4498b3 renamed that step to
"🚀 Drill <apps> per distro: <distros>" and did not touch roundtrip.sh, so at
19d56f07e the literal existed in exactly one place in the tree: the grep itself.
It could no longer match anything, and four of the five workspace legs of run
30586213932 false-failed on deploys that had passed — svc-db-postgres converged,
answered on 5432, completed its DR drill and reported "Job succeeded" three lines
above the failure.
This is the second time the same coupling drifted. 0ca2ed56c is titled "align
roundtrip swarm gate to the live workflow step name" and records that the
previous stale marker made every swarm leg false-fail on a passing deploy.
Nothing cross-checked the literal, so it rots silently after each rename.
The marker now comes from the drill itself, printed after 07_assert_state.sh.
Every step before it runs under set -euo pipefail and the EXIT trap tears down
without printing, so the line is unreachable unless the drill completed for that
app. APP_ID and INFINITO_DISTRO are :?-guarded at the top of the script, so the
marker cannot degrade into an empty string that matches everything. The gate
matches it with grep -qF and names what it checked instead of asserting "empty
matrix" — a diagnosis that was never true and sent this investigation into the
wrong layer.
The gate itself stays. 214672a94 added it because act exits 0 on a zero-leg
matrix, which silently false-passed every local swarm roundtrip.
tests/unit/scripts/test_roundtrip_marker.py pins emitter and gate to the same
literal, pins the marker behind the state assertion, and fails if the old step
name reappears — the cross-check whose absence allowed both drifts.
Verified against the failing run's log: 0 occurrences of the old literal, while
DR drill PASSED, ran=1 skipped=0 and "Job succeeded" prove the drill reached the
point where the new marker is emitted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two problems, same two files, neither ever gated: the earlier run died in
autoformat before test-unit started.
ruff wanted Path.stat/Path.chmod over their os counterparts and an explicit
check on subprocess.run — worth stating here, since down() is called precisely
to observe a non-zero exit while its sibling git() sets check=True.
The write-bit pinning the four remaining tests rely on does not restrain root,
and the suite runs as root in the container: the script deleted the directory
anyway, the assertion inverted, and the finally block then chmod'd a path that
was already gone. Those four now skip when euid is 0, because the scenario
cannot exist there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
git worktree remove deletes the checkout first and the metadata dir
second. Under the agent sandbox, which holds config.worktree and
commondir as read-only bind mounts, it therefore dies with EBUSY after
the checkout is already gone, aborting make worktree-down with rc 255
even though the branch is in fact free. down.sh now falls back to the
unregistration prune.sh already used (drop gitdir + HEAD, best-effort
rm -rf) and reports any dir the sandbox still pins. That routine moves
into lib.sh so both callers cannot drift apart, and gains a third
outcome for the case where gitdir/HEAD themselves survive: only then is
the branch still claimed, and only then must the teardown fail.
The discriminator for the fallback is whether the checkout survived,
not a match on "Device or resource busy" — the identical half-removed
state arises from EACCES, EPERM and EROFS.
A failing git status probe no longer reads as clean: on a broken .git
link it exits 128 with empty output, which used to satisfy the guard,
tear down the compose stack, and only then die at git worktree remove
with an unexplained 128.
Measured against the unfixed scripts with real fixtures: pinned
metadata dir 255 -> 0, broken .git link 128 -> 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The loop may drop a distro when the budget runs out, and both skip paths fell through to an implicit exit 0, so a run that never touched two distros still reported success. That hid the coverage gap: arch, fedora and centos get no package cache (cache.override.yml is apt-only), are therefore the slowest, and are the first the remaining < max_seen heuristic drops.
Skipping stays non-fatal and the order stays randomised, but every distro now shows up in a GITHUB_STEP_SUMMARY table in execution order, as passed, failed with its rc, or skipped with the reason. The table is also rendered before an aborting exit, since that is the run whose table matters. Two contract tests cover it; the order assertion checks the logged shuf order rather than the input list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The serial distro loop (shuffle, time budget, fast-fail skip, exit on the first failing distro) lived inline in scripts/tests/deploy/ci/all.sh and was therefore reachable only from the compose and host deploys. Swarm and guide have to iterate distros with the same semantics, so the loop moves into scripts/tests/deploy/distros.sh and takes the per-distro command as "$@".
ci/all.sh keeps the app-level preamble and execs the loop with the new ci/one.sh, which resolves the CI image and dispatches to the per-app override or dedicated.sh.
Image resolution leaves the loop as well: scripts/meta/resolve/image/ci.sh renders the reference through cli.meta.ci.image_ref instead of respelling ghcr.io/... locally, which meta/distros.yml declares as the single source.
The two CI time knobs are declared in default.env instead of being repeated as literals, so no caller has to invent a budget.
Three paths are dropped as dead: the empty budget that disabled the limit, the failed counter that could only ever print 0, and the branch for an unset deadline.
Both new scripts are pinned by contract tests. Those tests clear BASH_ENV because the test container exports it as the env loader, which would otherwise re-inject every stripped key and make the assertions pass for the wrong reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
