The small GitHub runner shape (AWS, 72G root) ran out of disk in swarm and compose rows (#1329: WordPress#2 of run 41261531965, Nextcloud#6 and #7 of run 41262286962) while an empty instance-store disk nvme1n1 sat unused. mount_spare_disk.sh formats and mounts it at /mnt, but only on a GitHub-hosted runner outside act, and only a whole unpartitioned disk without any signature, mount or swap use of at least 75G; every decline leaves a notice. relocate_docker_dataroot.sh then bind-mounts /var/lib/containerd from the spare disk and moves the Docker data-root to /mnt/docker-host (not /mnt/docker, which is the nested daemon's volume that dedicated.sh wipes). It refuses before stopping anything when images, volumes, build cache, foreign containerd namespaces or shims exist or daemon.json is unreadable, and rolls back the bind mount and daemon.json on any later failure or cancel. Swap goes on / once containerd lives on /mnt, dump_state.sh logs disk size, model and free space per storage root, and the swarm watchdog samples / and /var/lib/containerd every 10 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
0d8ed0ba8 bounded these call sites with one ceiling for both operations.
The two are not equally safe to cut. A SIGTERM landing mid-postinst
leaves dpkg interrupted, and in docker_dataroot.sh that fails every later
filesystem candidate and sends the run through decline() - one truncated
package install becomes a dead job. An interrupted update has no such
cascade; it just repeats.
Measured on ubuntu:latest with the exact commands: update 10s, install
zfsutils-linux 20s. 1200s is a 60x margin on the operation that carries
the cascade, and update keeps the 600s it cannot plausibly need more than.
This breaks the single-number invariant with scripts/install/apt.sh:19,
which is already broken in fact - that call site carries a different
Acquire option set than these three do.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A run-wide draw proved one distribution and said nothing about the other four, while reporting green. Both are matrix-row axes now, assigned by the same deterministic rotation mode and tor already use, so one sweep exercises every distribution and every filesystem.
The two read the row's position like an odometer, the distro as the low digit. Turning both on the position directly covers only n of the n x m pairs whenever the pools are the same length, and no sweep unlocks it.
Both are part of a row's identity: glyphs in the job label, columns in the plan table, shards in the artifact name. Both are selectable through %distro and /filesystem, and a retrigger replays the distro off a failed title. It leaves the filesystem out, because a title states the kind the matrix assigned and a deploy may have fallen back from it.
A row's filesystem is a preference unless a human names it, on the run or in a token. Reading the matrix's own narrowing as a demand would fail rows for every transient condition the applying step is built to tolerate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ba8a72309 bounded scripts/install/apt.sh and d50ba46b0 bounded the swarm
bootstrap; these three were the remainder, and one of them was still
costing whole jobs. docker_dataroot.sh hung Nextcloud#7 (job 96179137223,
run 32284644282) for 5h40m: "installing zfsutils-linux for zpool" at
19:50:39 is the last line before "rc=124 after 20400s" at 01:30:39, with
nothing in between. distros.sh's global budget is the only ceiling below
these calls, so one stalled fetch also takes the untried distros with it.
Both update and install are wrapped, not just update - that hang sat in
the install, and apt-get install fetches from the same mirror. Constants
come verbatim from scripts/install/apt.sh:19-20 rather than being
re-derived.
host_deploy.sh runs inside a container via `docker exec ... sh -c`, where
POSIX sh has no arrays, so its options are a word-split string instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In job 91983553540 all five lab nodes derived the same pool name,
infinito_ci_unknown, and the run died on `cannot destroy
'infinito_ci_unknown': pool is busy`. The name came from `hostname`, which
returns non-zero inside the node containers, so the `|| echo unknown` fallback
answered for every one of them. One node won the race and created the pool; the
others found it, took it for a leftover, and tried to tear down a pool a
sibling was actively using.
234c1158c already fixed this collision class three days ago by deriving the
name from `hostname` instead of a constant, and its message named the zfs
exclusion in resolve.sh as the only thing keeping the collision out of sight.
That fix was never exercised, because the exclusion stayed up until ed1121053
removed it - and it does not work, because the call it rests on fails in the
containers it was written for. Guessing the identity is the defect; the
playbook has known it all along as inventory_hostname, and ci/one.sh knows its
own deploy mode. Both now state it, as a required fourth argument with no
default, the way SIZE is already stated. verdict() stops guessing too: every
row it wrote into the job summary said `unknown`, which is why the summary was
never worth reading.
The second half is independent of the naming. reclaim_stale_pool ran
`zpool destroy -f` as a bare command under `set -euo pipefail`, the only
failure path in this script that bypassed decline(). A pool that refuses to be
destroyed is by definition not a leftover - something holds it - so that case
now declines through the usual contract: a drawn pick falls back and reports,
a stated one fails loudly as required-but-unavailable. A reclaim can no longer
be more destructive than the state it exists to clean up.
Verified over the real invocation path rather than a model of it: an
ansible-playbook run using the same `script:` task and docker connection as
compose/swarm/playbook.yml, against three containers in one play, has each node
address its own pool - infinito_ci_race-mgr-01, -wrk-01, -nfs-01 - with
failed=0 throughout. A stubbed-pool harness covers the paths that need a kernel
module this machine does not have: a sibling holding the pool declines with rc
0 instead of ending the run, the same case with a stated pick exits 1 through
the contract, a genuine leftover is still destroyed, and a missing identity is
a hard error. The earlier replacement harness (ext4 recognised, btrfs
replacing it, no loop leaked) still passes against the stricter signature.
Pool names now carry dashes, which zfs_namecheck.c accepts; at 42 characters
they sit far inside the 240 limit and match no reserved prefix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pick was made once per matrix entry, in a workflow step ahead of the drill,
and every distro of that entry then ran on it. Coverage followed: one entry
exercised one filesystem, and the swarm never reached zfs at all, because
resolve.sh's userland_for() kept it out of any entry containing arch, centos or
fedora - which is every automatic sweep. The draw now happens inside the loop in
distros.sh, once per distro, so a five-distro entry covers zfs with 87% and all
three kinds with 62% probability instead of one kind with certainty.
The workflow input changes meaning with it. It was a single kind, offered as a
choice in the manual entry; it is now a space-separated allow-list. Empty draws
from all three, and naming kinds narrows the draw to them and marks the pick
required, so a host that cannot deliver one of them fails rather than quietly
running on something else. A name that is not a filesystem this project puts a
data root on is rejected outright instead of reaching the applying step.
For the draw to mean anything the images have to carry the userlands, so
scripts/install/zfs.sh becomes scripts/install/filesystem.sh and installs ext4
and btrfs alongside zfs. That is not symmetric work: btrfs-progs does not exist
in the EL family at all, because RHEL dropped btrfs, and a plain
`dnf install btrfs-progs` there fails the whole transaction and takes e2fsprogs
with it. EPEL carries it, so the EL family goes through EPEL from the start -
split by distro rather than by a failed first attempt, since Fedora has no
epel-release to fall back to and routing a transient failure there would turn it
into a fatal one. verify() now demands all four binaries, so a packaging
regression fails the build instead of leaving a hole the drill discovers later.
Measured on all five bases: debian trixie, ubuntu 26.04, arch, fedora 44 and
centos 10 each end with mkfs.ext4, mkfs.btrfs, zfs and zpool present and ldd
fully resolved.
Two defects only became reachable once a single job could ask for different
filesystems in turn:
* docker_dataroot.sh answered "already prepared" and returned for any mount it
found, so from the second distro on, every draw would have been silently
ignored on a host that survives the sweep - the runner in compose and host
mode. It now compares what is mounted against what was asked for and
dismantles the old root when they differ. Everything docker cached there
goes with it; that is what covering more than one filesystem per entry
costs.
* reclaim_stale_pool ran only when the new pick was zfs, so a zfs to btrfs
transition left the previous distro's pool imported on the shared kernel,
pinning a loop device over an image its dead container had already unlinked.
It now runs on every pick, which needed the /dev/zfs handling lifted out of
the zfs branch into zfs_device_ready().
The restore_docker trap is armed before anything can stop docker, rather than
after, so a failure while dismantling a data root cannot leave a host without a
daemon. resolve.sh splits its allow-list into an array before testing it, so a
whitespace-only input is the empty list it looks like rather than a narrowed one
that yields an empty pick marked required. pick.sh clears both keys when it
declines to draw: they are declared in default.env, so a generated .env carries
whatever was exported when it was made, and a caller gating its apply on the
value would otherwise repoint the developer's own docker in the very case the
skip exists to prevent.
Verified by execution rather than by reading: a privileged container ran
docker_dataroot.sh with real loop devices and mkfs through ext4 -> ext4 -> btrfs
-> ext4, confirming the recognition, the replacement in both directions and that
no loop leaks across it; a second harness with a recording zpool stub confirmed
that a btrfs pick does reclaim a stale pool, that no pool means no destroy, and
that a host without /dev/zfs still completes an ext4 pick without touching zpool.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dockerd's zfs graphdriver resolves `zfs` on PATH and opens /dev/zfs before it
accepts a data root (moby daemon/graphdriver/zfs/zfs.go:54,59), so the userland
has to sit in the image, not on the host. Only the apt images carried it, which
is why resolve.sh's userland_for() kept zfs out of every entry containing arch,
centos or fedora - that is every automatic swarm sweep, so the zfs data root and
the zfs branch of the backup launcher were never once exercised. Run
30781723102 shows it: "zfs left out: not every distro of this entry carries the
userland".
scripts/install/zfs.sh joins the three existing install scripts in the image
build and takes a different route per distro, each one run in a real container
of the matching base image before landing here:
debian trixie 2.3.2 zfsutils-linux, contrib enabled first
ubuntu 26.04 2.4.1 zfsutils-linux
arch 2.4.3 the AUR PKGBUILD, which configures --with-config=user
fedora 44 2.4.3 zfsonlinux repo plus a dummy zfs-kmod provider
centos 10 2.2.10 the same, from the epel path
The dummy is what keeps the RPM route usable: the `zfs` package carries an
ungated `Requires: %{name}-kmod`, so a plain `dnf install zfs` resolves through
zfs-dkms and installs 389 extra packages and 1.44 GB of kernel and toolchain to
build a module against a kernel the container never boots. Providing zfs-kmod
from an empty package keeps the same install at 60 packages with no kernel
package at all, and the transaction is checked afterwards so a regression fails
the build instead of bloating the image. The repository is removed again once
the userland is in, because the dummy pins one version and an upgrade would
otherwise re-resolve the kmod at deploy time.
Version drift against the host module is not a concern: zpool create enables a
feature only when the loaded module reports it through /sys/module/zfs/features
.pool (cmd/zpool/zpool_main.c, module/zcommon/zfeature_common.c), and libzfs
compares no versions at all (lib/libzfs/os/linux/libzfs_util_os.c:123-134), so a
newer userland degrades rather than breaking.
With every image carrying it, userland_for() and every_distro_carries() have
nothing left to decide and go. Two defects that only became reachable once zfs
could be drawn are fixed in the same move:
* /dev/zfs is a misc device, so the old lookup in /proc/devices could never
find its major and the node was never created. A node container's /dev is
populated at creation and udevd is masked, so nothing else fills the gap
either: the first bring-up of every job would have declined, blaming a
module that modprobe had just loaded. The minor now comes from /proc/misc.
* Pools live in the kernel the lab nodes share with the runner, while the
teardown only removes containers, and the node hostnames repeat across the
distro sweep. From the second distro on, zpool create hit the previous
iteration's pool. It is destroyed and its loop detached first.
The node image drops its own apt-only zfs branch and instead masks the six boot
units the userland enables by preset, which on a node can only fail: the pool is
created after boot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every lab node got an identical 30 GiB loop device because docker_dataroot.sh
defaulted SIZE to 30G and no caller ever stated it. The swarm manager hosts the
in-cluster registry, ~60 containers, and - by deliberate placement, since /data
is NFS-unsafe - the SeaweedFS object store, whose sizing filter claims 14 GiB
of slots plus a 2 GiB minFreeSpace floor. The idle nfs-server node got the same
30 GiB and used 1% of it.
In run 30859490335 the manager's device stood at 30G/30G (98%), SeaweedFS
crossed its floor at 02:04:45 and marked all 14 volumes unwritable, permanently.
Nextcloud in variant 0 stores objects in that SeaweedFS via OBJECTSTORE_S3_*,
so every authenticated page render performed an object write that failed as a
10-attempt assign ladder - 15 requests, 15 bursts of exactly 10, burst end
matching the nginx 500 to the second. That is the "🐝 web-app-nextcloud 0"
failure: 5 addon specs, 3 attempts each, all dead on the post-login page.
The default is gone: SIZE is now a required argument and a missing value dies
naming the call site. The swarm playbook states it per node - 60G for the
manager, 30G for the rest. 60G covers the observed demand (>=31 GiB: a full
device plus the denied writes) with headroom; the image is sparse, so the
raised ceiling costs only what is actually written, and the runner-side
watchdog at 6 GiB free remains the backstop. The compose and host workflows
state their unchanged 30G explicitly.
Not addressed here: what fills the first 28 GiB (registry + image layers +
volumes composition is unmeasured - the rescue artifact holds exactly one df
snapshot), and the drill-side shadow restore that supplied the last ~1 GiB;
both are separate changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A job whose nodes all declined the filesystem tested none of it, and the table
row saying so reads like an ordinary result. That is how the debian arm lost its
zfs coverage without anyone noticing: docker_dataroot.sh exists to exercise the
snapshot mode of svc-bkp-volume-2-local, which engages only on btrfs and zfs,
and on that arm it has been engaging on neither.
The summary now carries an explicit warning when no node applied the requested
filesystem and at least one declined it. Partial coverage is coverage and stays
unflagged; a job with no verdicts at all still exits early.
Deliberately a finding, not a failure. Stating a filesystem instead of `auto`
would set REQUIRED=true in resolve.sh and turn a decline into a hard error, but
that would paint the debian arm red for a cause that is understood and not yet
fixed - the node image has no zfsutils candidate without the package cache. The
goal was never to enforce the pick, only to stop losing it silently.
Exercised against three real job logs from run 30781723102: the debian arm
(5 nodes, all declined) produces the warning; the ubuntu arm (10 nodes, zfs
applied) and an ext4 job do not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The script stops the daemon at :157-158 and starts it again only on the success
path at :220. decline() exits 0 and set -e exits non-zero, and both walk past
that start, so any other outcome leaves the host without docker. One of the
three callers is test-deploy-host.yml:171, where the script runs on the GitHub
runner itself rather than in a lab node container.
An EXIT trap rather than a restart inside decline(), because decline is not the
only affected exit: arm_autoclear's exit 1 and every set -e abort at mkdir,
truncate, zfs create, mkfs, mount, btrfs subvolume create and the daemon.json
writer sit in the same window. The trap is armed only after the stop, and
DOCKER_STOPPED=false after a successful start makes it a no-op on the green
path. A failed restart is reported rather than swallowed, so it cannot hide the
failure that led there.
This hazard predates 02dc003bb: the loop-device decline added in the commit
before it has the same property. 02dc003bb widened it by putting a second
decline behind the stop.
Verified with a stubbed-PATH harness driving the real script: after a failed
zpool create the call log contains `systemctl start docker`, and it comes after
`losetup -d`, so the teardown still precedes the restart. REQUIRED=true still
exits 1, the success path is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three places where a data-root verdict was produced and then lost.
report.sh matched `requested=[a-z]+`, which never matches `ext4`. Against the
real deploy log of a run that recorded five ext4 verdicts, the old pattern
yields 0 matches and the summary table was silently empty of them; the fixed
pattern aggregates the same five into one row.
docker_dataroot.sh wrote the reason only to GITHUB_STEP_SUMMARY, which is unset
inside the lab node containers - so on exactly the machines that decline, the
reason went nowhere. It now rides on the machine-readable report line too. This
is what leaves the mariadb drill's decline reason unrecoverable from run
30781723102.
resolve.sh's exclusions went to stderr only, i.e. into the deploy log rather
than the job summary.
The reason is emitted as a list below the table, not as a column: reasons embed
per-node data (the pool name carries the hostname, plus the loop device), so
folding them into the `sort | uniq -c` key would split one five-node row into
five one-node rows exactly in the failure case the reason exists to explain.
`reason=[^']*` keeps ansible's closing quote out of the value, and `|` is
replaced so a reason cannot break the markdown table.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The script promises at :12-13 that a host which cannot deliver the stated
filesystem declines and reports when REQUIRED is false. The zfs branch did not:
`zpool create` stood unguarded, so a random pick that the host could not satisfy
exited 1. In run 30781723102 that killed the swarm lab bring-up on all five
nodes of the nested svc-db-postgres drill (compose/swarm/playbook.yml:84) and
cascaded through swarm-zombie to a red Workspace (debian:bookworm) job.
The pre-flight could not have caught it: it only tested that /dev/zfs exists as
a character device, and on those nodes it did while the module was unusable.
Guarding the real operation instead removes the predictor entirely - whatever
makes zfs undeliverable now routes through decline, and REQUIRED=true still
fails hard there. The teardown is required because a loop device is already
attached to the image at that point and decline exits 0.
This does NOT fix why the module is unusable on debian nodes. That is
pre-existing and was already silent: the mariadb drill in the same job declined
zfs on all five nodes and stayed green, while ubuntu applied zfs on ten. The
change makes the two paths consistent; it does not restore the lost coverage.
To make the loss diagnosable rather than invisible, modprobe's output is no
longer discarded into `|| true` but reported and attached to the decline reason,
and verdict() now records the reason in the step summary, folded to one line so
the markdown list item survives a multi-line stderr.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three conflicts, each resolved by combining both sides rather than picking one.
compose.yml passes both env vars through to the container: swarm-nfs adds
INFINITO_SWARM_MGR_IP, this branch adds INFINITO_TOR_SOCKS_PORT, and neither
substitutes for the other.
daemon.json.j2 takes the data-root block from swarm-nfs and keeps HEAD's dns
key. The incoming side writes networks.internet.dns directly; lookup(
'container_dns') already returns that value as its clearnet entry and prepends
the docker0 bridge on a Tor node in compose mode, so the incoming form is a
strict subset and dropping it loses nothing.
The WordPress CA-trust addon spec gains skipUnlessServiceEnabled("sso") from
swarm-nfs alongside HEAD's gotoOnion and resolveTimeout, and asserts on the
Plugin Name the mu-plugin header declares - the mustuse screen renders that,
not the filename - with resolveTimeout applied to the literal the incoming
side left bare.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
losetup -f prints the device with a trailing (lost) marker when its node is missing, and both claim sites fed the whole string to mknod. The recovery path that materialises /dev/loop8 and beyond therefore never worked, which is why the pool appeared to stop at the eight preallocated nodes even though max_loop is 0. Strip the marker before the node is built.
Nothing released the data-root loop a node claims, so each distro round leaked four of them: across 140 job logs leg 1 always applies 5 and declines 0, leg 2 applies 3 and declines 2, leg 3 declines 4. The declining nodes fall back to overlayfs, which disables the btrfs snapshot mode the script exists to exercise, and the DR drill then finds no device at all.
Detach right after the mount instead. The mount holds the opener, so the kernel arms autoclear rather than detaching, and the loop plus its sparse image are reclaimed when the node container takes its mount namespace down. Verified in a privileged container: the mount survives the detach, AUTOCLEAR reads 1, the filesystem stays writable, btrfs subvolume create still works, and the loop is released on umount.
Drop the teardown block added earlier: it filtered on (lost), which marks a missing device node rather than a missing backing file, so it selected exactly the devices losetup -d cannot detach, ran unprivileged, and printed success unconditionally. Restore the (deleted) filter in the drill. Require mountpoint up front so a missing tool declines instead of killing the node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
grep exits 1 when it matches nothing, and under set -euo pipefail the command substitution killed the script before the '[ -n VERDICTS ] || exit 0' line beneath it could express that empty is fine -- which the file header already promises. Three jobs of run 30699990912 were red for this alone: web-app-matrix 2, web-app-penpot 0 and the nested act drill in test-workspace, all with their deploys intact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The early exit at already-prepared never touched pool properties, so a pool created before 9c0e1efdc would keep acltype=off and setfacl would keep failing on a reused runner. Guard on the pool existing under its current name: e816fc1c9 made that name per-host, so a host carrying an old infinito_ci falls through untouched rather than aborting under set -e.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ZFS pools live in the kernel's namespace, which the five lab node containers share, and compose/swarm/playbook.yml runs this script on hosts: all. Each created the same 'infinito_ci', so the first would win and the other four would die on 'pool already exists' under set -e. Only resolve.sh dropping zfs for arch/centos/fedora nodes has kept that out of sight; a dispatch limited to debian or ubuntu exposes it.
The mountpoint needs no suffix: mount namespaces are not shared, only the pool name is global.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OpenZFS defaults acltype=off on Linux, so setfacl on anything under the data root returns 'Operation not supported' -- which killed three compose jobs of run 30695207248 the moment the filesystem matrix drew zfs. Set it at pool creation; -O properties are inherited by the docker dataset, so no second edit is needed. xattr=sa is the companion OpenZFS recommends alongside posixacl and it lowers the space cost.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RUNNER_POOL and NODE_POOL were assertions. With a distro list of just debian
and ubuntu the node pool offered zfs, because the table records that those
distros ship the zfs userland - but the module comes from the runner's kernel,
which carries none. One run in three would then have drawn zfs and either
declined silently, testing ext4 while the summary claimed zfs, or hard-failed on
all five nodes with nothing actually broken.
Ask the kernel instead: /proc/filesystems for what is loaded, modinfo for what
could be. Both answer unprivileged, which resolve.sh needs since it runs without
sudo. The modinfo half carries weight rather than being belt-and-braces - xfs is
absent from /proc/filesystems on this host while modinfo finds the module, so
without it the pool would shrink to whatever happens to be loaded.
The distro table stays above it for the other question: which userland a
container can install. Two layers because they are two things, and only the
kernel one cannot be fixed by a package.
Every exclusion now names its reason on stderr and distinguishes a missing tool
from a missing capability; a filesystem used to drop out of the pool in silence.
For the full five-distro sweep the pool is unchanged: ext4 btrfs, measured
before and after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The distro table pinned centos to ext4, and since the node pool is the
intersection across every distro in a run, the full five-distro sweep always
landed on ext4. btrfs and zfs are what engage the backup snapshot mode, so that
path was never exercised in swarm.
The premise held for a centos system and not for a centos container: Red Hat
dropped btrfs from its own kernel, but a node container runs the host kernel,
and btrfs-progs is merely absent from the base repos - EPEL carries it. Measured
in a centos:stream9 container on this host: mkfs.btrfs, mount, subvolume create,
snapshot -r, and inode 256 at the subvolume root, which is what baudolo checks.
install_packages is the load-bearing part. docker_dataroot.sh ships its own
installer and runs before the ansible package layer, so meta/packages.yml never
reaches it; without the epel fallback there, opening the pool would have let a
centos node be handed btrfs it cannot deliver, decline silently, and leave the
cluster running four btrfs nodes and one ext4 - the very inconsistency the
intersection exists to prevent.
Verified by running the script itself, not extracted parts: it reports
requested=btrfs effective=btrfs on centos.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The data-root task runs on all five swarm nodes at once, and they are
containers on one host kernel sharing its loop devices. losetup -f only names
the next free one; by the time the following losetup attached it, a sibling had
taken it - 'failed to set up loop device: Device or resource busy'. Claim with
--find --show, which picks and attaches in one call, and retry with a short
pause when a sibling wins anyway.
The step runs before anything is pulled, so on some nodes docker is not
installed yet and the closing 'systemctl start docker' died with rc=5, taking
the whole task with it under set -e. The two systemctl stop calls above it had
tolerated that all along. Start only when the unit exists, and report
applied-pending-docker instead of applied so the job summary stops claiming a
data root that no daemon has picked up yet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve conflicts, resolved by intent rather than by side.
Six shared one shape: the incoming side adds the new rescue_index.sh call but
drops this branch's `# nocheck: shell-or-true` markers. The markers carry a
lint contract here, so the incoming structure was taken and the markers put
back — dedicated.sh, guide/one.sh, guide/host_deploy.sh, swarm/routine/00_one.sh
and trigger_units.sh, where the incoming version additionally writes the full
journal to a file instead of only echoing a tail.
web-app-matrix keeps HEAD's _synapse_extra_args accumulator (the superset: CA
mount plus the onion case) but adopts the incoming improvement inside it — the
inlined regex_replace gives way to lookup('subnet_gateway').
handlers/__init__.py drops rescue_diagnostics_dir from the old import group
(the incoming side moved it under .infinito.rescue, keeping both would import
it twice) and keeps tor_socks_port, a HEAD-only handler ORDERED_HANDLERS still
needs.
svc-bkp-volume-2-local takes the incoming side throughout: both branches landed
the same feature, but swarm-nfs evolved it — the knob moved from the global
STORAGE_BACKUP_SNAPSHOT_MODE to services.volume-2-local.snapshot_mode, plus the
subvolume helper, 04_snapshot_subject.yml, meta/packages.yml and baudolo 3.2.2.
Three files that left behind were removed: files/baudolo_snapshot.py and the two
worktree tests at their pre-refactor paths, all byte-identical to their moved
counterparts.
Six follow-on failures the merge produced, fixed at the root:
- STORAGE_BACKUP_SNAPSHOT_MODE lost its last reader with the config-lookup
design and is gone from group_vars.
- utils/github/variant_bundle_limits.py sat next to variant_bundles.py, which
the prefix lint forbids; the pair now lives in utils/github/variant/ as
bundles.py and bundle_limits.py, with the -m entry point, the ruff
per-file-ignore, the design-doc link and the mirroring test path updated.
- The incoming WordPress addon spec navigated with a raw page.goto and carried
three bare timeout literals; both route through gotoOnion and resolveTimeout
now, which is what makes navigation survive onion transport.
- Six unmarked `|| true` got a concrete reason each: docker.socket is absent on
distros that ship docker without it, the loop device is already detached when
the probe failed before attaching, and modprobe zfs may fail because the
/dev/zfs check decides.
make quality green across all four targets. The cold-domains budget test that
had to be waived here passes on its own now — 19d56f07e raises it to 100s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make lint-action was red repo-wide: a type:choice input may not carry an
unnamed option, and entry-manual.yml used the empty string to mean 'pick one
at random'. The lint gated nothing while it failed on every run.
Call the case 'auto', the spelling the workspace and instructions inputs in
the same file already use, and resolve it in resolve.sh - the one place that
decides random-versus-stated. The pass-through stays verbatim, so the six
downstream declarations keep seeing '' when a caller says nothing and 'auto'
when a human deliberately chose random. Both resolve the same way; the origin
stays distinguishable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-matrix selection never picked anything but ext4. candidates()
intersected what every distribution in the entry supports, and with the list the
orchestrator passes - arch debian ubuntu fedora centos - that leaves ext4 alone:
centos stream has no btrfs, zfs is debian/ubuntu only. btrfs was reachable only
by stating it in the manual input.
The intersection is right for swarm and wrong for the other two. distros.sh
runs 00_one.sh per distribution and each round builds its own lab, so a swarm
pick must hold in every round's nodes. compose and host carve the data root on
the runner itself, whose OS is fixed; there DISTROS names the application images
and has nothing to do with the filesystem underneath docker.
A fourth argument says which, rather than parsing it out of the label, and
guards with `:?` so a forgotten call site fails loudly. The runner pool omits
zfs: the runner-image build installs neither the zfs userland nor
linux-modules-extra, where the Ubuntu azure kernel keeps zfs.ko, so a drawn zfs
would decline on every run. A stated one still overrides the pool and is marked
required, so `filesystem: zfs` fails loudly instead of quietly falling back.
The test executes the resolver rather than reading it, and pins the relation
that breaks silently: the keys it writes to GITHUB_ENV are the keys the three
workflows read. A typo on either side hands docker_dataroot.sh an empty string,
which reports "no filesystem stated" and exits 0 - the feature switched off
under a green run. Pool contents are policy and are not asserted; the per-distro
expectation is derived from the resolver's own answers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The summary named the filesystem that was picked, never the one the run got.
A host that declines - no root, missing tooling, no /dev/zfs - still deploys,
and nothing recorded on what, so a red run could not be reproduced.
docker_dataroot.sh now ends every path with a verdict that measures the data
root with `stat -f -c %T` and reports applied, declined, already-prepared,
unchanged or required-but-unavailable. Compose and host write it straight into
the step summary; swarm runs the script inside each node, where that variable
does not exist, so the playbook lifts the verdict line into the deploy log and
report.sh aggregates it afterwards.
`hostname` is resolved with a fallback: it is not installed by default on the
RedHat family, and under `set -euo pipefail` a bare command substitution would
have turned the decline path - which exits 0 - into a hard failure.
test-deploy-host.yml gains the `filesystem` input. single-node has been passing
it since the per-matrix selection landed, and a reusable workflow that receives
an undeclared input fails the whole run with a startup_failure whose only trace
is a banner on the run page. The single-node line was skipped in the run that
would have shown it. A lint now pairs every local `uses:` with the callee's
declared inputs, and also demands that required inputs without a default are
supplied.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The backup snapshot mode only engages on btrfs or zfs, so on CI - where the
data root is whatever the runner or the node image happens to provide - it
was never exercised at all. Nothing proved the snapshot path beyond unit
tests and a synthetic probe.
`entry-manual` gains a `filesystem` choice that reaches both deploy modes.
Empty means each matrix entry draws its own, so coverage spreads over runs
instead of pinning one path forever; the pick and its origin go to the step
summary, because reproducing a red run means re-stating it verbatim. The
draw is restricted to what every distribution of that entry can deliver -
centos stream has no btrfs at all, and zfs is a third-party repository
outside debian and ubuntu - so a run never dies on a packaging gap. A
stated filesystem skips that restriction: it is an instruction, and a host
that cannot deliver it fails loudly rather than deploying on something
else.
docker_dataroot.sh formats a sparse loop image, mounts it at /mnt/docker-fs
and carves the data root inside it, then points the daemon there through
daemon.json - merged, not overwritten, so the registry-cache entry
survives. It runs on the runner for compose and inside each node for swarm,
before anything is pulled, and installs the filesystem tooling it needs.
Two container facts it has to work around: a privileged container's /dev is
a fresh tmpfs, so neither /dev/loopN nor /dev/zfs exists there and both are
mknod'ed from /proc/devices before use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
