The script promises at :12-13 that a host which cannot deliver the stated
filesystem declines and reports when REQUIRED is false. The zfs branch did not:
`zpool create` stood unguarded, so a random pick that the host could not satisfy
exited 1. In run 30781723102 that killed the swarm lab bring-up on all five
nodes of the nested svc-db-postgres drill (compose/swarm/playbook.yml:84) and
cascaded through swarm-zombie to a red Workspace (debian:bookworm) job.
The pre-flight could not have caught it: it only tested that /dev/zfs exists as
a character device, and on those nodes it did while the module was unusable.
Guarding the real operation instead removes the predictor entirely - whatever
makes zfs undeliverable now routes through decline, and REQUIRED=true still
fails hard there. The teardown is required because a loop device is already
attached to the image at that point and decline exits 0.
This does NOT fix why the module is unusable on debian nodes. That is
pre-existing and was already silent: the mariadb drill in the same job declined
zfs on all five nodes and stayed green, while ubuntu applied zfs on ten. The
change makes the two paths consistent; it does not restore the lost coverage.
To make the loss diagnosable rather than invisible, modprobe's output is no
longer discarded into `|| true` but reported and attached to the decline reason,
and verdict() now records the reason in the step summary, folded to one line so
the markdown list item survives a multi-line stderr.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three conflicts, each resolved by combining both sides rather than picking one.
compose.yml passes both env vars through to the container: swarm-nfs adds
INFINITO_SWARM_MGR_IP, this branch adds INFINITO_TOR_SOCKS_PORT, and neither
substitutes for the other.
daemon.json.j2 takes the data-root block from swarm-nfs and keeps HEAD's dns
key. The incoming side writes networks.internet.dns directly; lookup(
'container_dns') already returns that value as its clearnet entry and prepends
the docker0 bridge on a Tor node in compose mode, so the incoming form is a
strict subset and dropping it loses nothing.
The WordPress CA-trust addon spec gains skipUnlessServiceEnabled("sso") from
swarm-nfs alongside HEAD's gotoOnion and resolveTimeout, and asserts on the
Plugin Name the mu-plugin header declares - the mustuse screen renders that,
not the filename - with resolveTimeout applied to the literal the incoming
side left bare.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The routine flowchart marked a terminator-driven remote-2-local start on bkp-01, failing for lack of the ssh trust the drill installs later. Both halves are wrong: the lab deploys the app stack and the backup host with --skip-backup, so MODE_BACKUP is false, svc-bkp-remote-2-local never sets system_service_force_flush_final, and the unit never enters system_service_run_list.
The drill is the only caller that starts the pull. The unit journal of the failed run holds exactly one invocation, at 20:58:30 - one line after 01_ssh_trust.sh installed the key, and that first rsync moved 5.8 GB. A terminator-started pull would have run at ~20:55:4x, before the key existed.
The box also loses its trigger colouring, since nothing fires there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
losetup -f prints the device with a trailing (lost) marker when its node is missing, and both claim sites fed the whole string to mknod. The recovery path that materialises /dev/loop8 and beyond therefore never worked, which is why the pool appeared to stop at the eight preallocated nodes even though max_loop is 0. Strip the marker before the node is built.
Nothing released the data-root loop a node claims, so each distro round leaked four of them: across 140 job logs leg 1 always applies 5 and declines 0, leg 2 applies 3 and declines 2, leg 3 declines 4. The declining nodes fall back to overlayfs, which disables the btrfs snapshot mode the script exists to exercise, and the DR drill then finds no device at all.
Detach right after the mount instead. The mount holds the opener, so the kernel arms autoclear rather than detaching, and the loop plus its sparse image are reclaimed when the node container takes its mount namespace down. Verified in a privileged container: the mount survives the detach, AUTOCLEAR reads 1, the filesystem stays writable, btrfs subvolume create still works, and the loop is released on umount.
Drop the teardown block added earlier: it filtered on (lost), which marks a missing device node rather than a missing backing file, so it selected exactly the devices losetup -d cannot detach, ran unprivileged, and printed success unconditionally. Restore the (deleted) filter in the drill. Require mountpoint up front so a missing tool declines instead of killing the node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role runs ganesha at -N NIV_EVENT, and roughly eighteen init steps between the CLIENT commits and the listener log below that level, so the silence in run 30699990912 marked the edge of visibility rather than the fault. Raise INIT and DISPATCH under MODE_DEBUG: nfs_Init_svc(), which binds 2049, logs under COMPONENT_DISPATCH, so INIT alone would have missed exactly the step in question.
Add a userspace backtrace beside the kernel stack dump, which only ever says futex_do_wait. eu-stack names the function holding the lock; elfutils goes into the lab image rather than the role, since a production NFS server should not carry a debugger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
grep exits 1 when it matches nothing, and under set -euo pipefail the command substitution killed the script before the '[ -n VERDICTS ] || exit 0' line beneath it could express that empty is fine -- which the file header already promises. Three jobs of run 30699990912 were red for this alone: web-app-matrix 2, web-app-penpot 0 and the nested act drill in test-workspace, all with their deploys intact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every node runs docker_dataroot.sh, which attaches a loop for its data root, and nothing ever detached them. Run 30699990912 shows the consequence: all eight kernel loop devices bound to docker-fs.img in two inode groups, i.e. two rounds' worth, leaving none for the DR drill's LUKS image.
Release the orphaned ones after the nodes are gone -- only then is a loop unambiguously ours to reclaim, rather than a concurrent run's. Match (lost) as well as (deleted): a backing file inside a removed container's overlay becomes unfindable rather than merely unlinked, which is why the existing cleanup in 02_luks_device.sh never fired.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The early exit at already-prepared never touched pool properties, so a pool created before 9c0e1efdc would keep acltype=off and setfacl would keep failing on a reused runner. Guard on the pool existing under its current name: e816fc1c9 made that name per-host, so a host carrying an old infinito_ci falls through untouched rather than aborting under set -e.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
compose names the swarm-test volume after its project, so every cluster
left a <cluster>_nfs-export behind that no teardown path touched. At
roughly 5GB each they filled the host disk after ~30 runs, which showed
up as unexplained playwright timeouts once writes started stalling.
Remove it per cluster in teardown, per worktree subnet in lab-reclaim,
and host-wide in swarm-clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ZFS pools live in the kernel's namespace, which the five lab node containers share, and compose/swarm/playbook.yml runs this script on hosts: all. Each created the same 'infinito_ci', so the first would win and the other four would die on 'pool already exists' under set -e. Only resolve.sh dropping zfs for arch/centos/fedora nodes has kept that out of sight; a dispatch limited to debian or ubuntu exposes it.
The mountpoint needs no suffix: mount namespaces are not shared, only the pool name is global.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OpenZFS defaults acltype=off on Linux, so setfacl on anything under the data root returns 'Operation not supported' -- which killed three compose jobs of run 30695207248 the moment the filesystem matrix drew zfs. Set it at pool creation; -O properties are inherited by the docker dataset, so no second edit is needed. xattr=sa is the companion OpenZFS recommends alongside posixacl and it lowers the space cost.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill died with cryptsetup's 'loop device with autoclear flag is required', which names neither the resource nor the reason. Claim a loop for the image and release it again first, creating the /dev node when the container's tmpfs carries none, so cryptsetup finds what it needs and a genuine shortage stops the drill with a message that says so.
The cryptsetup calls themselves are unchanged and keep owning their own autoclear device, so nothing lingers once the mapper closes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill's cryptsetup step died claiming a loop device and the artifact carried nothing to say why: no losetup -a, no /dev listing, and the backup host appears in no other collector loop. Record both views, what the kernel has attached and which nodes the container's /dev carries, so an exhausted pool and a missing device node are told apart at a glance.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The parenthetical claimed the capture names the address and errno of a failed bind. dnsmasq logs that at startup only, and a startup bind failure is fatal, so the half-bound state seen in the artifact cannot originate there. The dynamic bind path is measurably silent on success and untested on failure. The neighbouring markers in this collector carry no description either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The socket table already shows which listening address is missing, but nothing recorded why. A build that cannot resolve dies on the node whose dnsmasq never bound the advertised resolver, and the journal is the only place naming the address and errno. Captured whole rather than tailed, because a failed bind is logged at startup and the neighbouring captures keep only the tail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RUNNER_POOL and NODE_POOL were assertions. With a distro list of just debian
and ubuntu the node pool offered zfs, because the table records that those
distros ship the zfs userland - but the module comes from the runner's kernel,
which carries none. One run in three would then have drawn zfs and either
declined silently, testing ext4 while the summary claimed zfs, or hard-failed on
all five nodes with nothing actually broken.
Ask the kernel instead: /proc/filesystems for what is loaded, modinfo for what
could be. Both answer unprivileged, which resolve.sh needs since it runs without
sudo. The modinfo half carries weight rather than being belt-and-braces - xfs is
absent from /proc/filesystems on this host while modinfo finds the module, so
without it the pool would shrink to whatever happens to be loaded.
The distro table stays above it for the other question: which userland a
container can install. Two layers because they are two things, and only the
kernel one cannot be fixed by a package.
Every exclusion now names its reason on stderr and distinguishes a missing tool
from a missing capability; a filesystem used to drop out of the pool in silence.
For the full five-distro sweep the pool is unchanged: ext4 btrfs, measured
before and after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An opencloud snapshot carried 174 entries, and every line repeated the whole
path - the deepest ran about 150 characters, most of it the same prefix as the
line above. Print the basename at its depth instead, with a glyph per kind:
folder, json, journal or log, plain text.
The separator is swapped for \001 across the sort. Without it a directory and
its children come apart: '/' outranks '.', so a sibling file sorts between the
two and reads as though it lived inside. The swap also makes the order
independent of the locale, which was deciding it before.
The existing cut-short test looked for 'logs/container.log', which no longer
appears, so it now checks the basename; a fourth case pins the structure rather
than the wording - a child is indented deeper than its folder and siblings sit
level.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The distro table pinned centos to ext4, and since the node pool is the
intersection across every distro in a run, the full five-distro sweep always
landed on ext4. btrfs and zfs are what engage the backup snapshot mode, so that
path was never exercised in swarm.
The premise held for a centos system and not for a centos container: Red Hat
dropped btrfs from its own kernel, but a node container runs the host kernel,
and btrfs-progs is merely absent from the base repos - EPEL carries it. Measured
in a centos:stream9 container on this host: mkfs.btrfs, mount, subvolume create,
snapshot -r, and inode 256 at the subvolume root, which is what baudolo checks.
install_packages is the load-bearing part. docker_dataroot.sh ships its own
installer and runs before the ansible package layer, so meta/packages.yml never
reaches it; without the epel fallback there, opening the pool would have let a
centos node be handed btrfs it cannot deliver, decline silently, and leave the
cluster running four btrfs nodes and one ext4 - the very inconsistency the
intersection exists to prevent.
Verified by running the script itself, not extracted parts: it reports
requested=btrfs effective=btrfs on centos.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The data-root task runs on all five swarm nodes at once, and they are
containers on one host kernel sharing its loop devices. losetup -f only names
the next free one; by the time the following losetup attached it, a sibling had
taken it - 'failed to set up loop device: Device or resource busy'. Claim with
--find --show, which picks and attaches in one call, and retry with a short
pause when a sibling wins anyway.
The step runs before anything is pulled, so on some nodes docker is not
installed yet and the closing 'systemctl start docker' died with rc=5, taking
the whole task with it under set -e. The two systemctl stop calls above it had
tolerated that all along. Start only when the unit exists, and report
applied-pending-docker instead of applied so the job summary stops claiming a
data root that no daemon has picked up yet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve conflicts, resolved by intent rather than by side.
Six shared one shape: the incoming side adds the new rescue_index.sh call but
drops this branch's `# nocheck: shell-or-true` markers. The markers carry a
lint contract here, so the incoming structure was taken and the markers put
back — dedicated.sh, guide/one.sh, guide/host_deploy.sh, swarm/routine/00_one.sh
and trigger_units.sh, where the incoming version additionally writes the full
journal to a file instead of only echoing a tail.
web-app-matrix keeps HEAD's _synapse_extra_args accumulator (the superset: CA
mount plus the onion case) but adopts the incoming improvement inside it — the
inlined regex_replace gives way to lookup('subnet_gateway').
handlers/__init__.py drops rescue_diagnostics_dir from the old import group
(the incoming side moved it under .infinito.rescue, keeping both would import
it twice) and keeps tor_socks_port, a HEAD-only handler ORDERED_HANDLERS still
needs.
svc-bkp-volume-2-local takes the incoming side throughout: both branches landed
the same feature, but swarm-nfs evolved it — the knob moved from the global
STORAGE_BACKUP_SNAPSHOT_MODE to services.volume-2-local.snapshot_mode, plus the
subvolume helper, 04_snapshot_subject.yml, meta/packages.yml and baudolo 3.2.2.
Three files that left behind were removed: files/baudolo_snapshot.py and the two
worktree tests at their pre-refactor paths, all byte-identical to their moved
counterparts.
Six follow-on failures the merge produced, fixed at the root:
- STORAGE_BACKUP_SNAPSHOT_MODE lost its last reader with the config-lookup
design and is gone from group_vars.
- utils/github/variant_bundle_limits.py sat next to variant_bundles.py, which
the prefix lint forbids; the pair now lives in utils/github/variant/ as
bundles.py and bundle_limits.py, with the -m entry point, the ruff
per-file-ignore, the design-doc link and the mirroring test path updated.
- The incoming WordPress addon spec navigated with a raw page.goto and carried
three bare timeout literals; both route through gotoOnion and resolveTimeout
now, which is what makes navigation survive onion transport.
- Six unmarked `|| true` got a concrete reason each: docker.socket is absent on
distros that ship docker without it, the loop device is already detached when
the probe failed before attaching, and modprobe zfs may fail because the
/dev/zfs check decides.
make quality green across all four targets. The cold-domains budget test that
had to be waived here passes on its own now — 19d56f07e raises it to 100s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
find and du write what they already collected and only then return nonzero, so
the guards added in ce1c5b5fe were throwing away the evidence the index exists
to report. One vanished entry mid-walk blanked the whole listing: '0 file(s)',
no paths, on a tree that was mostly readable. The rescue tree is indexed while
a failed run is still tearing down, so that is the normal case, not the corner
one.
Stop resetting LISTING, fall back on SIZE only when du produced nothing, and
turn the early exit into a warning so the partial listing still prints.
The existing test could not see this: its stub find emitted nothing, so it only
covered a total failure. The new case stubs a find that prints rows and then
fails, and it fails against ce1c5b5fe.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make lint-action was red repo-wide: a type:choice input may not carry an
unnamed option, and entry-manual.yml used the empty string to mean 'pick one
at random'. The lint gated nothing while it failed on every run.
Call the case 'auto', the spelling the workspace and instructions inputs in
the same file already use, and resolve it in resolve.sh - the one place that
decides random-versus-stated. The pass-through stays verbatim, so the six
downstream declarations keep seeing '' when a caller says nothing and 'auto'
when a human deliberately chose random. Both resolve the same way; the origin
stays distinguishable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rescue artifact is a tarball of whatever the collectors managed to grab, so
reading a failed run meant downloading it to find out which files exist at all.
Run rescue_index.sh right after each collection instead, so the job log names
the tree the artifact contains.
All four sites call it without a fallback, which is what the preceding commit's
exit-status contract is for. host_deploy.sh and one.sh hoist the rescue path
into a variable first - it is now needed twice, once for the collector's
INFINITO_RESCUE_DIAGNOSTICS_DIR and once for the index.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every caller runs this in a cleanup path: three shell teardown handlers and a
GitHub step under bash -e, none of them with a fallback. A nonzero rc therefore
aborts the caller's remaining teardown or adds a second red step on top of the
failure being rescued. Until now that could not happen only because each
internal failure path happened to be caught - a property that dies with the
next edit. A terminal 'trap exit 0' states the guarantee once and makes it
independent of the enumeration.
The failure is not hypothetical: a find that rejects -printf (BusyBox) makes
the old script exit 1, reproduced with a stub.
Collapses three walks of the same tree into one, and counts files with awk
rather than grep -c, which answers 1 on zero matches and takes the assignment
down with it under pipefail. A walk that fails now says so - it used to print
'0 file(s)', which reads exactly like an empty tree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The per-matrix selection never picked anything but ext4. candidates()
intersected what every distribution in the entry supports, and with the list the
orchestrator passes - arch debian ubuntu fedora centos - that leaves ext4 alone:
centos stream has no btrfs, zfs is debian/ubuntu only. btrfs was reachable only
by stating it in the manual input.
The intersection is right for swarm and wrong for the other two. distros.sh
runs 00_one.sh per distribution and each round builds its own lab, so a swarm
pick must hold in every round's nodes. compose and host carve the data root on
the runner itself, whose OS is fixed; there DISTROS names the application images
and has nothing to do with the filesystem underneath docker.
A fourth argument says which, rather than parsing it out of the label, and
guards with `:?` so a forgotten call site fails loudly. The runner pool omits
zfs: the runner-image build installs neither the zfs userland nor
linux-modules-extra, where the Ubuntu azure kernel keeps zfs.ko, so a drawn zfs
would decline on every run. A stated one still overrides the pool and is marked
required, so `filesystem: zfs` fails loudly instead of quietly falling back.
The test executes the resolver rather than reading it, and pins the relation
that breaks silently: the keys it writes to GITHUB_ENV are the keys the three
workflows read. A typo on either side hands docker_dataroot.sh an empty string,
which reports "no filesystem stated" and exits 0 - the feature switched off
under a green run. Pool contents are policy and are not asserted; the per-distro
expectation is derived from the resolver's own answers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The collector carried a shell-side default for INFINITO_RESCUE_DIAGNOSTICS_DIR,
which the SPOT lint forbids because a second source drifts unnoticed - and this
one already had: the fallback spelled /tmp/rescue-diagnostics, the value of
INFINITO_RESCUE_DIAGNOSTICS_BASE, while the variable it stood in for resolves to
/tmp/infinito-rescue-diagnostics. Had it ever been used, the collector would have
written where no artifact collects.
A `:?` guard replaces it, matching the six the DR drill already carries. The
only caller sets the variable explicitly for both collectors, and _context.sh
sources the env layer regardless, so the guard should never fire.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The nextcloud drill watched /dev/root go from 40G used to 124G used with 21G
free on a 145G disk shared by five nested nodes, the NFS export, both backup
repos and a 14.2GB pull. Four staging trees survived the whole run - grep for
`rm -rf` found nothing after the secrets leg - so the drill kept every copy it
made until the containers went away.
DR_RESTORE_STAGE, DR_VOL_STAGE, DR_SEC_STAGE and RESTORE_ROOT are released
after their last reader, each checked individually against the line that reads
it. All four sit on the success path: `set -e` aborts before them on any
failure, so a drill that fails still hands its evidence to the rescue
collector.
The device step now measures free space before it starts and refuses with both
numbers rather than running into ENOSPC inside a nested systemd unit. The
threshold is USB_SIZE_MB, which the script already computes, so no new constant
enters - and it states the real ledger: the drill holds the pulled tree twice,
once for the device sync and again while the restore root exists alongside the
image. It sits after 02_luks_device.sh because that is where the only stale-loop
reaper runs; a guard placed earlier would abort forever on a leaked loop device
and the reap that would clear it would never happen.
Reading free space is split from evaluating it. Under pipefail a df that
rejects --output fails the command substitution and set -e kills the drill
before the check can speak - which is how the guard managed to produce exactly
the silence it exists to prevent. Driving the body against six df behaviours
(roomy, tight, no header, option unsupported, empty, non-numeric) is what
surfaced that.
This does not address why the observed run failed. It died in step [4/9] with
21G still free, so this guard would not have fired on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three of the six failures triaged today were explainable only after downloading
a rescue artifact, and two not even then. The job log carried "not converged:
docker_docker", a bare "rc=1", and "failed after 3 attempt(s)".
The convergence gate now prints the task rows of a service that will not
converge - but only those whose error field is filled. A stack coming up
normally stays silent, while the one that killed svc-db-postgres would have
shown `error=Head "https://ghcr.io/...": request canceled` in the job log
instead of hiding it in an artifact. awk rather than grep: grep exits 1 when
nothing matches, which under set -e would take the script with it.
The remote pull decodes its exit status. os.system returns rc << 8, so the
message said 5888 where rsync said 23, and every attempt now names its code.
The sleep after the final attempt is gone - five wasted minutes per run.
The matomo poller hands its last 40 lines to stderr on deadline, redacted at
the source, and the failing task interpolates them the way the compose flavor
already does. No no_log: MASK_CREDENTIALS_IN_LOGS is force-set anyway and would
have censored exactly the payload this exists to deliver.
The `-p warning` filter added to trigger_units.sh a few commits ago is removed.
systemd journals a unit's stdout and stderr at info, so that filter would have
swallowed the per-provider attribution the same commit introduced.
stack_ready.sh gets its first tests: replica comparison, the completed-oneshot
branch, and both halves of the new reporting - the error reaches the caller, the
error-free rows do not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The summary named the filesystem that was picked, never the one the run got.
A host that declines - no root, missing tooling, no /dev/zfs - still deploys,
and nothing recorded on what, so a red run could not be reproduced.
docker_dataroot.sh now ends every path with a verdict that measures the data
root with `stat -f -c %T` and reports applied, declined, already-prepared,
unchanged or required-but-unavailable. Compose and host write it straight into
the step summary; swarm runs the script inside each node, where that variable
does not exist, so the playbook lifts the verdict line into the deploy log and
report.sh aggregates it afterwards.
`hostname` is resolved with a fallback: it is not installed by default on the
RedHat family, and under `set -euo pipefail` a bare command substitution would
have turned the decline path - which exits 0 - into a hard failure.
test-deploy-host.yml gains the `filesystem` input. single-node has been passing
it since the per-matrix selection landed, and a reusable workflow that receives
an undeclared input fails the whole run with a startup_failure whose only trace
is a banner on the run page. The single-node line was skipped in the run that
would have shown it. A lint now pairs every local `uses:` with the callee's
declared inputs, and also demands that required inputs without a default are
supplied.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill's remote pull exits 1 after 65 minutes and nothing says why. The
unit loops over two providers and only counts errors, so which one failed is
unrecoverable; the pull runs `rsync -abP`, whose --progress writes one journal
line per file - 77137 of them on a nextcloud pull - and the drill captured
`journalctl -n 60`. The entire diagnostic was therefore rsync filenames.
The unit now names the host it could not pull from. rsync keeps --partial and
trades --progress for --info=stats2, turning the flood into a closing balance.
trigger_units.sh writes the complete journal of a failed unit into the rescue
dump directory the artifact ships as local-dumps, and prints only the lines
from warning upwards.
That directory arrives as an argument instead of a second copy of the paths
SPOT: the drill reads INFINITO_RESCUE_DIAGNOSTICS_DIR once, which the env
handler derives from group_vars/all/05_paths.yml, and hands it to every call
site. A test pins that contract, and it earned its place immediately - it found
two call sites that a grep for the file name cannot see, and both were the
units this change exists for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Of six CI failures triaged today, three were explainable only from downloaded
artifacts and two not even there: the job log carried "not converged:
docker_docker" or a bare "rc=1", while this collector drowned that same log in
compose trees, env listings and ganesha thread stacks.
diagnostics.sh switches its stdout per section with exec, so every topic lands
in its own file under the rescue directory and the log keeps one line. An
unwritable output directory degrades to stdout rather than killing the
collector - a failed mkdir would otherwise abort the script at the first
redirection, which the caller swallows with `|| true`, leaving no diagnostics
at all and no sign of it. For the same reason the collector says so when it
ends up having captured nothing.
00_one.sh hands the same directory to both collectors. The variable was scoped
to the python one, so the shell collector wrote where no artifact picks it up.
The capture grows by exactly what today's investigations went without: per-node
resolv.conf, daemon.json, listening sockets, NAT rules and a resolve probe
against the three hosts that killed builds, plus df and docker system df.
rescue_index.sh runs last and lists what was collected together with the note
that it is downloadable. The index is the one thing worth printing.
A test pins the routing against a stub docker: one file per section, no bleed
between them, a single log line, and the degrade path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The backup snapshot mode only engages on btrfs or zfs, so on CI - where the
data root is whatever the runner or the node image happens to provide - it
was never exercised at all. Nothing proved the snapshot path beyond unit
tests and a synthetic probe.
`entry-manual` gains a `filesystem` choice that reaches both deploy modes.
Empty means each matrix entry draws its own, so coverage spreads over runs
instead of pinning one path forever; the pick and its origin go to the step
summary, because reproducing a red run means re-stating it verbatim. The
draw is restricted to what every distribution of that entry can deliver -
centos stream has no btrfs at all, and zfs is a third-party repository
outside debian and ubuntu - so a run never dies on a packaging gap. A
stated filesystem skips that restriction: it is an instruction, and a host
that cannot deliver it fails loudly rather than deploying on something
else.
docker_dataroot.sh formats a sparse loop image, mounts it at /mnt/docker-fs
and carves the data root inside it, then points the daemon there through
daemon.json - merged, not overwritten, so the registry-cache entry
survives. It runs on the runner for compose and inside each node for swarm,
before anything is pulled, and installs the filesystem tooling it needs.
Two container facts it has to work around: a privileged container's /dev is
a fresh tmpfs, so neither /dev/loopN nor /dev/zfs exists there and both are
mknod'ed from /proc/devices before use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The swarm gate asked whether the drill ran by grepping the job log for
"Matrix-deploy <app>: provision/deploy/e2e/verify per round" — a copy of a
GitHub Actions step display name. d9a4498b3 renamed that step to
"🚀 Drill <apps> per distro: <distros>" and did not touch roundtrip.sh, so at
19d56f07e the literal existed in exactly one place in the tree: the grep itself.
It could no longer match anything, and four of the five workspace legs of run
30586213932 false-failed on deploys that had passed — svc-db-postgres converged,
answered on 5432, completed its DR drill and reported "Job succeeded" three lines
above the failure.
This is the second time the same coupling drifted. 0ca2ed56c is titled "align
roundtrip swarm gate to the live workflow step name" and records that the
previous stale marker made every swarm leg false-fail on a passing deploy.
Nothing cross-checked the literal, so it rots silently after each rename.
The marker now comes from the drill itself, printed after 07_assert_state.sh.
Every step before it runs under set -euo pipefail and the EXIT trap tears down
without printing, so the line is unreachable unless the drill completed for that
app. APP_ID and INFINITO_DISTRO are :?-guarded at the top of the script, so the
marker cannot degrade into an empty string that matches everything. The gate
matches it with grep -qF and names what it checked instead of asserting "empty
matrix" — a diagnosis that was never true and sent this investigation into the
wrong layer.
The gate itself stays. 214672a94 added it because act exits 0 on a zero-leg
matrix, which silently false-passed every local swarm roundtrip.
tests/unit/scripts/test_roundtrip_marker.py pins emitter and gate to the same
literal, pins the marker behind the state assertion, and fails if the old step
name reappears — the cross-check whose absence allowed both drifts.
Verified against the failing run's log: 0 occurrences of the old literal, while
DR drill PASSED, ran=1 skipped=0 and "Job succeeded" prove the drill reached the
point where the new marker is emitted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The compose and host deploy jobs tee the whole deploy stdout into the file
ANSIBLE_LOG_PATH names, and both downstream steps parse that stream. Only
stdout carries the host-side "matrix-deploy: round N/M ... PASS n" markers,
so it is the sole source the per-variant runtime tables can come from.
dedicated.sh's cleanup then copied the container-side ansible log onto that
same path. docker cp unlinks and recreates the destination, so tee kept
writing into an orphaned inode and the visible file became ansible's own
log_path output — whose "<ts> p= u= n= LEVEL| " line prefix defeats the
anchored recap matchers in logparse.py while the unanchored has_sections
substring test still fires. in_roles_recap never flips, every role-time line
is skipped, and the step dies with "no profile_roles entries found" although
the deploy succeeded. Run 30578244959: 14 of 14 compose+host jobs.
Why it looked like a flake: docker cp cannot read a /tmp that is a tmpfs
mounted inside the container, and the Debian image enables systemd's
tmp.mount while CentOS Stream 10 does not. The copy silently failed on
debian (host file survived, jobs green) and landed on centos (host file
destroyed). feature/swarm-nfs never had the block; it returned here through
merge drift from main, where the copy is still load-bearing because those
workflows do not tee.
The copy is dropped: stdout is a strict superset of what it provided, and
the in-container verifier reads the log inside the container. Alongside it:
the swarm workflow's file-exists guard is adopted so an aborted deploy does
not paint the summary step red on top; the log_path prefix moves to
utils/ansible_log.py and is applied in logparse.py, so an ansible run log
parses instead of yielding an empty summary; a lint pins that nothing writes
onto ANSIBLE_LOG_PATH again; and the parser regression test now covers the
recap headers, not just the role rows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make quality rejected the new reclaim script for missing the executable
bit, and the S310 suppression in the Home Assistant token probe for
carrying no reason. A security suppression must state why it is safe, so
the loopback justification goes back on the urlopen call; ruff dropped
the second marker on Request(), where S310 never fires.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lab subnet is pinned per worktree slot while lab network names carry
the cluster id, so a cluster kept from one app blocked the next run of
another with 'invalid pool request: Pool overlaps'. The pre-run teardown
only cleared the same cluster id, which never matched. Matching on the
IPAM subnet instead clears exactly the blocker and leaves a sibling
worktree's cluster alone; swarm-clean stays the host-wide tool.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The CI deploy bootstrap hand-listed two packages and let the ubuntu-latest runner image supply the rest, so a new host-side import surfaced as a red deploy job instead of a failed install. Size [project.dependencies] to what runs outside the project image and move the deployment stack into a deploy extra, then have every install site read that instead of naming packages itself: the image build and make install-python take the deploy extra, the CI host and the mirror and instructions workflows take the base.
Declare cryptography, which utils/manager/value_generator.py imports directly and which only reached the environment transitively through ansible. Drop numpy, which nothing in the repository imports. Guard the boundary with a lint test that blocks every deploy package and asserts the runner-host CLI still builds its parser.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The storage-constrained work put utils.storage.constrained into the host-side import chain of cli.administration.deploy.development, which pulls humanfriendly. The CI dependency line installed only ruamel.yaml and PyYAML, so every compose and host deploy job aborted at parser build time with ModuleNotFoundError.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three conflicts, none resolved by picking a side wholesale:
- Makefile: took the incoming ACT_INPUTS, which routes through SWARM_DISTROS
instead of inlining the same expression a second time.
- test-e2e-playwright/02_run_one.yml: took the incoming rc handling. An absent
rc now counts as failure; the previous default(0) let an aborted Playwright
run pass silently, contradicting the debug message right above it.
- web-app-peertube spec: combined both. Kept gotoOnion and resolveTimeout,
which this branch's lint enforces, and adopted the incoming retry-inside-poll
for the OIDC button plus its networkidle wait.
The topology SPOT read default.env only, so every lab took the same subnet no matter what the generated .env said. Two labs on one host therefore collided with "invalid pool request: Pool overlaps", which is what stopped a worktree lab from running beside the primary checkout's.
Collect the generated .env first, then default.env, and emit each assignment as := so precedence becomes exported environment, then .env, then default.env. default.env stays the fallback because the workflow sources this file before make dotenv has run, which was the original reason for reading it directly.
A line that does not parse as a balanced assignment now aborts naming the line. Sourced as-is it would have become an unconditional assignment and silently defeated the precedence this change exists to establish.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An aborted Playwright run - watchdog SIGTERM, task timeout - returns a result dict with neither stdout nor rc. The reporting task dereferenced both, so it became the fatal error itself: the log ended on "object of type 'dict' has no attribute 'stdout'" while the real cause, a disk-exhaustion abort with rc 75, sat eight thousand lines earlier. Guard all three reads, fall back to msg for stderr because that is what Ansible fills on an abort, and treat an absent rc as a failure so the run is recorded and named by the aggregate check rather than vanishing behind a template error.
The distro sweep's summary also mislabelled the distros it never started: they kept the initial "not reached within the budget" note even when the sweep aborted on a failure with 12214 seconds of budget left, pointing at a budget problem that did not exist. Say what happened instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
act merges its --env-file, which defaults to .env, after the workflow-level env block (pkg/runner/run_context.go:84 at v0.2.89) and interpolates only afterwards, so INFINITO_DISTROS: ${{ inputs.distros }} was replaced by the repository's own five-distro list before the expression was ever evaluated. Every workspace leg therefore swept all five distros instead of its own: 25 distro deploys where 5 were intended, which is what walked the five legs into their 345-minute cap. The drill step's name kept rendering the single distro because it is interpolated from the workflow model rather than from the env map, which is why the logs read as a contradiction.
Pass INFINITO_DISTROS as an --env pair instead, the layer act itself wins with, where --env beats the env-file because the file only fills keys that are absent (cmd/root.go:364-366). The expression was duplicated across ACT_ENV and ACT_INPUTS, so name it once.
Wrapping the assignment for readability required the splitter to tolerate whitespace: inside double quotes the shell removes the backslash-newline but keeps the continuation line's indentation, so every pair after the first arrived with a leading tab.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merge of the swarm-nfs work moved utils/unmount_nfs_mounts.sh to utils/unmount/nfs_mounts.sh and updated teardown.sh and the bootstrap, but clean/all.sh kept the old path. make swarm-clean died at the unmount step with "No such file or directory" before it reached container removal, which is exactly the state that needs cleaning up. The host-state release is not added here because all.sh already unmounts DIR_VAR_LIB itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f660dd6bd wrote the placeholder into a parameter expansion, `${INFINITO_IMAGE:-<none, bootstrap will build it>}`, which tests/lint/filesystem/shell/test_no_infinito_defaults.py rejects: an INFINITO_* default in a shell file is a second source next to default.env that drifts silently, and only the empty form is allowed. It was the sole failure of test-lint.
Split the message into the resolved and the unresolved case instead of suppressing the lint; the output is also no longer self-contradictory, since 'published <none>' could not both be true.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
