The backup snapshot mode only engages on btrfs or zfs, so on CI - where the
data root is whatever the runner or the node image happens to provide - it
was never exercised at all. Nothing proved the snapshot path beyond unit
tests and a synthetic probe.
`entry-manual` gains a `filesystem` choice that reaches both deploy modes.
Empty means each matrix entry draws its own, so coverage spreads over runs
instead of pinning one path forever; the pick and its origin go to the step
summary, because reproducing a red run means re-stating it verbatim. The
draw is restricted to what every distribution of that entry can deliver -
centos stream has no btrfs at all, and zfs is a third-party repository
outside debian and ubuntu - so a run never dies on a packaging gap. A
stated filesystem skips that restriction: it is an instruction, and a host
that cannot deliver it fails loudly rather than deploying on something
else.
docker_dataroot.sh formats a sparse loop image, mounts it at /mnt/docker-fs
and carves the data root inside it, then points the daemon there through
daemon.json - merged, not overwritten, so the registry-cache entry
survives. It runs on the runner for compose and inside each node for swarm,
before anything is pulled, and installs the filesystem tooling it needs.
Two container facts it has to work around: a privileged container's /dev is
a fresh tmpfs, so neither /dev/loopN nor /dev/zfs exists there and both are
mknod'ed from /proc/devices before use.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The file list came from `find .`, so every path carried a leading './'.
shellcheck writes those paths verbatim into its `-f diff` output, and
`git apply` rejects a diff whose paths start with './' as "invalid path",
which aborts the whole autoformat run.
The autofix branch was therefore dead from the start and nobody noticed:
it only runs when shellcheck actually has something to correct, and until
now it never did. The first real finding turned a silent no-op into a red
`make quality`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The swarm gate asked whether the drill ran by grepping the job log for
"Matrix-deploy <app>: provision/deploy/e2e/verify per round" — a copy of a
GitHub Actions step display name. d9a4498b3 renamed that step to
"🚀 Drill <apps> per distro: <distros>" and did not touch roundtrip.sh, so at
19d56f07e the literal existed in exactly one place in the tree: the grep itself.
It could no longer match anything, and four of the five workspace legs of run
30586213932 false-failed on deploys that had passed — svc-db-postgres converged,
answered on 5432, completed its DR drill and reported "Job succeeded" three lines
above the failure.
This is the second time the same coupling drifted. 0ca2ed56c is titled "align
roundtrip swarm gate to the live workflow step name" and records that the
previous stale marker made every swarm leg false-fail on a passing deploy.
Nothing cross-checked the literal, so it rots silently after each rename.
The marker now comes from the drill itself, printed after 07_assert_state.sh.
Every step before it runs under set -euo pipefail and the EXIT trap tears down
without printing, so the line is unreachable unless the drill completed for that
app. APP_ID and INFINITO_DISTRO are :?-guarded at the top of the script, so the
marker cannot degrade into an empty string that matches everything. The gate
matches it with grep -qF and names what it checked instead of asserting "empty
matrix" — a diagnosis that was never true and sent this investigation into the
wrong layer.
The gate itself stays. 214672a94 added it because act exits 0 on a zero-leg
matrix, which silently false-passed every local swarm roundtrip.
tests/unit/scripts/test_roundtrip_marker.py pins emitter and gate to the same
literal, pins the marker behind the state assertion, and fails if the old step
name reappears — the cross-check whose absence allowed both drifts.
Verified against the failing run's log: 0 occurrences of the old literal, while
DR drill PASSED, ran=1 skipped=0 and "Job succeeded" prove the drill reached the
point where the new marker is emitted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The compose and host deploy jobs tee the whole deploy stdout into the file
ANSIBLE_LOG_PATH names, and both downstream steps parse that stream. Only
stdout carries the host-side "matrix-deploy: round N/M ... PASS n" markers,
so it is the sole source the per-variant runtime tables can come from.
dedicated.sh's cleanup then copied the container-side ansible log onto that
same path. docker cp unlinks and recreates the destination, so tee kept
writing into an orphaned inode and the visible file became ansible's own
log_path output — whose "<ts> p= u= n= LEVEL| " line prefix defeats the
anchored recap matchers in logparse.py while the unanchored has_sections
substring test still fires. in_roles_recap never flips, every role-time line
is skipped, and the step dies with "no profile_roles entries found" although
the deploy succeeded. Run 30578244959: 14 of 14 compose+host jobs.
Why it looked like a flake: docker cp cannot read a /tmp that is a tmpfs
mounted inside the container, and the Debian image enables systemd's
tmp.mount while CentOS Stream 10 does not. The copy silently failed on
debian (host file survived, jobs green) and landed on centos (host file
destroyed). feature/swarm-nfs never had the block; it returned here through
merge drift from main, where the copy is still load-bearing because those
workflows do not tee.
The copy is dropped: stdout is a strict superset of what it provided, and
the in-container verifier reads the log inside the container. Alongside it:
the swarm workflow's file-exists guard is adopted so an aborted deploy does
not paint the summary step red on top; the log_path prefix moves to
utils/ansible_log.py and is applied in logparse.py, so an ansible run log
parses instead of yielding an empty summary; a lint pins that nothing writes
onto ANSIBLE_LOG_PATH again; and the parser regression test now covers the
recap headers, not just the role rows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make quality rejected the new reclaim script for missing the executable
bit, and the S310 suppression in the Home Assistant token probe for
carrying no reason. A security suppression must state why it is safe, so
the loopback justification goes back on the urlopen call; ruff dropped
the second marker on Request(), where S310 never fires.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The lab subnet is pinned per worktree slot while lab network names carry
the cluster id, so a cluster kept from one app blocked the next run of
another with 'invalid pool request: Pool overlaps'. The pre-run teardown
only cleared the same cluster id, which never matched. Matching on the
IPAM subnet instead clears exactly the blocker and leaves a sibling
worktree's cluster alone; swarm-clean stays the host-wide tool.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
git worktree remove deletes the checkout first and the metadata dir
second. Under the agent sandbox, which holds config.worktree and
commondir as read-only bind mounts, it therefore dies with EBUSY after
the checkout is already gone, aborting make worktree-down with rc 255
even though the branch is in fact free. down.sh now falls back to the
unregistration prune.sh already used (drop gitdir + HEAD, best-effort
rm -rf) and reports any dir the sandbox still pins. That routine moves
into lib.sh so both callers cannot drift apart, and gains a third
outcome for the case where gitdir/HEAD themselves survive: only then is
the branch still claimed, and only then must the teardown fail.
The discriminator for the fallback is whether the checkout survived,
not a match on "Device or resource busy" — the identical half-removed
state arises from EACCES, EPERM and EROFS.
A failing git status probe no longer reads as clean: on a broken .git
link it exits 128 with empty output, which used to satisfy the guard,
tear down the compose stack, and only then die at git worktree remove
with an unexplained 128.
Measured against the unfixed scripts with real fixtures: pinned
metadata dir 255 -> 0, broken .git link 128 -> 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
git worktree remove deletes the checkout first and the metadata dir
second. Under the agent sandbox, which holds config.worktree and
commondir as read-only bind mounts, it therefore dies with EBUSY after
the checkout is already gone, aborting make worktree-down with rc 255
even though the branch is in fact free. down.sh now falls back to the
unregistration prune.sh already used (drop gitdir + HEAD, best-effort
rm -rf) and reports any dir the sandbox still pins. That routine moves
into lib.sh so both callers cannot drift apart, and gains a third
outcome for the case where gitdir/HEAD themselves survive: only then is
the branch still claimed, and only then must the teardown fail.
The discriminator for the fallback is whether the checkout survived,
not a match on "Device or resource busy" — the identical half-removed
state arises from EACCES, EPERM and EROFS.
A failing git status probe no longer reads as clean: on a broken .git
link it exits 128 with empty output, which used to satisfy the guard,
tear down the compose stack, and only then die at git worktree remove
with an unexplained 128.
Measured against the unfixed scripts with real fixtures: pinned
metadata dir 255 -> 0, broken .git link 128 -> 1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The CI deploy bootstrap hand-listed two packages and let the ubuntu-latest runner image supply the rest, so a new host-side import surfaced as a red deploy job instead of a failed install. Size [project.dependencies] to what runs outside the project image and move the deployment stack into a deploy extra, then have every install site read that instead of naming packages itself: the image build and make install-python take the deploy extra, the CI host and the mirror and instructions workflows take the base.
Declare cryptography, which utils/manager/value_generator.py imports directly and which only reached the environment transitively through ansible. Drop numpy, which nothing in the repository imports. Guard the boundary with a lint test that blocks every deploy package and asserts the runner-host CLI still builds its parser.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The storage-constrained work put utils.storage.constrained into the host-side import chain of cli.administration.deploy.development, which pulls humanfriendly. The CI dependency line installed only ruamel.yaml and PyYAML, so every compose and host deploy job aborted at parser build time with ModuleNotFoundError.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three conflicts, none resolved by picking a side wholesale:
- Makefile: took the incoming ACT_INPUTS, which routes through SWARM_DISTROS
instead of inlining the same expression a second time.
- test-e2e-playwright/02_run_one.yml: took the incoming rc handling. An absent
rc now counts as failure; the previous default(0) let an aborted Playwright
run pass silently, contradicting the debug message right above it.
- web-app-peertube spec: combined both. Kept gotoOnion and resolveTimeout,
which this branch's lint enforces, and adopted the incoming retry-inside-poll
for the OIDC button plus its networkidle wait.
It shipped with the persistent-base change; the constraint it described - git forbids a branch feature beside feature/x, so the nested directories cannot collide - belongs in that commit's message, where it already is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The topology SPOT read default.env only, so every lab took the same subnet no matter what the generated .env said. Two labs on one host therefore collided with "invalid pool request: Pool overlaps", which is what stopped a worktree lab from running beside the primary checkout's.
Collect the generated .env first, then default.env, and emit each assignment as := so precedence becomes exported environment, then .env, then default.env. default.env stays the fallback because the workflow sources this file before make dotenv has run, which was the original reason for reading it directly.
A line that does not parse as a balanced assignment now aborts naming the line. Sourced as-is it would have become an unconditional assignment and silently defeated the precedence this change exists to establish.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An aborted Playwright run - watchdog SIGTERM, task timeout - returns a result dict with neither stdout nor rc. The reporting task dereferenced both, so it became the fatal error itself: the log ended on "object of type 'dict' has no attribute 'stdout'" while the real cause, a disk-exhaustion abort with rc 75, sat eight thousand lines earlier. Guard all three reads, fall back to msg for stderr because that is what Ansible fills on an abort, and treat an absent rc as a failure so the run is recorded and named by the aggregate check rather than vanishing behind a template error.
The distro sweep's summary also mislabelled the distros it never started: they kept the initial "not reached within the budget" note even when the sweep aborted on a failure with 12214 seconds of budget left, pointing at a budget problem that did not exist. Say what happened instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
act merges its --env-file, which defaults to .env, after the workflow-level env block (pkg/runner/run_context.go:84 at v0.2.89) and interpolates only afterwards, so INFINITO_DISTROS: ${{ inputs.distros }} was replaced by the repository's own five-distro list before the expression was ever evaluated. Every workspace leg therefore swept all five distros instead of its own: 25 distro deploys where 5 were intended, which is what walked the five legs into their 345-minute cap. The drill step's name kept rendering the single distro because it is interpolated from the workflow model rather than from the env map, which is why the logs read as a contradiction.
Pass INFINITO_DISTROS as an --env pair instead, the layer act itself wins with, where --env beats the env-file because the file only fills keys that are absent (cmd/root.go:364-366). The expression was duplicated across ACT_ENV and ACT_INPUTS, so name it once.
Wrapping the assignment for readability required the splitter to tolerate whitespace: inside double quotes the shell removes the backslash-newline but keeps the continuation line's indentation, so every pair after the first arrived with a leading tab.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pipeline gate only listed twelve of seventeen jobs in its needs, leaving out wait-fork-prereq-run, fork-prereqs-ready, build-ci-images, images-mirror-missing and test-dns. A failure in any of them turns its dependents into skipped rather than failure, and the gate's check matches only failure and cancelled, so the run would have concluded green without images ever being built.
A caller-supplied whitelist was also passed through verbatim. Run 30492550883 was dispatched with web-app-prometheu; discovery dropped the unknown id without a warning, the empty resolution became [], the deploy workflows skipped their whole matrix on that condition and the parent job still concluded success. Validate every id against roles/<id> and exit naming the offenders, so a typo cannot quietly remove an application from a run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The default base was /tmp, which is tmpfs on this host: a reboot took a whole worktree and its uncommitted work with it. Derive the base from the origin remote instead, as ~/.local/share/worktrees/<domain>/<account>/<repo>, and let the branch keep its slashes so it nests, which also makes the infinito-wt- prefix redundant. Without an origin remote it now fails loudly rather than inventing a location.
up.sh bumped only INFINITO_SWARM_LAB_SUBNET per slot while the five swarm lab node IPs stayed at default.env's 192.168.244.x, so every worktree from slot 1 on got a lab network in a different /24 than the nodes it places in it, and collided with the primary checkout's lab. Pin all six values from one octet.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The merge of the swarm-nfs work moved utils/unmount_nfs_mounts.sh to utils/unmount/nfs_mounts.sh and updated teardown.sh and the bootstrap, but clean/all.sh kept the old path. make swarm-clean died at the unmount step with "No such file or directory" before it reached container removal, which is exactly the state that needs cleaning up. The host-state release is not added here because all.sh already unmounts DIR_VAR_LIB itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
act fills the GITHUB_REPOSITORY* vars with its own placeholders, so the
existing owner guard passes and the resolver hands back ghcr.io/nektos/act/
<distro>. That path does not exist and the swarm node build died on a 403
three minutes into every local run. Returning nothing is what the callers
already treat as 'build it locally'.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f660dd6bd wrote the placeholder into a parameter expansion, `${INFINITO_IMAGE:-<none, bootstrap will build it>}`, which tests/lint/filesystem/shell/test_no_infinito_defaults.py rejects: an INFINITO_* default in a shell file is a second source next to default.env that drifts silently, and only the empty form is allowed. It was the sole failure of test-lint.
Split the message into the resolved and the unresolved case instead of suppressing the lint; the output is also no longer self-contradictory, since 'published <none>' could not both be true.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
act fills the GITHUB_REPOSITORY* vars with its own placeholders, so the
existing owner guard passes and the resolver hands back ghcr.io/nektos/act/
<distro>. That path does not exist and the swarm node build died on a 403
three minutes into every local run. Returning nothing is what the callers
already treat as 'build it locally'.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f660dd6bd introduced `${INFINITO_IMAGE_TAG:?}` in the image resolution, a hard requirement the replaced call deliberately did not have -- ci.sh exits 0 before its own assert when no registry owner is in scope. The check sat after the EXIT trap was armed and before the env layer is sourced, so a missing tag ran collect_and_teardown against a cluster that was never created, and the variable appeared in neither the # Param header nor the strict guard block that carries the other four required inputs.
Move it into that guard block, which runs before mktemp and before the trap, and drop the now-redundant inline assert.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
841a5e0e1 claimed to keep entries whose checkout cannot be inspected, but the check resolved dirname(dirname(gitdir)) -- the checkout's PARENT, e.g. /tmp -- which always exists. The branch was dead code and every entry fell through to `rm -f gitdir HEAD`, including a live worktree whose metadata git can no longer read: that path drops it from the liveness list and it was then unregistered, which is the reported symptom.
Make the checkout directory itself the criterion, and refuse to conclude anything when its parent is not traversable -- `test -d` succeeds on a directory without +x, so a first attempt using -d alone reproduced the same blind spot. Reproduced in throwaway repos: a live worktree with a 0-byte commondir keeps its registration, a deleted checkout is pruned, and a metadata corpse without gitdir is cleaned.
Remaining limit: `git worktree prune`, which the script runs first, shares the blind spot and wins where it can unlink; this guard therefore only decides in the sandbox-pinned case that produced the bug.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every workspace leg pinned INFINITO_DISTRO=debian and drilled all five distros sequentially, pulling its swarm nodes from ghcr <distro>:latest -- an image from 2026-03-21 whose nfs-ganesha 7.2 segfaults on any accepted TCP connection on 2049. The controller mount then failed with ECONNREFUSED for 26 minutes and took run 30446813110 down after 5h13m, with 3 of 5 distros never reached.
Derive the distro from the leg's own matrix entry, build that image, and drill only it: swarm-zombie takes an optional distros= that defaults to INFINITO_DISTRO, and 00_one.sh prefers the locally built image over the published one. 12_swarm.sh refuses to start when that image is missing, so a broken build fails loudly instead of being masked by a stale registry pull.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Staleness was decided by reading gitdir inside the metadata dir. Under the agent sandbox that dir is pinned as read-only bind mounts, so a live worktree can read as having no gitdir at all -- prune then removed gitdir and HEAD and left the checkout as 'fatal: not a git repository' with no way back. Decide liveness from git's own worktree list plus each checkout's .git pointer instead, keep entries whose checkout base is not visible rather than guessing, drop the expected EBUSY noise, and report the pinned leftovers once with the command that clears them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f660dd6bd wrote the placeholder into a parameter expansion, `${INFINITO_IMAGE:-<none, bootstrap will build it>}`, which tests/lint/filesystem/shell/test_no_infinito_defaults.py rejects: an INFINITO_* default in a shell file is a second source next to default.env that drifts silently, and only the empty form is allowed. It was the sole failure of test-lint.
Split the message into the resolved and the unresolved case instead of suppressing the lint; the output is also no longer self-contradictory, since 'published <none>' could not both be true.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
act fills the GITHUB_REPOSITORY* vars with its own placeholders, so the
existing owner guard passes and the resolver hands back ghcr.io/nektos/act/
<distro>. That path does not exist and the swarm node build died on a 403
three minutes into every local run. Returning nothing is what the callers
already treat as 'build it locally'.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f660dd6bd introduced `${INFINITO_IMAGE_TAG:?}` in the image resolution, a hard requirement the replaced call deliberately did not have -- ci.sh exits 0 before its own assert when no registry owner is in scope. The check sat after the EXIT trap was armed and before the env layer is sourced, so a missing tag ran collect_and_teardown against a cluster that was never created, and the variable appeared in neither the # Param header nor the strict guard block that carries the other four required inputs.
Move it into that guard block, which runs before mktemp and before the trap, and drop the now-redundant inline assert.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
841a5e0e1 claimed to keep entries whose checkout cannot be inspected, but the check resolved dirname(dirname(gitdir)) -- the checkout's PARENT, e.g. /tmp -- which always exists. The branch was dead code and every entry fell through to `rm -f gitdir HEAD`, including a live worktree whose metadata git can no longer read: that path drops it from the liveness list and it was then unregistered, which is the reported symptom.
Make the checkout directory itself the criterion, and refuse to conclude anything when its parent is not traversable -- `test -d` succeeds on a directory without +x, so a first attempt using -d alone reproduced the same blind spot. Reproduced in throwaway repos: a live worktree with a 0-byte commondir keeps its registration, a deleted checkout is pruned, and a metadata corpse without gitdir is cleaned.
Remaining limit: `git worktree prune`, which the script runs first, shares the blind spot and wins where it can unlink; this guard therefore only decides in the sandbox-pinned case that produced the bug.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every workspace leg pinned INFINITO_DISTRO=debian and drilled all five distros sequentially, pulling its swarm nodes from ghcr <distro>:latest -- an image from 2026-03-21 whose nfs-ganesha 7.2 segfaults on any accepted TCP connection on 2049. The controller mount then failed with ECONNREFUSED for 26 minutes and took run 30446813110 down after 5h13m, with 3 of 5 distros never reached.
Derive the distro from the leg's own matrix entry, build that image, and drill only it: swarm-zombie takes an optional distros= that defaults to INFINITO_DISTRO, and 00_one.sh prefers the locally built image over the published one. 12_swarm.sh refuses to start when that image is missing, so a broken build fails loudly instead of being masked by a stale registry pull.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Staleness was decided by reading gitdir inside the metadata dir. Under the agent sandbox that dir is pinned as read-only bind mounts, so a live worktree can read as having no gitdir at all -- prune then removed gitdir and HEAD and left the checkout as 'fatal: not a git repository' with no way back. Decide liveness from git's own worktree list plus each checkout's .git pointer instead, keep entries whose checkout base is not visible rather than guessing, drop the expected EBUSY noise, and report the pinned leftovers once with the command that clears them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The swarm branch predates the shell-or-true lint, so its EXIT-trap collectors
arrived unmarked. They are the legitimate case: best-effort diagnostics and
teardown that must not mask the original exit code, which the trailing
return "${rc}" preserves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mapper name is host-global and nothing closes it once a drill succeeds:
the only teardowns sit inside the recover step and in the post-luksOpen
error path. A preserved cluster therefore hands the next round a live
'usbdrill' and cryptsetup refuses with 'Device usbdrill already exists'.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test-environment.yml and test-guide.yml become test-workspace.yml and
test-instructions.yml. Orchestrator job ids, the distros-SPOT matrix
lint, the act example, the diagnostics slug and the rescue artefact
names follow.
Replace the roles_only input with per-job workspace/instructions
tri-state inputs (auto|true|false). 'auto' derives the former
roles_only skip straight from whitelist: both jobs run on a global
scope (whitelist empty or __ALL__) and are skipped on a role-scoped
run, which keeps the PR behaviour identical because the diff resolver
never emits an empty whitelist. entry-manual exposes both as choice
dropdowns defaulting to false, so a manual run opts in explicitly.
The now redundant roles_only output is dropped from
pr_affected_roles.sh and subset_roles.py.
Branch-protection required-status-check names change along with the
job renames.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The artifact of the wedged run carried journal, unit status and mounts but no process data, so the hang could be localized to the export config commit and not pinned to a syscall. wchan and syscall per thread close that gap.
MainPID 0 is its own likely state given Restart=on-failure with the start-rate limit disabled, so that branch reports the unit counters instead. Every read keeps 2>&1 because /proc stack and syscall are EPERM on hardened kernels.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
git worktree prune removes the metadata directory as a whole, but the agent
sandbox pins config.worktree and commondir as read-only bind mounts, so the
unlink fails with EBUSY and the stale entry keeps claiming its branch.
Dropping gitdir and HEAD unregisters it without needing the pinned files gone.
The LUKS step of the DR drill died on `The file /dev/mapper/usbdrill does
not exist and no size was specified.` -- from mkfs, not from cryptsetup,
because luksOpen had exited 0. The node did have a udev, and it had
created the mapper entry: as a symlink to ../dm-1. Modern udev no longer
mknods, it only manages symlinks, and the /dev/dm-N target it points at is
a devtmpfs node that a container's tmpfs /dev never receives. Without a
udev libdevmapper creates the node itself, which is why the debian nodes
passed -- their image ships no udevd at all, while arch cannot install
systemd without one.
Masking udevd in the node image removes the whole class rather than one
call site. DM_DISABLE_UDEV rides on the node containers instead of the
script because the drill opens the volume a second time through the
recover CLI in a separate docker exec, which would not inherit an export.
The block-device check then fails loudly with both prerequisites named,
and closes the mapping first so an abort cannot leak it onto the runner --
device-mapper is not namespaced and every distro round shares the VM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ganesha unregisters its RPC programs from inside nfs_Init_svc(), before
the sockets are ever put into listen(). That unregister reaches
libntirpc's rpcb_unset -> CLNT_DESTROY -> svc_vc_destroy_it, which spins
on nanosleep({0,0}) with no timeout -- and it only takes that path when
the AF_LOCAL connect to /var/run/rpcbind.sock succeeds. So a reachable
rpcbind is what wedges the server, not a missing one.
The wedged instance burned 5m30s of CPU while serving nothing, never
logged NFS SERVER INITIALIZED, and left the controller mount retrying
into ECONNREFUSED for 1706s before the deploy failed 20 minutes further
downstream. Masking and stopping both rpcbind units before ganesha starts
makes the spin unreachable: without a local socket rpcb_unset returns
false immediately. Wants=rpcbind.service is not Requires=, so a masked
unit is logged and ignored, and v4 registration is explicitly optional in
ganesha -- which is why this holds only while the export set stays
NFSv4-only.
The unit list is probed first so distros without rpcbind are skipped
rather than failing the play, and the probe distinguishes "no such unit"
from an unreachable manager so the mask can never be skipped silently.
The Type=forking unit reports started while the daemon is still spinning,
so a wait_for on 2049 turns a 28-minute mystery into a named failure, and
the diagnostics now carry rpcbind's journal plus what actually listens.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guide job builds its boot image from a pinned pkgmgr-arch parent that
can be weeks old, and `pacman -Sy systemd` against it is the partial
upgrade arch upstream documents as unsupported. It pulled systemd 261.2-1
next to the image's systemd-sysvcompat, which pins systemd exactly, so the
transaction could not be satisfied and every arch guide run died before
the README sequence started.
Refreshing archlinux-keyring first and then running a full -Syu --needed
fixes it; the cache wipe matches what the apt and dnf branches already do.
The upgrade lands in the PREP container, which runs sleep infinity, so
systemd is only ever replaced before it becomes PID 1 in the committed
image.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
git worktree prune removes the metadata directory as a whole, but the agent
sandbox pins config.worktree and commondir as read-only bind mounts, so the
unlink fails with EBUSY and the stale entry keeps claiming its branch.
Dropping gitdir and HEAD unregisters it without needing the pinned files gone.
The node container has no udev, so LOOP_CTL_GET_FREE hands out a minor whose
/dev node never appears; it also cannot load kernel modules, so those nodes
stay dead unless the host carries the loop driver. Materialize the nodes,
verify one can actually be attached, and expose the host prerequisite as a
bootstrap step instead of failing three hours in with a cryptsetup message.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The controller mount of DIR_VAR_LIB survived the teardown, so the next
distro cycle inherited an NFS session whose server container was already
gone: the kernel logged 'server not responding, timed out' and every
access under /var/lib/infinito wedged, taking the deploy down with it.
All five swarm jobs of the last manual run died this way, each on a
different distro but always on the third lab cycle.
Both halves of the old teardown were unable to clear it. The guard
`mountpoint -q` blocks on exactly the dead mount it is asked about, so
the timeout dropped the unmount that was needed; and `umount -lf` ran
without sudo on a non-root runner, so the EPERM went into `|| true`.
Detection now reads /proc/self/mountinfo, which touches no filesystem,
the unmount elevates when the caller is not root, and a mount that
survives all of it fails loudly instead of silently poisoning the next
cycle. The bootstrap runs the same release before bringing a lab up, so
a cycle whose teardown never ran cannot hand the mount on either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
