01_bootstrap.sh ran apt-get update with no timeout, so a stalled mirror
connection hung until the drill's global budget killed it - twice in run
32284541284 (jobs 96178507396 funkwhale, 96178507443 friendica), each
after hours of silence. The journal pins it exactly: sudo[462047] opened
19:51:36 and closed 00:07:30, 4h15m54s in a single apt-get update, and
the session only ended because timeout fired. Healthy runs of the same
line in the same jobs took 3s, 21s and 41s.
The cost is not one red job. distros.sh:130 is the only ceiling and it
wraps the whole per-distro sequence in the remaining global budget, so
one stalled fetch also takes the untried distros with it: funkwhale ran
2 of 5, friendica 3 of 5, both ending at budget remaining=-19s/-20s.
-qq hid the stall and || true cannot catch a hang.
Same failure class and same remedy as ba8a72309, which bounded
scripts/install/apt.sh; this closes the sibling call site it left open.
Constants are taken verbatim from there rather than re-derived. timeout
runs under sudo so SIGTERM reaches apt-get instead of the sudo wrapper -
the 4h session above is what the other order looks like. -E stays so the
package-cache proxy variables survive.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 32118850138 left a dnsmasq that was alive with zero sockets,
nothing logged, and a 6m39s death window between two host DNS
consumers. Capture each dnsmasq's fd table and rlimits in the rescue
snapshot (closed listeners vs exhausted fd table vs lost netlink
watch), and sample the :53 listener set every 10s inside the DiD
container so the next occurrence is bounded to one bucket.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Iterating on one app meant restarting the whole stack or reaching for raw
docker. The target mirrors compose-app-logs and goes through the
repository's own container wrapper.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The restore cycle stops every subject of the host, replays into the database
services alone, and starts the rest again at the very end. Any exit in
between skips that last step, so the host keeps its subjects down - and
nothing recorded it.
The next deploy then failed on a symptom of the teardown rather than on
whatever ended the drill. It cost two runs: once 'rake db:migrate' against a
database the cycle had emptied but not yet replayed, once
Redis::CannotConnectError because the cycle never restarted redis. Both times
the message pointed away from the cause, and fixing what it named would have
fixed nothing.
A trap armed before the teardown now records the teardown and the exit code,
and mode=update refuses to run while that record stands. purge=true and
mode=reinstall pass, because both rebuild the state the drill destroyed; a
completed cycle clears the record itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both were left by earlier commits and are collapsed by markdownlint-cli2 and shfmt respectively, so every make quality run left the tree dirty and the change would eventually have ridden along in an unrelated commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The converge gate polls STACK_NAME_PRIMARY_SERVICE_KEY. Four roles never
create that name and each was given a static `workload: node-local` to make
the gate skip itself:
21fe9e984 web-svc-cdn ships only cdn_build, never cdn_cdn
2c6c8b763 web-app-matrix names its services matrix_synapse and friends
76508c946 svc-runner deploys node-local compose projects per node
cb52a72f2 bigbluebutton phantom service, and the DR drill must still run
Four hand-written claims for one observable fact. The claim is also static
where the truth is not: web-app-discourse renders `services: {}` only while
its sidecars are shared, and job 94965456921 of run 31864448027 polled
discourse_discourse ninety times for a stack that had none. Declaring the
role node-local would have blinded the gate in the variants where its
sidecars are local and a stack does exist.
has_swarm_service asks the deployed stack for the exact name instead. Every
caller runs behind `if rc == 0`, so an absent service is the role's shape and
not a broken deploy. Exact-line matching, not docker's name filter, which
matches substrings.
No coverage is lost: 03_wait_converge cannot pass without that service, and
05/06/07 plus the backup drill only run once it has. The four roles keep the
skip they were given; discourse and every future case get it without a
declaration. The teardown branch in the drill keeps its meaning - stack rm
where the service exists, compose stop per node where it does not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dnsmasq and apt-resolution investigations both stalled on facts the
snapshot never recorded.
The snapshot is taken by docker exec inside the DiD container and the
recursion only descends, so the runner's own docker - where the resolver
runs as ${INFINITO_RUNNER_PREFIX}-coredns - was structurally invisible.
dedicated.sh now runs a second pass on the CI host, with RESCUE_SEEN
seeded with the DiD id so it does not collect the first snapshot twice.
For the rest: a process table, so the MainPID that service-state reports
can be checked against something; the dnsmasq drop-in directory, because
this project's config lives at /etc/dnsmasq.d/tor-onion.conf and never in
the main file; nsswitch.conf and hosts, without which a getent verdict is
ambiguous; one probe per name plus, where the distro ships dig or
nslookup, one per nameserver, so a refusing resolver is distinguishable
from an unreachable mirror; the full firewall table set, since a dropped
query is recorded in filter and -t nat was actively hiding it; routes and
neighbours; os-release, because the matrix is distro-shaped and a bundle
could not say which leg it came from; the runtime's networks, which hold
the forwarder behind every container's 127.0.0.11; free inodes, which a
df -h reads as free space; and the runtime's disk, volume and image
inventory, the last with full references, since an engine was once picked
from a registry host embedded in an image name.
ps runs without args: the bundle is uploaded as an artifact and argv
carries whatever credentials a process was invoked with - a real proxy
password showed up in the first live run of this capture.
Two exec budgets are corrected while the per-container cost grows to
three calls. The per-container journal was borrowing the 600s meant for a
host boot journal, so one wedged container could cost 840s; container
journals span a deploy, not a boot. The cheap questions get 30s, and the
one collection site that had no outer bound gets the timeout 1500 the
others already carry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
call-test-deploy-{compose,host,swarm,single-node}.yml differed only in
which mode they pinned and which artifact prefix they wrote; compose and
host already ran the identical drill. They become one call-test-deploy.yml
whose matrix rows carry the mode, so the swarm-specific steps are guarded
on it and everything else is shared.
The orchestrator loses its eight deploy jobs and the sequencing job for
three serial chunk blocks. GitHub Actions cannot generate a variable
number of jobs, so the blocks are written out and INFINITO_CI_MAX_CHUNKS
tells the budget how many exist; blocks whose slice is empty skip
themselves. chunk_gate stops the chain at the first red chunk,
resume_from_chunk re-enters after a fix instead of re-running the green
chunks, and modes collapses to a single mode choice.
apps.sh now emits matrix entries for one chunk rather than a token list
for one mode, and no longer sorts them: the discovery order is what the
chunker slices on. split_runner_apps.sh reads its runner quota from
INFINITO_CI_CONCURRENCY instead of carrying a second copy of the 20.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
30% was a guess. Measuring it: across 60 successor distro runs in 22 jobs of run
31756400129, each against the running max of the runs before it, the median
overshoot is 0.991 - a successor is typically slightly faster - and the worst
ever observed is 1.149 (fedora 5541s against a 4822s max_seen).
15% therefore covers every observed case, and nothing above it buys more. It
costs less, too: 30% refuses 23 of 74 start decisions, 15% refuses 18. Five
distro runs that would have fitted, for no additional protection.
The tests read the percentage back out of the log line and check the formula
against it, so they hold at any value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
508669809 booked a budget SIGKILL as skipped so the job went green. That is a
soft-skip: a kill is indistinguishable from the command hanging, so it hid a
real failure behind a green tick.
Fail on it again, and prevent it upstream instead. A distro now starts only
while the remaining budget covers the slowest run so far plus 30%. Fediwall run
31756400129 shows why the bare comparison was not enough: max_seen=6770s against
remaining=6878s let the distro start, and it took 6942s, 64s past the deadline.
max_seen is the maximum of the durations already observed in the same sweep -
a small sample over distros of genuinely different weight, drawn in randomised
order, so it underestimates systematically.
Removing the soft-skip also removed its diagnosis, so a killed distro now
reports both numbers without claiming a cause: the run only knows the exit code
and how much budget was left when it started.
The suite drops from 85s to 4.1s. The 68s case made a process ignore SIGTERM to
force the -k escalation, which exercised coreutils rather than this script; the
124 path reaches the same branch in a second.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ad9222630 wrapped each distro in timeout so a run that outlives the budget stops instead of drifting into the action timeout, and mapped rc 124 to skipped because stopping at the budget is the documented intent rather than a defect. It verified that timeout returns 124 for its own kill. It did not cover the other one: timeout -k escalates to SIGKILL when the command survives SIGTERM, and then reports 137, which is equally its own kill. An ansible run with live docker children survives SIGTERM routinely, so 137 is the common case rather than the exotic one.
The result was that the governor never fired as documented. Fediwall swarm and Nextcloud swarm 0 in run 31756400129 and 31756560361 both ended on [ERROR] Run failed for distro=arch (rc=137) with budget=20400s remaining=-60s. Fediwalls playbook had already printed PLAYBOOK RECAP, the stack had converged and every node port answered; it was booked as a failure 0.3 seconds later.
The two narrowing conditions carry the discrimination and are unchanged. dur >= remaining - 5 rejects a SIGKILL that arrives at an arbitrary time, such as an OOM kill mid-run, because a budget kill lands at the deadline by construction. passed > 0 still requires an earlier distro to have succeeded.
A unit case in tests/unit/scripts/tests/deploy/test_distros.py drives a distro that traps SIGTERM and sleeps. Without the change it reproduces the CI failure locally, exit 137 and Run failed for distro; with it the sweep exits 0 and the summary marks the distro skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill starts the backup units itself, but the deploy arms their calendar
timers first, and a job that runs long enough crosses every backup window in
group_vars/all/07_schedule.yml. web-app-decidim ran 23:05-02:24 and hit exactly
that: the drill's own volume backup succeeded at 01:00:49, then the timer fired
a second, unattended run at 01:03:18 while step [8b/9] was lazily unmounting
/var/lib/infinito underneath it.
The failure is silent by construction. baudolo writes the dump in two
processes - 'pg_dump > x.tmp', then 'mv x.tmp x'. The redirect shell exits 0
because a shell never reports a failed writeback on a soft NFS mount, but the
unmount discarded that mount instance together with the unflushed file, so the
mv resolved through the fresh instance and got ENOENT. The unit died at
01:04:04, the same second the drill finished remounting, and 80 minutes later
the CLI test read 'failed' and aborted the job before running its own backup.
Disarming the timers makes the drill the only thing driving those units.
Stopping a timer does not stop its service, and both the drill and
roles/svc-bkp-volume-2-local/files/test/test.sh start their units explicitly,
so nothing under test loses coverage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docker's name filter matches substrings, so every stack whose sidecar is
named after its app answered with two containers and two services. The
drill took the first, which after a reschedule was the MCP sidecar: it
declares no healthcheck and does not listen on the app port, so the step
waited sixty seconds on the wrong process. Selection now goes through the
swarm service label and an exact name match, both behind one helper.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill probed for a fixed 60s. A container without a shell can only be
observed through its docker health status, and mattermost declares
start_period 30s with interval 1m, so that status cannot report healthy
before 90s. The step was unreachable for such an app however fast it
started. The budget now comes from the container's own healthcheck and
stays at 60s for every container that declares none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The quiesce guard spelled the dev container as the literal
infinito_nexus_debian, so it only ever matched a checkout whose .env was
generated for debian without a project prefix. It missed the self-hosted
runner form infinito_<project>_nexus_<distro>, any custom.env pin, and any
.env generated for another distro — and it missed them silently, because a
non-matching grep just skips the block and leaves the competing onion
publisher running.
INFINITO_CONTAINER is the SPOT for that name: compose.yml uses it as
container_name, and scripts/meta/env/load.sh (already sourced at line 19)
exports it. Proven against the container running on the author's host: the
literal does not match it, the variable does.
The outer `docker ps | grep -qx` pre-check is gone with it — `docker exec`
on an absent container already exits non-zero with stderr suppressed, and
the `if` absorbs that even under `set -euo pipefail`.
The variable joins the line-21 assertion so a broken env fails by name
instead of dying inside a pipeline subshell, which would reproduce exactly
the silent-skip class this commit removes. compose.yml already requires the
same variable with `:?`, so this is not a new demand on the environment.
tests/lint/repository/test_no_hardcoded_container_name.py pins it: run
against the pre-fix file it flags all three lines, and it stays green on
the distro-agnostic spellings in stack_refresh.sh and worktree/up.sh.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dump listed the state path itself, so the volume directories showed up but
nothing inside them. Ownership questions about a specific file were therefore
unanswerable from the artifact alone, which is how a permission failure on a
database file cost two investigations without ever being settled.
Records mode, numeric owner and size one level in. Numeric because the
container uids have no names on the nfs server, and those uids are what the
questions are about.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Docker hands out the lowest free address, so a node attached without one lands
in the same band where the cache services pin themselves and can take an
address its owner then cannot claim. That happened: a node held the
registry-cache address and every swarm cycle died at the image pull.
Each node now takes a fixed address, offset per instance because worktrees
share the primary's cache network while clearing only their own cluster, so a
surviving cluster elsewhere no longer collides with a fresh one.
A failed attach is classified rather than swallowed. The nodes already carry
extra_hosts pointing at the cache addresses, so a node that silently fails to
attach reaches neither apt nor the registry and reports it as network
flakiness. Only the markers the docker_network_noop filter already treats as
benign are tolerated; anything else stops the bootstrap.
The subnet is asserted rather than assumed, since the address arithmetic only
holds for a x.y.z.0/24.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The routine only carried an EXIT trap, so a run terminated by the enclosing
`timeout` in distros.sh:128 left no rescue diagnostics, no playwright reports
and no teardown — the exact case where the artifacts matter most.
Add a TERM trap and make the handler take the status as an argument, so EXIT
keeps using `$?` while TERM passes 143 explicitly. The `_teardown_done` latch
is required because the EXIT trap still fires after the TERM trap; without it
the whole collect-and-teardown block, including its 1500s and 900s timeouts,
would run twice.
This works because GNU timeout signals the whole process group: bash defers a
trap while it waits on a foreground child, so a bare kill to the script PID
alone would not surface it until that child exited. Reproduced under the real
shape (timeout -k): TEARDOWN rc=143, exactly once, after 2.01s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The swarm DR stage aborted with "no backup unit installed on <mgr> or
<nfs-server>" on all five workspace distros. routine/backup/base.sh triggers
svc-bkp-volume-2-local / svc-bkp-secrets-2-local on the manager and
svc-bkp-nfs-2-local on the export host; all three trigger_units calls returned
rc=2 (no unit matches), so _triggered stayed 0.
The units were never installed because the roles were never in the closure.
utils/tests/swarm/README.md pins the contract: derive_includes pulls
svc-bkp-volume-2-local in via the `container_backup` consumer and
svc-bkp-nfs-2-local via `nfs_backup`. derive_includes._disabled_provider_roles
maps those same keys back to those roles and `found -= disabled_roles` prunes
them, so any caller that lists them in `disable` silently removes what the
drill then requires.
scripts/tests/environment/12_swarm.sh and 13_roundtrip.sh both do exactly that:
`variable_services <app> node nfs-server` keeps three keys and disables every
other group-conditional one, which sweeps up container_backup and nfs_backup as
collateral. Reproduced offline byte-identically to the CI line:
$ python3 -m cli.meta.roles.applications.dynamic_services \
--exclude mariadb node nfs-server
asset,...,container_backup,...,nfs_backup,...
Filtering in those two callers would leave `make swarm-zombie app=X disable=...`
and any future caller exposed, so the filter goes to the single entry point every
swarm drill passes through: the CI matrix reaches routine/00_one.sh via
distros.sh, and swarm-zombie reaches the same script through act. Both `disable`
consumers -- the provision-time services_disabler and derive_includes in the
matrix subprocess -- read the env var this script re-exports.
Measured before/after on the derive_includes closure:
APP_ID=svc-db-mariadb -> {} / {svc-bkp-secrets-2-local, svc-bkp-volume-2-local}
APP_ID=svc-storage-nfs-server-> {} / {svc-bkp-nfs-2-local}
node and nfs-server join the set for the same reason db87a1a1c added them to the
callers: they build the lab cluster itself.
Token matching is exact, so nfs_backup2 and mynode survive; `IFS=', ' read -a`
avoids the pathname expansion an unquoted split would have allowed. The CI swarm
workflow passes no `disable` at all, so the filter is a no-op there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Diagnosing a failing app meant reading its container logs, and the only
route to them was raw docker. The target mirrors compose-app-exec and
goes through the same container wrapper, so the log route is no wider
than the exec route already is.
The workflow directory mixed entry points, reusable stages and scheduled jobs into one flat list sorted by topic, so the first question about any file - what starts it - was the one the name did not answer. Every workflow now carries one of three prefixes, derived from its on: block: cron- for a schedule trigger, call- for one only reachable through workflow_call, entry- for everything a repository event or a human starts. Scheduled workflows additionally carry an alarm clock as the first character of their name.
Two lints keep it that way. One derives the expected prefix from the triggers and compares it against the file name; it flagged all 17 offenders before the rename and none after. The other resolves every .github/workflows/<file> string in the repository against the directory, which is what a rename silently breaks in Markdown links, Python constants and shell API URLs alike.
Renaming pulled 69 pre-existing comments into the comment lint's changed-file scope. The vendored CodeQL template got the nocheck marker its scorecard sibling already carried, two workflow headers moved above name: where they count as file headers, and the remaining banners were deleted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
GitHub shipped the concurrency queue property on 2026-05-07. actionlint 1.7.12 and act v0.2.89 both predate it: actionlint reports an unexpected key, act refuses to parse the whole workflow. Ignore the finding in the lint call, and hand act a copy with those lines stripped so local iteration survives once the workflows use the key.
Refs: rhysd/actionlint#654, nektos/act#6095, nektos/act#6152
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One conflict, in the category catalogue: this branch added a virt category for
the sandboxed container runtimes, the other did not.
Keeping our side alone would have closed the conflict and broken the build. The
same merge introduces a test requiring every category to declare a hanzi label,
unique among its siblings, and virt predates that rule and carries none. It now
declares 沙箱, checked against its ten siblings for a collision.
The local exec helpers covered the development container and a throwaway
sidecar next to it, but nothing reached into an application the stack had
already deployed. Swarm had swarm-app-exec for that; compose had no
counterpart, so the only remaining route was a nested docker exec, which the
contributor rules forbid.
The helper goes through container.sh's positional form rather than its cmd
environment variable. container.sh runs that variable through `sh -lc`, so an
already-composed command would be split a second time; passing argv through
untouched leaves exactly one shell level, inside the target container.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
web-app-openwebui 0,1,2 was killed by the 350-minute action timeout with no
failed task anywhere in its 1.33M log lines. The sweep admitted fedora at
remaining=10372s after debian took 10028s, then never looked again: the
budget was only checked between distros. Its internal deadline passed 594
seconds before GitHub killed the job - the run knew it was over time for ten
minutes and had no way to act. Worst-case runtime was deadline plus the full
duration of the last admitted distro, unbounded.
The trigger was a single stalled task (803s for a no-op that takes 1.3s
warm, and a fully cold transaction 8 minutes later cost 24s), but that only
decided which job hit the wall. The wall itself is structural.
remaining is already computed at the top of the loop, so wrapping the call
in timeout adds no new state. rc 124 becomes 'skipped' rather than 'failed',
because a job that stops at its budget is the documented intent, not a
defect - test_distro_budget_fits_timeout.py already asserts this invariant,
which until now was simply untrue.
Two conditions narrow it, and both matter:
dur >= remaining - 5 keeps an inner timeout that fires early as a failure.
Verified: timeout returns 124 only for its own kill and passes inner exit
codes through unchanged (0 for success, 3 for an inner exit 3).
passed > 0 is the one that protects the test result. Without it, a job whose
very first distro overruns would report 'skipped' and exit 0 - green while
having verified nothing. That would have been a silent false pass across all
243 jobs of a run. The guard was executed over the full truth table,
including passed=0.
-k 60 is deliberate: one.sh execs into dedicated.sh and its ansible and
docker children can outlive SIGTERM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
07_compose_minimal.sh claimed to disable 'all variable (group-conditional)
services' and then named seven by hand. tor was not among them, so a
deploy meant to be minimal pulled svc-net-tor, which hard-depends on
web-svc-mirror, whose group-conditional prometheus flag the closure
resolver treats as true — dragging the whole web stack in. Steps 12 and
13 passed no disable at all, which is why a mariadb swarm test deployed
keycloak, mailu, matomo, prometheus and dashboard.
The set is now derived from the contract the repo already enforces
(tests/integration/roles/meta/services/test_dynamic_flags.py): a key is
group-conditional when enabled OR shared is a Jinja expression. Checking
only enabled would miss mariadb, postgres, redis, memcached, typesense.
Exclusions are load-bearing, not cosmetic: dashboard because disabling
the primary app aborts init; javascript because it aliases cdn, which
web-app-dashboard declares a hard dependency with no runtime fallback;
mariadb and postgres because steps 12 and 13 test them.
variable_services assigns to a local first — disable="$(cmd)" does not
set the exit status of a simple command, so a failing CLI would have
yielded an empty disable set and steps 12/13 would have run the full
deploy and reported green.
roundtrip could not take disable at all; it now reaches both the compose
and the swarm leg.
Not verified by a deploy: 33 of the keys are new to step 07, and the
variant axis of svc-db-mariadb/svc-db-postgres collapses because tor is
its only dimension.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
web-app-openwebui 0,1,2 was killed by the 350-minute action timeout with no
failed task anywhere in its 1.33M log lines. The sweep admitted fedora at
remaining=10372s after debian took 10028s, then never looked again: the
budget was only checked between distros. Its internal deadline passed 594
seconds before GitHub killed the job - the run knew it was over time for ten
minutes and had no way to act. Worst-case runtime was deadline plus the full
duration of the last admitted distro, unbounded.
The trigger was a single stalled task (803s for a no-op that takes 1.3s
warm, and a fully cold transaction 8 minutes later cost 24s), but that only
decided which job hit the wall. The wall itself is structural.
remaining is already computed at the top of the loop, so wrapping the call
in timeout adds no new state. rc 124 becomes 'skipped' rather than 'failed',
because a job that stops at its budget is the documented intent, not a
defect - test_distro_budget_fits_timeout.py already asserts this invariant,
which until now was simply untrue.
Two conditions narrow it, and both matter:
dur >= remaining - 5 keeps an inner timeout that fires early as a failure.
Verified: timeout returns 124 only for its own kill and passes inner exit
codes through unchanged (0 for success, 3 for an inner exit 3).
passed > 0 is the one that protects the test result. Without it, a job whose
very first distro overruns would report 'skipped' and exit 0 - green while
having verified nothing. That would have been a silent false pass across all
243 jobs of a run. The guard was executed over the full truth table,
including passed=0.
-k 60 is deliberate: one.sh execs into dedicated.sh and its ansible and
docker children can outlive SIGTERM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In job 91983553540 all five lab nodes derived the same pool name,
infinito_ci_unknown, and the run died on `cannot destroy
'infinito_ci_unknown': pool is busy`. The name came from `hostname`, which
returns non-zero inside the node containers, so the `|| echo unknown` fallback
answered for every one of them. One node won the race and created the pool; the
others found it, took it for a leftover, and tried to tear down a pool a
sibling was actively using.
234c1158c already fixed this collision class three days ago by deriving the
name from `hostname` instead of a constant, and its message named the zfs
exclusion in resolve.sh as the only thing keeping the collision out of sight.
That fix was never exercised, because the exclusion stayed up until ed1121053
removed it - and it does not work, because the call it rests on fails in the
containers it was written for. Guessing the identity is the defect; the
playbook has known it all along as inventory_hostname, and ci/one.sh knows its
own deploy mode. Both now state it, as a required fourth argument with no
default, the way SIZE is already stated. verdict() stops guessing too: every
row it wrote into the job summary said `unknown`, which is why the summary was
never worth reading.
The second half is independent of the naming. reclaim_stale_pool ran
`zpool destroy -f` as a bare command under `set -euo pipefail`, the only
failure path in this script that bypassed decline(). A pool that refuses to be
destroyed is by definition not a leftover - something holds it - so that case
now declines through the usual contract: a drawn pick falls back and reports,
a stated one fails loudly as required-but-unavailable. A reclaim can no longer
be more destructive than the state it exists to clean up.
Verified over the real invocation path rather than a model of it: an
ansible-playbook run using the same `script:` task and docker connection as
compose/swarm/playbook.yml, against three containers in one play, has each node
address its own pool - infinito_ci_race-mgr-01, -wrk-01, -nfs-01 - with
failed=0 throughout. A stubbed-pool harness covers the paths that need a kernel
module this machine does not have: a sibling holding the pool declines with rc
0 instead of ending the run, the same case with a stated pick exits 1 through
the contract, a genuine leftover is still destroyed, and a missing identity is
a hard error. The earlier replacement harness (ext4 recognised, btrfs
replacing it, no loop leaked) still passes against the stricter signature.
Pool names now carry dashes, which zfs_namecheck.c accepts; at 42 characters
they sit far inside the 240 limit and match no reserved prefix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role README claimed the backup machinery "operates transparently against the mount
paths" and demoted the source-of-truth rule to advice in a prose box - both written
before 2efc73d3b made that rule the code's behaviour. It now states what is enforced:
this role skips export-backed volumes, svc-bkp-nfs-2-local is their single capture, and
manager-pinned roles, roles forcing another compose_mode_force and nfs: false opt-outs
stay covered here. It also names the limit the code actually has - the exclusion checks
that some host is in the svc-bkp-nfs-2-local inventory group, not that the role ever ran
successfully - because a README promising more than the code delivers is worse than one
promising less.
Its Schema section carried the role's timer flow; it now carries the mechanism the
change set is about: meta/volumes.yml -> swarm_nfs_backed() as the single predicate ->
its two consumers (the compose rewrite and the backup exclusion) -> the inventory-group
gate -> the export host's single capture -> the restore path with its mountpoint probe
-> the drill's [3b] guard against over-exclusion.
The two swarm READMEs drew the drill with eight steps and pre-773a25ab6 semantics. Both
now draw the nine that exist, including what makes the proof a proof: the pulled tree is
wiped before the device recover, so the device holds the only copy left; the live export
is wiped with the marker proven gone before it is restored; and [9] deletes the marker
server-side before restoring the volume through its bound backing store.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three steps of the DR drill, one theme: every step now proves what it claims.
[9] restored into the volume's mountpoint while the stack was down - a bare directory,
because docker mounts a volume declaring its own backing store only while a container
holds it. The payload landed on the node disk and the real mount shadowed it on the next
deploy: success reported, nothing delivered, and roughly a gigabyte added to the very
device the drill later runs out of. It now deletes the marker server-side, binds the
backing store at the mountpoint, runs the recover CLI through it, and asserts the marker
on the NFS server's own disk. That also exercises the positive branch of the guard from
773a25ab6 - its refusal path is unit-tested, its pass-through was tested nowhere. When
the manager exposes no backing store the branch skips with a reason: docker volume
inspect is node-local, and with replicas: 1 the volume object usually lives elsewhere.
[3b] replaces a proof that used to happen by accident. The double capture 2efc73d3b
removed also showed, incidentally, that the volume unit really writes - the drill found
its marker in the volume repo too. The new step checks it directly: the newest
backup-docker-to-local generation must be at least as new as DRILL_START_TS (taken on
the manager, the same clock that names generations) and hold a non-empty file under a
volume subtree. Its failure message names the suspicion it exists for: an exclusion that
swallowed every volume. With no volume unit deployed it skips, tracked by
VOLUME_TRIGGERED alongside the existing SECRETS_TRIGGERED.
[5] prints du -sm per machine and repo when the space precondition fails, so the next red
run names what fills the pulled tree instead of leaving it to arithmetic.
Four adversarial review rounds found five defects in this block and confirmed each fix:
an NFS mount reconstructed from driver options the codebase never emits (it is a bind),
the recover CLI's second positional passed a volume name where it takes a host, a missing
volume object aborting the drill instead of skipping, and an umount ordered before the
return-code check, which under set -e would have killed the drill on a busy mount with no
diagnostic and left the bind behind.
None of this has run in a swarm yet. Every failure path is loud and names its cause.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pick was made once per matrix entry, in a workflow step ahead of the drill,
and every distro of that entry then ran on it. Coverage followed: one entry
exercised one filesystem, and the swarm never reached zfs at all, because
resolve.sh's userland_for() kept it out of any entry containing arch, centos or
fedora - which is every automatic sweep. The draw now happens inside the loop in
distros.sh, once per distro, so a five-distro entry covers zfs with 87% and all
three kinds with 62% probability instead of one kind with certainty.
The workflow input changes meaning with it. It was a single kind, offered as a
choice in the manual entry; it is now a space-separated allow-list. Empty draws
from all three, and naming kinds narrows the draw to them and marks the pick
required, so a host that cannot deliver one of them fails rather than quietly
running on something else. A name that is not a filesystem this project puts a
data root on is rejected outright instead of reaching the applying step.
For the draw to mean anything the images have to carry the userlands, so
scripts/install/zfs.sh becomes scripts/install/filesystem.sh and installs ext4
and btrfs alongside zfs. That is not symmetric work: btrfs-progs does not exist
in the EL family at all, because RHEL dropped btrfs, and a plain
`dnf install btrfs-progs` there fails the whole transaction and takes e2fsprogs
with it. EPEL carries it, so the EL family goes through EPEL from the start -
split by distro rather than by a failed first attempt, since Fedora has no
epel-release to fall back to and routing a transient failure there would turn it
into a fatal one. verify() now demands all four binaries, so a packaging
regression fails the build instead of leaving a hole the drill discovers later.
Measured on all five bases: debian trixie, ubuntu 26.04, arch, fedora 44 and
centos 10 each end with mkfs.ext4, mkfs.btrfs, zfs and zpool present and ldd
fully resolved.
Two defects only became reachable once a single job could ask for different
filesystems in turn:
* docker_dataroot.sh answered "already prepared" and returned for any mount it
found, so from the second distro on, every draw would have been silently
ignored on a host that survives the sweep - the runner in compose and host
mode. It now compares what is mounted against what was asked for and
dismantles the old root when they differ. Everything docker cached there
goes with it; that is what covering more than one filesystem per entry
costs.
* reclaim_stale_pool ran only when the new pick was zfs, so a zfs to btrfs
transition left the previous distro's pool imported on the shared kernel,
pinning a loop device over an image its dead container had already unlinked.
It now runs on every pick, which needed the /dev/zfs handling lifted out of
the zfs branch into zfs_device_ready().
The restore_docker trap is armed before anything can stop docker, rather than
after, so a failure while dismantling a data root cannot leave a host without a
daemon. resolve.sh splits its allow-list into an array before testing it, so a
whitespace-only input is the empty list it looks like rather than a narrowed one
that yields an empty pick marked required. pick.sh clears both keys when it
declines to draw: they are declared in default.env, so a generated .env carries
whatever was exported when it was made, and a caller gating its apply on the
value would otherwise repoint the developer's own docker in the very case the
skip exists to prevent.
Verified by execution rather than by reading: a privileged container ran
docker_dataroot.sh with real loop devices and mkfs through ext4 -> ext4 -> btrfs
-> ext4, confirming the recognition, the replacement in both directions and that
no loop leaks across it; a second harness with a recording zpool stub confirmed
that a btrfs pick does reclaim a stale pool, that no pool means no destroy, and
that a host without /dev/zfs still completes an ext4 pick without touching zpool.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dockerd's zfs graphdriver resolves `zfs` on PATH and opens /dev/zfs before it
accepts a data root (moby daemon/graphdriver/zfs/zfs.go:54,59), so the userland
has to sit in the image, not on the host. Only the apt images carried it, which
is why resolve.sh's userland_for() kept zfs out of every entry containing arch,
centos or fedora - that is every automatic swarm sweep, so the zfs data root and
the zfs branch of the backup launcher were never once exercised. Run
30781723102 shows it: "zfs left out: not every distro of this entry carries the
userland".
scripts/install/zfs.sh joins the three existing install scripts in the image
build and takes a different route per distro, each one run in a real container
of the matching base image before landing here:
debian trixie 2.3.2 zfsutils-linux, contrib enabled first
ubuntu 26.04 2.4.1 zfsutils-linux
arch 2.4.3 the AUR PKGBUILD, which configures --with-config=user
fedora 44 2.4.3 zfsonlinux repo plus a dummy zfs-kmod provider
centos 10 2.2.10 the same, from the epel path
The dummy is what keeps the RPM route usable: the `zfs` package carries an
ungated `Requires: %{name}-kmod`, so a plain `dnf install zfs` resolves through
zfs-dkms and installs 389 extra packages and 1.44 GB of kernel and toolchain to
build a module against a kernel the container never boots. Providing zfs-kmod
from an empty package keeps the same install at 60 packages with no kernel
package at all, and the transaction is checked afterwards so a regression fails
the build instead of bloating the image. The repository is removed again once
the userland is in, because the dummy pins one version and an upgrade would
otherwise re-resolve the kmod at deploy time.
Version drift against the host module is not a concern: zpool create enables a
feature only when the loaded module reports it through /sys/module/zfs/features
.pool (cmd/zpool/zpool_main.c, module/zcommon/zfeature_common.c), and libzfs
compares no versions at all (lib/libzfs/os/linux/libzfs_util_os.c:123-134), so a
newer userland degrades rather than breaking.
With every image carrying it, userland_for() and every_distro_carries() have
nothing left to decide and go. Two defects that only became reachable once zfs
could be drawn are fixed in the same move:
* /dev/zfs is a misc device, so the old lookup in /proc/devices could never
find its major and the node was never created. A node container's /dev is
populated at creation and udevd is masked, so nothing else fills the gap
either: the first bring-up of every job would have declined, blaming a
module that modprobe had just loaded. The minor now comes from /proc/misc.
* Pools live in the kernel the lab nodes share with the runner, while the
teardown only removes containers, and the node hostnames repeat across the
distro sweep. From the second distro on, zpool create hit the previous
iteration's pool. It is destroyed and its loop detached first.
The node image drops its own apt-only zfs branch and instead masks the six boot
units the userland enables by preset, which on a node can only fail: the pool is
created after boot.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Docker mounts a volume that declares its own backing store (Options set: nfs,
cifs, bind) only while a container holds it. recover.py resolved the volume's
Mountpoint and rsynced into it regardless, so with the consuming stack stopped -
the state its own docstring demands - the payload landed on the node's disk
under the bare mountpoint directory, and the next stack start mounted the real
backing store over it: a restore that reports success, delivers nothing, and
leaves an invisible, unreclaimable orphan behind.
That orphan had weight. In run 30859490335 the swarm drill's [9/9] volume leg
restored ~1 GiB of nextcloud_data onto the manager's 30 GiB loop device in
exactly the minute the device crossed SeaweedFS's 2 GiB minFreeSpace floor -
the last straw on a device already >=93% consumed (the sizing is ea14535f4) -
after which every authenticated Nextcloud request died on an unwritable object
store.
recover.py now probes the mountpoint with `mountpoint -q` (over ssh for
--docker-host) whenever the volume declares Options, and dies naming the
mechanism and the way out instead of writing a single byte. Plain local volumes
without Options keep the old path untouched - their _data directory on the node
disk is the legitimate target.
The drill skips its [9/9] volume leg for such volumes with the reason in the
log: its data restore already happens in [8/9] straight into the NFS export,
which is the restore path the role's README prescribes for NFS-backed volumes
on swarm. The leg never proved anything for them - it only filled the disk.
Three new unit tests pin the contract: unmounted refused, mounted accepted,
remote probed via ssh with the exact command line.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every lab node got an identical 30 GiB loop device because docker_dataroot.sh
defaulted SIZE to 30G and no caller ever stated it. The swarm manager hosts the
in-cluster registry, ~60 containers, and - by deliberate placement, since /data
is NFS-unsafe - the SeaweedFS object store, whose sizing filter claims 14 GiB
of slots plus a 2 GiB minFreeSpace floor. The idle nfs-server node got the same
30 GiB and used 1% of it.
In run 30859490335 the manager's device stood at 30G/30G (98%), SeaweedFS
crossed its floor at 02:04:45 and marked all 14 volumes unwritable, permanently.
Nextcloud in variant 0 stores objects in that SeaweedFS via OBJECTSTORE_S3_*,
so every authenticated page render performed an object write that failed as a
10-attempt assign ladder - 15 requests, 15 bursts of exactly 10, burst end
matching the nginx 500 to the second. That is the "🐝 web-app-nextcloud 0"
failure: 5 addon specs, 3 attempts each, all dead on the post-login page.
The default is gone: SIZE is now a required argument and a missing value dies
naming the call site. The swarm playbook states it per node - 60G for the
manager, 30G for the rest. 60G covers the observed demand (>=31 GiB: a full
device plus the denied writes) with headroom; the image is sparse, so the
raised ceiling costs only what is actually written, and the runner-side
watchdog at 6 GiB free remains the backstop. The compose and host workflows
state their unchanged 30G explicitly.
Not addressed here: what fills the first 28 GiB (registry + image layers +
volumes composition is unmeasured - the rescue artifact holds exactly one df
snapshot), and the drill-side shadow restore that supplied the last ~1 GiB;
both are separate changes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
bdb3b29a9 passed the watchdog floor to the drill as INFINITO_DISK_FLOOR_MB and
broke test-lint:
AssertionError: INFINITO_* keys referenced in code without a registry entry
INFINITO_DISK_FLOOR_MB
The prefix is reserved for the deployment env registry, where every key needs a
default in default.env or a handler that sets it. This value is neither: it is a
handoff from matrix.py to the script matrix.py starts, and it only exists for
the duration of that call. DRILL_EXTRAS, one line above, already goes the same
way under an unprefixed name.
So the key becomes DISK_FLOOR_MB and its declaration moves up beside
DRILL_EXTRAS in the `: "${VAR:?...}"` form the file uses for matrix handoffs,
leaving the INFINITO_* block below to the values that really come from .env.
Registering it in default.env would have satisfied the lint by writing 6144 down
a second time, which is the duplication bdb3b29a9 set out to remove; a nocheck
would have hidden the naming mistake rather than fixed it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
hostname(1) is not installed in every node image. In run 30830377785 the arch
drill printed
trigger_units.sh: line 22: hostname: command not found
starting svc-bkp-local-2-device.11.6.0.infinito.nexus.service on
so the node name came out empty on all three call sites. The cosmetic one is the
progress line; the damaging one is the journal dump path, which becomes
"${unit}..journal.txt" and is identical on every node. Four hosts writing the
same filename into the shared dumps directory means the artifact keeps whichever
one wrote last, and a unit failure gets investigated against another host's
journal.
uname is coreutils and present wherever the drill runs. Reading it once also
drops three subshells from the loop.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill's space check required FREE >= PULLED, but the run is killed from
outside by matrix.py's watchdog as soon as the runner drops below 6G free. The
check knew nothing about that floor, so it waved through a drill that could not
finish: 19008M free against a 13961M pulled tree passes FREE >= PULLED, and the
sync then leaves 5047M, which is under the floor.
That is what happened to the arch swarm drill in run 30830377785. The device
sync reported the marker at 19:16:38.34, the watchdog fired at 19:16:39.73 on
its next 30s tick, and the step died with rc 75 thirteen minutes into work that
was already doomed when the precondition ran.
The requirement is PULLED + FLOOR, not FREE >= PULLED and not 2 * PULLED. FREE
is measured after the pull, so the first copy is already deducted; exactly one
further copy is ever added on top of it - the device copy during the sync, and
later the restore root, by which time base.sh has deleted the pulled tree. The
two extra copies are sequential, never simultaneous.
matrix.py owns the floor now and exports it as INFINITO_DISK_FLOOR_MB. It was
previously a bare `6 * 2**30` the shell had no way to see, which is why the two
sides could disagree about what "enough disk" means.
This makes the failure honest and immediate; it does not create disk. The arch
runner is short by about 1.1G, and closing that needs a capacity decision -
bigger runner, smaller test dataset, or reclaiming build cache before the drill.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A job whose nodes all declined the filesystem tested none of it, and the table
row saying so reads like an ordinary result. That is how the debian arm lost its
zfs coverage without anyone noticing: docker_dataroot.sh exists to exercise the
snapshot mode of svc-bkp-volume-2-local, which engages only on btrfs and zfs,
and on that arm it has been engaging on neither.
The summary now carries an explicit warning when no node applied the requested
filesystem and at least one declined it. Partial coverage is coverage and stays
unflagged; a job with no verdicts at all still exits early.
Deliberately a finding, not a failure. Stating a filesystem instead of `auto`
would set REQUIRED=true in resolve.sh and turn a decline into a hard error, but
that would paint the debian arm red for a cause that is understood and not yet
fixed - the node image has no zfsutils candidate without the package cache. The
goal was never to enforce the pick, only to stop losing it silently.
Exercised against three real job logs from run 30781723102: the debian arm
(5 nodes, all declined) produces the warning; the ubuntu arm (10 nodes, zfs
applied) and an ext4 job do not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard compared free space against USB_SIZE_MB = PULLED_MB*2+256, which is
the nominal size of an image 02_luks_device.sh creates with truncate - sparse,
so nothing ever allocates it. It also ran after that script, i.e. once the image
was already created, formatted and mounted, so the image's own footprint was
already subtracted from the df it read. It charged the pulled tree a second time
while that tree was counted as used, not free. In run 30781723102 it demanded
28108M against 16582M and refused a drill that fits: the peak is 14555M
additional, derived from rescue-diagnostics-swarm-web-app-nextcloud-0, and the
same run's debian job contains a full 9/9 completion of the same drill.
Rather than predicting the image's ext4 overhead - two earlier attempts at that
constant were wrong in both directions - both thresholds are now read:
df on DEV_MOUNT reports what the encrypted filesystem actually offers after
mkfs, and df on DIR_BACKUPS reports what the host still has for the copy the
sync writes into the sparse file and, later, the restore root. Both are compared
against PULLED_MB, a measured quantity. USB_SIZE_MB keeps sizing the image.
Every abort now tears the device down (umount, luksClose, rm -f); a refused
drill previously left the mapping and the image behind.
Exercised against the recorded numbers and six edge cases: 26624/16582/13926
passes where the old guard refused, an undersized device or host is refused, and
an empty or non-numeric df reading is reported as unreadable.
Both thresholds sit exactly at PULLED_MB, so second-order allocation inside the
image (extent trees, directory blocks) is uncovered - tens of MiB against a
13.9 GiB tree. The drill has not been run with this guard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The script stops the daemon at :157-158 and starts it again only on the success
path at :220. decline() exits 0 and set -e exits non-zero, and both walk past
that start, so any other outcome leaves the host without docker. One of the
three callers is test-deploy-host.yml:171, where the script runs on the GitHub
runner itself rather than in a lab node container.
An EXIT trap rather than a restart inside decline(), because decline is not the
only affected exit: arm_autoclear's exit 1 and every set -e abort at mkdir,
truncate, zfs create, mkfs, mount, btrfs subvolume create and the daemon.json
writer sit in the same window. The trap is armed only after the stop, and
DOCKER_STOPPED=false after a successful start makes it a no-op on the green
path. A failed restart is reported rather than swallowed, so it cannot hide the
failure that led there.
This hazard predates 02dc003bb: the loop-device decline added in the commit
before it has the same property. 02dc003bb widened it by putting a second
decline behind the stop.
Verified with a stubbed-PATH harness driving the real script: after a failed
zpool create the call log contains `systemctl start docker`, and it comes after
`losetup -d`, so the teardown still precedes the restart. REQUIRED=true still
exits 1, the success path is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three places where a data-root verdict was produced and then lost.
report.sh matched `requested=[a-z]+`, which never matches `ext4`. Against the
real deploy log of a run that recorded five ext4 verdicts, the old pattern
yields 0 matches and the summary table was silently empty of them; the fixed
pattern aggregates the same five into one row.
docker_dataroot.sh wrote the reason only to GITHUB_STEP_SUMMARY, which is unset
inside the lab node containers - so on exactly the machines that decline, the
reason went nowhere. It now rides on the machine-readable report line too. This
is what leaves the mariadb drill's decline reason unrecoverable from run
30781723102.
resolve.sh's exclusions went to stderr only, i.e. into the deploy log rather
than the job summary.
The reason is emitted as a list below the table, not as a column: reasons embed
per-node data (the pool name carries the hostname, plus the loop device), so
folding them into the `sort | uniq -c` key would split one five-node row into
five one-node rows exactly in the failure case the reason exists to explain.
`reason=[^']*` keeps ansible's closing quote out of the value, and `|` is
replaced so a reason cannot break the markdown table.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
