Moodle#0 of run 33445831564 was cancelled by the 355-minute job cap with almost no
artifacts. The cap did not end a deploy; it reaped a zombie.
00_one.sh wrapped `timeout` around `sudo`, so the guard ran as the unprivileged
runner and the payload as root. An unprivileged SIGKILL cannot reach a root process,
and the same holds for distros.sh's own `timeout -k` one level up. Probed both
shapes: the payload lands in a pgid the outer governor never addresses either way, so
--foreground would not have helped -- the privilege wall is what carries.
The log shows the consequence directly. The governor fired on schedule:
06:18:27.884 distros.sh: line 103: 15914 Killed timeout -k 60 "${remaining}s" "$@"
06:18:27.894 [ERROR] Run failed for distro=ubuntu: rc=137 after 20460s
and then ansible output kept arriving in the same tee'd log -- a TASK at 06:18:27.929,
another at 06:26:00.189, elapsed still advancing -- until
`##[error]The operation was canceled.` The orphan held the step's pipe open, so tee
never saw EOF and the step never returned.
954b9f1ae6 added the TERM trap precisely so a timeout kill would still collect
diagnostics, and its reasoning ("GNU timeout signals the whole process group") is
correct. It simply never gets signalled on the swarm path, because sudo stands
between. That is why this cell carries only inventory- and role-runtimes- while its
compose twin also has rescue-diagnostics- and playwright-. This completes that fix
rather than replacing it.
The ladder was also inverted: INFINITO_CI_DISTRO_BUDGET_SECONDS=20400 (340 min)
against INFINITO_SWARM_STEP_TIMEOUT_MINUTES=350, with default.env:97 asserting the
order as if correct. For the trap to run, the step guard must fire BEFORE the sweep
governor. Rather than re-tune two constants that start their clocks at different
points -- matrix start trails drill start by 1m49s to 2m04s, measured -- 00_one.sh now
derives its ceiling from the governor's deadline minus a teardown reserve, and keeps
its own configured value when that is smaller.
Not fixed here, and it must be said: Moodle#0 does not fit on a hosted runner at all.
Its pass-2 pacing tracks pass-1 within 5%, putting the pass end at roughly 362 minutes
after job start, past both the 355 cap and GitHub's own 360 ceiling. Raising the cap
would be masking and is arithmetically impossible anyway. What this buys is an honest
red cell at ~323 minutes WITH its diagnostics, instead of a silent cancellation with
none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
30% was a guess. Measuring it: across 60 successor distro runs in 22 jobs of run
31756400129, each against the running max of the runs before it, the median
overshoot is 0.991 - a successor is typically slightly faster - and the worst
ever observed is 1.149 (fedora 5541s against a 4822s max_seen).
15% therefore covers every observed case, and nothing above it buys more. It
costs less, too: 30% refuses 23 of 74 start decisions, 15% refuses 18. Five
distro runs that would have fitted, for no additional protection.
The tests read the percentage back out of the log line and check the formula
against it, so they hold at any value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
508669809 booked a budget SIGKILL as skipped so the job went green. That is a
soft-skip: a kill is indistinguishable from the command hanging, so it hid a
real failure behind a green tick.
Fail on it again, and prevent it upstream instead. A distro now starts only
while the remaining budget covers the slowest run so far plus 30%. Fediwall run
31756400129 shows why the bare comparison was not enough: max_seen=6770s against
remaining=6878s let the distro start, and it took 6942s, 64s past the deadline.
max_seen is the maximum of the durations already observed in the same sweep -
a small sample over distros of genuinely different weight, drawn in randomised
order, so it underestimates systematically.
Removing the soft-skip also removed its diagnosis, so a killed distro now
reports both numbers without claiming a cause: the run only knows the exit code
and how much budget was left when it started.
The suite drops from 85s to 4.1s. The 68s case made a process ignore SIGTERM to
force the -k escalation, which exercised coreutils rather than this script; the
124 path reaches the same branch in a second.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ad9222630 wrapped each distro in timeout so a run that outlives the budget stops instead of drifting into the action timeout, and mapped rc 124 to skipped because stopping at the budget is the documented intent rather than a defect. It verified that timeout returns 124 for its own kill. It did not cover the other one: timeout -k escalates to SIGKILL when the command survives SIGTERM, and then reports 137, which is equally its own kill. An ansible run with live docker children survives SIGTERM routinely, so 137 is the common case rather than the exotic one.
The result was that the governor never fired as documented. Fediwall swarm and Nextcloud swarm 0 in run 31756400129 and 31756560361 both ended on [ERROR] Run failed for distro=arch (rc=137) with budget=20400s remaining=-60s. Fediwalls playbook had already printed PLAYBOOK RECAP, the stack had converged and every node port answered; it was booked as a failure 0.3 seconds later.
The two narrowing conditions carry the discrimination and are unchanged. dur >= remaining - 5 rejects a SIGKILL that arrives at an arbitrary time, such as an OOM kill mid-run, because a budget kill lands at the deadline by construction. passed > 0 still requires an earlier distro to have succeeded.
A unit case in tests/unit/scripts/tests/deploy/test_distros.py drives a distro that traps SIGTERM and sleeps. Without the change it reproduces the CI failure locally, exit 137 and Run failed for distro; with it the sweep exits 0 and the summary marks the distro skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
web-app-openwebui 0,1,2 was killed by the 350-minute action timeout with no
failed task anywhere in its 1.33M log lines. The sweep admitted fedora at
remaining=10372s after debian took 10028s, then never looked again: the
budget was only checked between distros. Its internal deadline passed 594
seconds before GitHub killed the job - the run knew it was over time for ten
minutes and had no way to act. Worst-case runtime was deadline plus the full
duration of the last admitted distro, unbounded.
The trigger was a single stalled task (803s for a no-op that takes 1.3s
warm, and a fully cold transaction 8 minutes later cost 24s), but that only
decided which job hit the wall. The wall itself is structural.
remaining is already computed at the top of the loop, so wrapping the call
in timeout adds no new state. rc 124 becomes 'skipped' rather than 'failed',
because a job that stops at its budget is the documented intent, not a
defect - test_distro_budget_fits_timeout.py already asserts this invariant,
which until now was simply untrue.
Two conditions narrow it, and both matter:
dur >= remaining - 5 keeps an inner timeout that fires early as a failure.
Verified: timeout returns 124 only for its own kill and passes inner exit
codes through unchanged (0 for success, 3 for an inner exit 3).
passed > 0 is the one that protects the test result. Without it, a job whose
very first distro overruns would report 'skipped' and exit 0 - green while
having verified nothing. That would have been a silent false pass across all
243 jobs of a run. The guard was executed over the full truth table,
including passed=0.
-k 60 is deliberate: one.sh execs into dedicated.sh and its ansible and
docker children can outlive SIGTERM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An aborted Playwright run - watchdog SIGTERM, task timeout - returns a result dict with neither stdout nor rc. The reporting task dereferenced both, so it became the fatal error itself: the log ended on "object of type 'dict' has no attribute 'stdout'" while the real cause, a disk-exhaustion abort with rc 75, sat eight thousand lines earlier. Guard all three reads, fall back to msg for stderr because that is what Ansible fills on an abort, and treat an absent rc as a failure so the run is recorded and named by the aggregate check rather than vanishing behind a template error.
The distro sweep's summary also mislabelled the distros it never started: they kept the initial "not reached within the budget" note even when the sweep aborted on a failure with 12214 seconds of budget left, pointing at a budget problem that did not exist. Say what happened instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The loop may drop a distro when the budget runs out, and both skip paths fell through to an implicit exit 0, so a run that never touched two distros still reported success. That hid the coverage gap: arch, fedora and centos get no package cache (cache.override.yml is apt-only), are therefore the slowest, and are the first the remaining < max_seen heuristic drops.
Skipping stays non-fatal and the order stays randomised, but every distro now shows up in a GITHUB_STEP_SUMMARY table in execution order, as passed, failed with its rc, or skipped with the reason. The table is also rendered before an aborting exit, since that is the run whose table matters. Two contract tests cover it; the order assertion checks the logged shuf order rather than the input list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The serial distro loop (shuffle, time budget, fast-fail skip, exit on the first failing distro) lived inline in scripts/tests/deploy/ci/all.sh and was therefore reachable only from the compose and host deploys. Swarm and guide have to iterate distros with the same semantics, so the loop moves into scripts/tests/deploy/distros.sh and takes the per-distro command as "$@".
ci/all.sh keeps the app-level preamble and execs the loop with the new ci/one.sh, which resolves the CI image and dispatches to the per-app override or dedicated.sh.
Image resolution leaves the loop as well: scripts/meta/resolve/image/ci.sh renders the reference through cli.meta.ci.image_ref instead of respelling ghcr.io/... locally, which meta/distros.yml declares as the single source.
The two CI time knobs are declared in default.env instead of being repeated as literals, so no caller has to invent a budget.
Three paths are dropped as dead: the empty budget that disabled the limit, the failed counter that could only ever print 0, and the branch for an unset deadline.
Both new scripts are pinned by contract tests. Those tests clear BASH_ENV because the test container exports it as the env loader, which would otherwise re-inject every stripped key and make the assertions pass for the wrong reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5