fix(ci): enforce the distro budget while a distro is running

web-app-openwebui 0,1,2 was killed by the 350-minute action timeout with no
failed task anywhere in its 1.33M log lines. The sweep admitted fedora at
remaining=10372s after debian took 10028s, then never looked again: the
budget was only checked between distros. Its internal deadline passed 594
seconds before GitHub killed the job - the run knew it was over time for ten
minutes and had no way to act. Worst-case runtime was deadline plus the full
duration of the last admitted distro, unbounded.

The trigger was a single stalled task (803s for a no-op that takes 1.3s
warm, and a fully cold transaction 8 minutes later cost 24s), but that only
decided which job hit the wall. The wall itself is structural.

remaining is already computed at the top of the loop, so wrapping the call
in timeout adds no new state. rc 124 becomes 'skipped' rather than 'failed',
because a job that stops at its budget is the documented intent, not a
defect - test_distro_budget_fits_timeout.py already asserts this invariant,
which until now was simply untrue.

Two conditions narrow it, and both matter:

dur >= remaining - 5 keeps an inner timeout that fires early as a failure.
Verified: timeout returns 124 only for its own kill and passes inner exit
codes through unchanged (0 for success, 3 for an inner exit 3).

passed > 0 is the one that protects the test result. Without it, a job whose
very first distro overruns would report 'skipped' and exit 0 - green while
having verified nothing. That would have been a silent false pass across all
243 jobs of a run. The guard was executed over the full truth table,
including passed=0.

-k 60 is deliberate: one.sh execs into dedicated.sh and its ansible and
docker children can outlive SIGTERM.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kevinveenbirkenbachandClaude Opus 5 committed 2026-08-06 10:47:41 +02:00
1 parent 9b52a966c0
commit ad92226307
1 file changed
+12 -1
+12 -1
View File
@@ -46,6 +46,7 @@ echo "=== Global time budget: ${INFINITO_CI_DISTRO_BUDGET_SECONDS}s (deadline ep
max_seen=0
skipped=0
ran=0
passed=0
durations=()
statuses=()
seconds=()
@@ -124,7 +125,7 @@ for i in "${!distro_arr[@]}"; do
distro_start="$(date +%s)"
set +e
"$@"
timeout -k 60 "${remaining}s" "$@"
rc=$?
set -e
@@ -140,6 +141,15 @@ for i in "${!distro_arr[@]}"; do
echo ">>> Duration: distro=${distro} took ${dur}s (max_seen=${max_seen}s)"
if [[ $rc -eq 124 ]] && ((dur >= remaining - 5)) && ((passed > 0)); then
statuses[i]="skipped"
notes[i]="budget exhausted mid-run after ${dur}s"
skipped=$((skipped + 1))
ran=$((ran - 1))
echo "[WARN] Budget exhausted while distro=${distro} was running; stopping."
break
fi
if [[ $rc -ne 0 ]]; then
statuses[i]="failed"
notes[i]="rc=${rc}"
@@ -153,6 +163,7 @@ for i in "${!distro_arr[@]}"; do
fi
statuses[i]="passed"
passed=$((passed + 1))
done
render_summary