mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 19:01:54 +00:00
fix(ci): enforce the distro budget while a distro is running
web-app-openwebui 0,1,2 was killed by the 350-minute action timeout with no failed task anywhere in its 1.33M log lines. The sweep admitted fedora at remaining=10372s after debian took 10028s, then never looked again: the budget was only checked between distros. Its internal deadline passed 594 seconds before GitHub killed the job - the run knew it was over time for ten minutes and had no way to act. Worst-case runtime was deadline plus the full duration of the last admitted distro, unbounded. The trigger was a single stalled task (803s for a no-op that takes 1.3s warm, and a fully cold transaction 8 minutes later cost 24s), but that only decided which job hit the wall. The wall itself is structural. remaining is already computed at the top of the loop, so wrapping the call in timeout adds no new state. rc 124 becomes 'skipped' rather than 'failed', because a job that stops at its budget is the documented intent, not a defect - test_distro_budget_fits_timeout.py already asserts this invariant, which until now was simply untrue. Two conditions narrow it, and both matter: dur >= remaining - 5 keeps an inner timeout that fires early as a failure. Verified: timeout returns 124 only for its own kill and passes inner exit codes through unchanged (0 for success, 3 for an inner exit 3). passed > 0 is the one that protects the test result. Without it, a job whose very first distro overruns would report 'skipped' and exit 0 - green while having verified nothing. That would have been a silent false pass across all 243 jobs of a run. The guard was executed over the full truth table, including passed=0. -k 60 is deliberate: one.sh execs into dedicated.sh and its ansible and docker children can outlive SIGTERM. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
9b52a966c0
commit
ad92226307
1 file changed
+12
-1
@@ -46,6 +46,7 @@ echo "=== Global time budget: ${INFINITO_CI_DISTRO_BUDGET_SECONDS}s (deadline ep
|
||||
max_seen=0
|
||||
skipped=0
|
||||
ran=0
|
||||
passed=0
|
||||
durations=()
|
||||
statuses=()
|
||||
seconds=()
|
||||
@@ -124,7 +125,7 @@ for i in "${!distro_arr[@]}"; do
|
||||
distro_start="$(date +%s)"
|
||||
|
||||
set +e
|
||||
"$@"
|
||||
timeout -k 60 "${remaining}s" "$@"
|
||||
rc=$?
|
||||
set -e
|
||||
|
||||
@@ -140,6 +141,15 @@ for i in "${!distro_arr[@]}"; do
|
||||
|
||||
echo ">>> Duration: distro=${distro} took ${dur}s (max_seen=${max_seen}s)"
|
||||
|
||||
if [[ $rc -eq 124 ]] && ((dur >= remaining - 5)) && ((passed > 0)); then
|
||||
statuses[i]="skipped"
|
||||
notes[i]="budget exhausted mid-run after ${dur}s"
|
||||
skipped=$((skipped + 1))
|
||||
ran=$((ran - 1))
|
||||
echo "[WARN] Budget exhausted while distro=${distro} was running; stopping."
|
||||
break
|
||||
fi
|
||||
|
||||
if [[ $rc -ne 0 ]]; then
|
||||
statuses[i]="failed"
|
||||
notes[i]="rc=${rc}"
|
||||
@@ -153,6 +163,7 @@ for i in "${!distro_arr[@]}"; do
|
||||
fi
|
||||
|
||||
statuses[i]="passed"
|
||||
passed=$((passed + 1))
|
||||
done
|
||||
|
||||
render_summary
|
||||
Reference in new issue
Block a user