mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 02:54:26 +00:00
fix(ci): count a budget SIGKILL as skipped, like the SIGTERM it escalates from
ad9222630 wrapped each distro in timeout so a run that outlives the budget stops instead of drifting into the action timeout, and mapped rc 124 to skipped because stopping at the budget is the documented intent rather than a defect. It verified that timeout returns 124 for its own kill. It did not cover the other one: timeout -k escalates to SIGKILL when the command survives SIGTERM, and then reports 137, which is equally its own kill. An ansible run with live docker children survives SIGTERM routinely, so 137 is the common case rather than the exotic one.
The result was that the governor never fired as documented. Fediwall swarm and Nextcloud swarm 0 in run 31756400129 and 31756560361 both ended on [ERROR] Run failed for distro=arch (rc=137) with budget=20400s remaining=-60s. Fediwalls playbook had already printed PLAYBOOK RECAP, the stack had converged and every node port answered; it was booked as a failure 0.3 seconds later.
The two narrowing conditions carry the discrimination and are unchanged. dur >= remaining - 5 rejects a SIGKILL that arrives at an arbitrary time, such as an OOM kill mid-run, because a budget kill lands at the deadline by construction. passed > 0 still requires an earlier distro to have succeeded.
A unit case in tests/unit/scripts/tests/deploy/test_distros.py drives a distro that traps SIGTERM and sleeps. Without the change it reproduces the CI failure locally, exit 137 and Run failed for distro; with it the sweep exits 0 and the summary marks the distro skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
42449ebf0a
commit
5086698096
2 files changed
+13
-1
No files matched your search
@@ -141,7 +141,7 @@ for i in "${!distro_arr[@]}"; do
|
||||
|
||||
echo ">>> Duration: distro=${distro} took ${dur}s (max_seen=${max_seen}s)"
|
||||
|
||||
if [[ $rc -eq 124 ]] && ((dur >= remaining - 5)) && ((passed > 0)); then
|
||||
if [[ $rc -eq 124 || $rc -eq 137 ]] && ((dur >= remaining - 5)) && ((passed > 0)); then
|
||||
statuses[i]="skipped"
|
||||
notes[i]="budget exhausted mid-run after ${dur}s"
|
||||
skipped=$((skipped + 1))
|
||||
|
||||
@@ -118,6 +118,18 @@ class TestDistrosLoop(unittest.TestCase):
|
||||
self.assertEqual(len(record.split()), 1)
|
||||
self.assertRegex(proc.stdout, r"Skipping distro|budget exhausted")
|
||||
|
||||
def test_a_distro_that_ignores_sigterm_is_skipped_not_failed(self) -> None:
|
||||
proc, record = self._run(
|
||||
DISTROS,
|
||||
'echo "${INFINITO_DISTRO}" >> "${RECORD}"\n'
|
||||
'test "$(wc -l < "${RECORD}")" -gt 1 && { trap "" TERM; sleep 300; }\n'
|
||||
"exit 0",
|
||||
INFINITO_CI_DISTRO_BUDGET_SECONDS="8",
|
||||
)
|
||||
self.assertEqual(proc.returncode, 0, proc.stderr)
|
||||
self.assertRegex(proc.stdout, r"Budget exhausted while distro=")
|
||||
self.assertIn("🟦 skipped", self.summary)
|
||||
|
||||
def test_the_job_summary_tables_every_distro_in_execution_order(self) -> None:
|
||||
proc, _record = self._run(
|
||||
DISTROS,
|
||||
|
||||
Reference in new issue
Block a user