kevinveenbirkenbachandClaude Opus 5 3e9deab6f7 test(swarm): abort the drill before the runner dies ...

When the host runs out of memory the Actions runner loses its heartbeat. GitHub then reports only 'lost communication': no step completes, so every if-always upload step is skipped and the rescue diagnostics, the inventory artifact and the Playwright results are all lost. Job 92710765494 died exactly that way.

The 30s poll in _run already did this for disk. It now also samples MemAvailable, MemTotal and the memory PSI stall, prints one line per poll, and aborts below 6 percent of RAM. The abort returns 75, which propagates to the EXIT trap in 00_one.sh, so the existing post-mortem apparatus runs normally instead of dying with the runner.

The sample line is the main point: artifacts provably do not survive a lost runner, streamed log lines largely do, and the numbers discriminate memory thrash from a kernel wedge on the next occurrence. The floor is relative to MemTotal so a dev box does not inherit a runner-sized threshold; measured margin across four real drills is 9-15x. PSI is sampled but not used as a trigger, since no drill has ever measured it.

mem_total_mb and mem_stall_pct join the existing host probes in utils/env/runtime.py and share the /proc/meminfo parser. The probes read only /proc, never the host-side NFS mount, which would block in D-state under the very condition they exist to catch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-07 17:05:49 +02:00

..

swarm

test(swarm): abort the drill before the runner dies

2026-08-07 17:05:49 +02:00

__init__.py

…