Two jobs of run 39040406916 died with the runner's own annotation, 'The
hosted runner lost communication with the server'. Both left every
diagnostic step pending, so no artifact exists for either. For the swarm
one the in-band sampler ruled starvation out — 9324M of 15947M available,
stall60 0.0%, 25 seconds before the cut — and nothing in the repository
can tell the remaining causes apart.
A stuck NFS mount wedges tasks in uninterruptible D state, which shows as
I/O pressure while memory pressure stays flat, and this drill runs its own
NFS node. /proc/pressure/io and /proc/pressure/cpu were read nowhere in
the repository. They are now sampled beside memory, with the D-state count,
every ten seconds instead of thirty, and an I/O stall at or above half the
minute aborts the step while the runner is still alive to upload the
banner.
cpu reads 'some', not 'full': the full line is pinned at 0.00 system-wide,
measured 13.50 against 0.00 on this host, so reading it like memory would
have shipped a metric that can never move.
Dump runner state was switched off for swarm, so the one post-mortem that
survives a step failure never ran there. It runs now, and the swarm drill
gets the step timeout its compose sibling has had all along — 350, the
number test_distro_budget_fits_timeout already derives the distro budget
from.
The sampler moved to its own module: the additions took matrix.py past the
file-size cap, and the lint asks for a split rather than a marker.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When the host runs out of memory the Actions runner loses its heartbeat. GitHub then reports only 'lost communication': no step completes, so every if-always upload step is skipped and the rescue diagnostics, the inventory artifact and the Playwright results are all lost. Job 92710765494 died exactly that way.
The 30s poll in _run already did this for disk. It now also samples MemAvailable, MemTotal and the memory PSI stall, prints one line per poll, and aborts below 6 percent of RAM. The abort returns 75, which propagates to the EXIT trap in 00_one.sh, so the existing post-mortem apparatus runs normally instead of dying with the runner.
The sample line is the main point: artifacts provably do not survive a lost runner, streamed log lines largely do, and the numbers discriminate memory thrash from a kernel wedge on the next occurrence. The floor is relative to MemTotal so a dev box does not inherit a runner-sized threshold; measured margin across four real drills is 9-15x. PSI is sampled but not used as a trigger, since no drill has ever measured it.
mem_total_mb and mem_stall_pct join the existing host probes in utils/env/runtime.py and share the /proc/meminfo parser. The probes read only /proc, never the host-side NFS mount, which would block in D-state under the very condition they exist to catch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replace 8 bash env modules with a single Python generator
(cli.meta.env) that materialises env/static.env + runtime context
into a gitignored .env. Each variable lives in its own handler
under utils/env/handlers/; defaults carry inline one-line `# ...`
comments.
Rename to the INFINITO_* schema:
- workflow inputs: APPS, SERVICES_DISABLED, CMD, DEBUG, WHITELIST,
BUNDLES, FULL_CYCLE, VARIANT, RUN_FLAGS, LIMIT_HOST, PARALLEL
- inventory: INVENTORY_DIR / INVENTORY_FILE / HOST_VARS_FILE,
PW_FILE -> INVENTORY_PASSWORD_FILE
- venv: VENV_BASE / VENV_FALLBACK / VENV_NAME
- new SPOT defaults: COMPILE_SILENCE, CONTAINER_NO_CA,
WAIT_HEALTH_TIMEOUT_S, PACKAGE_CACHE_FRONTEND_INIT_IMAGE,
promoted IMAGE_TAG from GHA-only to general passthrough
compose.yml hardening:
- inline \${VAR:-default} removed; strict \${VAR:?...} everywhere
- new integration lint blocks regressions
Tooling:
- scripts/install/all.sh owns the stamp-file install logic
(Makefile target delegates)
- scripts/meta/env/load.sh is the single shell entry point;
8 bash modules deleted
- new shell lint: no INFINITO_* defaults outside env/static.env
Tests:
- 93 new unit tests for utils/env/{builder,parser,runtime,writer}
- restructure tests/lint into repository/ (file-content rules) vs
filesystem/{shell,python,env}/ (filesystem-shape rules)
- drift + smoke integration tests against the generator
Docs:
- variables.md is now a pointer; per-variable docs live next to
the key in env/static.env / env/ci.env
- compose.yml.md drops the Default column
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5