mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-09 20:26:33 +00:00
kevinveenbirkenbachandClaude Opus 5
cfe20f0726
The main merge left two step watchers: run.run_watched from main and the newer watchdog.run_watched from here. Their disk and memory floors are byte-identical (6144M, 6% of RAM, abort 75), so run.py now keeps only run_step and takes the watcher from watchdog. mesh.py reaches it through run_step and matrix.py already imported it directly. That alone would have been a regression. The watchdog aborts on io60 alone, which its own docstring says cannot separate a wedged mount from a busy disk, and the six mesh steps had no I/O kill before: a healthy sleep on a host with 126G free and no task in D state died at rc=75 after 10s where the old watcher let it finish. The first sample also lands 10s in, so 50 of the 60 seconds it averages belong to whatever ran before -- an image pull could abort its successor before that step read a byte. The ceiling now takes three things together: a task in D state on the same sample, WEDGE_SAMPLES breaches in a row, and a step old enough to own a whole STALL_WINDOW_S of the average. Both refuted scenarios are regression tests. A real wedge is caught about a minute later than before; the disk floor that covers ENOSPC is an instantaneous read and is untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
29 lines
724 B
Python
29 lines
724 B
Python
from __future__ import annotations
|
|
|
|
import time
|
|
|
|
from utils.tests.swarm.watchdog import run_watched
|
|
|
|
__all__ = ["run_step", "run_watched"]
|
|
|
|
|
|
def run_step(cmd: list[str], *, env: dict[str, str], label: str) -> int:
|
|
"""Announce a matrix step, run it, and report what it cost.
|
|
|
|
Args:
|
|
cmd: argv of the step.
|
|
env: environment the step runs with.
|
|
label: phase name for the banners.
|
|
|
|
Returns:
|
|
The step's exit code.
|
|
"""
|
|
print(f"=== swarm-matrix: {label} ===", flush=True)
|
|
started = time.monotonic()
|
|
rc = run_watched(cmd, env=env)
|
|
print(
|
|
f"=== swarm-matrix: {label} took {time.monotonic() - started:.0f}s (rc={rc}) ===",
|
|
flush=True,
|
|
)
|
|
return rc
|