0 Fork 0

mirror of https://github.com/kevinveenbirkenbach/computer-playbook.git synced 2026-11-08 11:46:35 +00:00

Files

T

kevinveenbirkenbachandClaude Opus 5 259ee8379e fix(sys-svc-container): keep a timed-out exec in the caller's process group

coreutils timeout without --foreground calls setpgid, so the command it
wraps leaves the caller's process group. A wrapped command that then
reads the controlling terminal takes SIGTTIN and stops forever instead
of being killed at the deadline, which inverts the purpose of the
wrapper.

That is the cause behind d31b49e30d: container exec wraps docker in
this prefix, an ansible.builtin.script task gives the remote shell a
pty, and docker exec -i stops on its first read. Three jobs of run
39186571513 idled 1h37m to 2h22m before the outer budget killed them.
The scripts there close their own stdin; this closes the class.

utils/diagnostics/container.py gets the flag too. docker events reads
no stdin and was never at risk, but leaving one non-compliant
construction would have forced an exemption on the new rule's first
day.

The rule itself cannot be a text search. The prefix comes from a
function and the -i from the caller's argv, so timeout and the command
that reads stdin never meet in one line of source: a grep over the
tree scores zero hits both before and after the defect. The lint reads
the parsed tree instead and requires --foreground in any list literal
that starts with timeout. Checked against the pre-fix line, it reports
it.

Cost, deliberately accepted: --foreground signals the direct child
only, not a process tree. For docker exec the direct child is the CLI,
which is what has to die to release the task, but a wrapped command
that spawns children now leaves them where the group kill would have
taken them.  covers the case where that
trade goes the other way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-10-20 19:06:49 +02:00

__init__.py

refactor(diagnostics): recursive workflow-level rescue snapshot, no in-play rescue blocks

2026-07-12 03:15:53 +02:00

container.py

fix(sys-svc-container): keep a timed-out exec in the caller's process group

2026-10-20 19:06:49 +02:00

README.md

refactor(diagnostics): recursive workflow-level rescue snapshot, no in-play rescue blocks

2026-07-12 03:15:53 +02:00

README.md

Diagnostics: recursive container rescue snapshot

container.py is the single failure-diagnostics entry point. The CI workflows (compose + swarm) call it once on the runner when a job fails; it captures the current level and then recurses itself through every Docker-in-Docker boundary, so one call collects evidence from the outermost runner down to the innermost application container. It always exits 1.

flowchart TB
    subgraph entry["🩺 One workflow call on failure()"]
        wf["⚙️ CI step (compose + swarm identical)<br/>INFINITO_RESCUE_DIAGNOSTICS_DIR=/tmp/rescue-diagnostics/&lt;app&gt;<br/>python3 utils/diagnostics/container.py &lt;app&gt; &lt;context&gt;"]:::infra
    end

    subgraph level["📸 Per-level capture (repeated on every depth)"]
        direction TB
        host["🖥️ Host facts<br/>meta, uname, df/free/uptime,<br/>journal, dmesg OOM evidence"]:::store
        dumps["📂 local-dumps/<br/>in-play role evidence from<br/>DIR_RESCUE_DIAGNOSTICS (pg_hba, xwiki)"]:::store
        rtc["🐳 Runtime state<br/>info, stats, ps -a,<br/>per-container logs + inspect"]:::store
        svc["🐝 Swarm services<br/>ls, per-service ps + logs"]:::store
    end

    subgraph recursion["🔁 Recursion (RESCUE_MAX_DEPTH, default 3)"]
        direction TB
        probe["🔍 For every RUNNING container:<br/>has python3 + docker/container runtime?"]:::step
        copy["📤 Copy itself in<br/>(stdin → /tmp/rescue-self.py)"]:::step
        nested["🩺 Run nested capture<br/>RESCUE_DEPTH+1, own staging dir"]:::step
        pull["📥 tar the nested snapshot back to<br/>containers/&lt;name&gt;/nested/ + clean up"]:::step
        probe -->|yes| copy --> nested --> pull
        probe -.->|no: plain app container,<br/>logs already captured| skipped["⏭️ skip"]:::skip
    end

    wf --> level --> recursion
    nested -.->|same program,<br/>one level deeper| level

    result["🗂️ Artifact tree (uploaded by CI)<br/>&lt;app&gt;-&lt;stamp&gt;-&lt;pid&gt;/<br/>├── meta/system/journal/…<br/>├── local-dumps/…<br/>├── containers/&lt;name&gt;.log + .inspect.json<br/>├── containers/&lt;did-node&gt;/nested/&lt;app&gt;-…/ (recursion)<br/>└── services/&lt;svc&gt;.log + .ps.txt"]:::store
    recursion --> result

    classDef infra fill:#f3f7fb,stroke:#d6e2ee,color:#455a64;
    classDef store fill:#f1f8f6,stroke:#d5e7e2,color:#455a64;
    classDef step fill:#fbf6f0,stroke:#eee1cf,color:#455a64;
    classDef skip fill:#f7f8f9,stroke:#e3e6e9,color:#455a64;
    style entry fill:#ffffff,stroke:#e6e6e6;
    style level fill:#ffffff,stroke:#e6e6e6;
    style recursion fill:#ffffff,stroke:#e6e6e6;

Ansible rescue: blocks are forbidden (tests/lint/ansible/structure/test_no_rescue_blocks.py); the rare justified block (# nocheck: rescue-<reason>) leaves its role-specific evidence in DIR_RESCUE_DIAGNOSTICS, which every capture level ships as local-dumps/.