0 Fork 0

mirror of https://github.com/kevinveenbirkenbach/computer-playbook.git synced 2026-11-09 20:26:33 +00:00

Files

T

kevinveenbirkenbachandClaude Opus 5 392972de9c feat(diagnostics): capture what the two unsolved CI failures needed

The dnsmasq and apt-resolution investigations both stalled on facts the
snapshot never recorded.

The snapshot is taken by docker exec inside the DiD container and the
recursion only descends, so the runner's own docker - where the resolver
runs as ${INFINITO_RUNNER_PREFIX}-coredns - was structurally invisible.
dedicated.sh now runs a second pass on the CI host, with RESCUE_SEEN
seeded with the DiD id so it does not collect the first snapshot twice.

For the rest: a process table, so the MainPID that service-state reports
can be checked against something; the dnsmasq drop-in directory, because
this project's config lives at /etc/dnsmasq.d/tor-onion.conf and never in
the main file; nsswitch.conf and hosts, without which a getent verdict is
ambiguous; one probe per name plus, where the distro ships dig or
nslookup, one per nameserver, so a refusing resolver is distinguishable
from an unreachable mirror; the full firewall table set, since a dropped
query is recorded in filter and -t nat was actively hiding it; routes and
neighbours; os-release, because the matrix is distro-shaped and a bundle
could not say which leg it came from; the runtime's networks, which hold
the forwarder behind every container's 127.0.0.11; free inodes, which a
df -h reads as free space; and the runtime's disk, volume and image
inventory, the last with full references, since an engine was once picked
from a registry host embedded in an image name.

ps runs without args: the bundle is uploaded as an artifact and argv
carries whatever credentials a process was invoked with - a real proxy
password showed up in the first live run of this capture.

Two exec budgets are corrected while the per-container cost grows to
three calls. The per-container journal was borrowing the 600s meant for a
host boot journal, so one wedged container could cost 840s; container
journals span a deploy, not a boot. The cheap questions get 30s, and the
one collection site that had no outer bound gets the timeout 1500 the
others already carry.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-08-16 03:38:42 +02:00

__init__.py

refactor(diagnostics): recursive workflow-level rescue snapshot, no in-play rescue blocks

2026-07-12 03:15:53 +02:00

container.py

feat(diagnostics): capture what the two unsolved CI failures needed

2026-08-16 03:38:42 +02:00

README.md

refactor(diagnostics): recursive workflow-level rescue snapshot, no in-play rescue blocks

2026-07-12 03:15:53 +02:00

README.md

Diagnostics: recursive container rescue snapshot

container.py is the single failure-diagnostics entry point. The CI workflows (compose + swarm) call it once on the runner when a job fails; it captures the current level and then recurses itself through every Docker-in-Docker boundary, so one call collects evidence from the outermost runner down to the innermost application container. It always exits 1.

flowchart TB
    subgraph entry["🩺 One workflow call on failure()"]
        wf["⚙️ CI step (compose + swarm identical)<br/>INFINITO_RESCUE_DIAGNOSTICS_DIR=/tmp/rescue-diagnostics/&lt;app&gt;<br/>python3 utils/diagnostics/container.py &lt;app&gt; &lt;context&gt;"]:::infra
    end

    subgraph level["📸 Per-level capture (repeated on every depth)"]
        direction TB
        host["🖥️ Host facts<br/>meta, uname, df/free/uptime,<br/>journal, dmesg OOM evidence"]:::store
        dumps["📂 local-dumps/<br/>in-play role evidence from<br/>DIR_RESCUE_DIAGNOSTICS (pg_hba, xwiki)"]:::store
        rtc["🐳 Runtime state<br/>info, stats, ps -a,<br/>per-container logs + inspect"]:::store
        svc["🐝 Swarm services<br/>ls, per-service ps + logs"]:::store
    end

    subgraph recursion["🔁 Recursion (RESCUE_MAX_DEPTH, default 3)"]
        direction TB
        probe["🔍 For every RUNNING container:<br/>has python3 + docker/container runtime?"]:::step
        copy["📤 Copy itself in<br/>(stdin → /tmp/rescue-self.py)"]:::step
        nested["🩺 Run nested capture<br/>RESCUE_DEPTH+1, own staging dir"]:::step
        pull["📥 tar the nested snapshot back to<br/>containers/&lt;name&gt;/nested/ + clean up"]:::step
        probe -->|yes| copy --> nested --> pull
        probe -.->|no: plain app container,<br/>logs already captured| skipped["⏭️ skip"]:::skip
    end

    wf --> level --> recursion
    nested -.->|same program,<br/>one level deeper| level

    result["🗂️ Artifact tree (uploaded by CI)<br/>&lt;app&gt;-&lt;stamp&gt;-&lt;pid&gt;/<br/>├── meta/system/journal/…<br/>├── local-dumps/…<br/>├── containers/&lt;name&gt;.log + .inspect.json<br/>├── containers/&lt;did-node&gt;/nested/&lt;app&gt;-…/ (recursion)<br/>└── services/&lt;svc&gt;.log + .ps.txt"]:::store
    recursion --> result

    classDef infra fill:#f3f7fb,stroke:#d6e2ee,color:#455a64;
    classDef store fill:#f1f8f6,stroke:#d5e7e2,color:#455a64;
    classDef step fill:#fbf6f0,stroke:#eee1cf,color:#455a64;
    classDef skip fill:#f7f8f9,stroke:#e3e6e9,color:#455a64;
    style entry fill:#ffffff,stroke:#e6e6e6;
    style level fill:#ffffff,stroke:#e6e6e6;
    style recursion fill:#ffffff,stroke:#e6e6e6;

Ansible rescue: blocks are forbidden (tests/lint/ansible/structure/test_no_rescue_blocks.py); the rare justified block (# nocheck: rescue-<reason>) leaves its role-specific evidence in DIR_RESCUE_DIAGNOSTICS, which every capture level ships as local-dumps/.