0 Fork 0

mirror of https://github.com/kevinveenbirkenbach/computer-playbook.git synced 2026-11-06 10:59:59 +00:00

Files

T

kevinveenbirkenbachandClaude Opus 5 56b8f1716b fix(diagnostics): let the in-zone probes read the drop-in that holds the zone

d14354e892 added both probes reading only /etc/dnsmasq.conf, but svc-net-tor writes its onion zone to /etc/dnsmasq.d/tor-onion.conf and leaves the main file with nothing but a conf-dir pointer. Every Tor host of run 33698844864 therefore captured resolve-inzone.txt as '[no output, exit 2]' and all 471 sampler lines as inzone=unconfigured: during the resolver outage that ended Gitea#0, nobody ever asked dnsmasq for an onion name, and two investigations went without the one measurement that would have settled them.

Both probes now follow the conf-dir pointer, main file first so a lab node with its zone in the main config answers as before. Each is exercised by a test that runs it against real files rather than comparing its text, which the rescue constant had no test for at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-09-03 12:50:58 +02:00

__init__.py

…

container.py

fix(diagnostics): let the in-zone probes read the drop-in that holds the zone

2026-09-03 12:50:58 +02:00

README.md

…

README.md

Diagnostics: recursive container rescue snapshot

container.py is the single failure-diagnostics entry point. The CI workflows (compose + swarm) call it once on the runner when a job fails; it captures the current level and then recurses itself through every Docker-in-Docker boundary, so one call collects evidence from the outermost runner down to the innermost application container. It always exits 1.

flowchart TB
    subgraph entry["🩺 One workflow call on failure()"]
        wf["⚙️ CI step (compose + swarm identical)<br/>INFINITO_RESCUE_DIAGNOSTICS_DIR=/tmp/rescue-diagnostics/&lt;app&gt;<br/>python3 utils/diagnostics/container.py &lt;app&gt; &lt;context&gt;"]:::infra
    end

    subgraph level["📸 Per-level capture (repeated on every depth)"]
        direction TB
        host["🖥️ Host facts<br/>meta, uname, df/free/uptime,<br/>journal, dmesg OOM evidence"]:::store
        dumps["📂 local-dumps/<br/>in-play role evidence from<br/>DIR_RESCUE_DIAGNOSTICS (pg_hba, xwiki)"]:::store
        rtc["🐳 Runtime state<br/>info, stats, ps -a,<br/>per-container logs + inspect"]:::store
        svc["🐝 Swarm services<br/>ls, per-service ps + logs"]:::store
    end

    subgraph recursion["🔁 Recursion (RESCUE_MAX_DEPTH, default 3)"]
        direction TB
        probe["🔍 For every RUNNING container:<br/>has python3 + docker/container runtime?"]:::step
        copy["📤 Copy itself in<br/>(stdin → /tmp/rescue-self.py)"]:::step
        nested["🩺 Run nested capture<br/>RESCUE_DEPTH+1, own staging dir"]:::step
        pull["📥 tar the nested snapshot back to<br/>containers/&lt;name&gt;/nested/ + clean up"]:::step
        probe -->|yes| copy --> nested --> pull
        probe -.->|no: plain app container,<br/>logs already captured| skipped["⏭️ skip"]:::skip
    end

    wf --> level --> recursion
    nested -.->|same program,<br/>one level deeper| level

    result["🗂️ Artifact tree (uploaded by CI)<br/>&lt;app&gt;-&lt;stamp&gt;-&lt;pid&gt;/<br/>├── meta/system/journal/…<br/>├── local-dumps/…<br/>├── containers/&lt;name&gt;.log + .inspect.json<br/>├── containers/&lt;did-node&gt;/nested/&lt;app&gt;-…/ (recursion)<br/>└── services/&lt;svc&gt;.log + .ps.txt"]:::store
    recursion --> result

    classDef infra fill:#f3f7fb,stroke:#d6e2ee,color:#455a64;
    classDef store fill:#f1f8f6,stroke:#d5e7e2,color:#455a64;
    classDef step fill:#fbf6f0,stroke:#eee1cf,color:#455a64;
    classDef skip fill:#f7f8f9,stroke:#e3e6e9,color:#455a64;
    style entry fill:#ffffff,stroke:#e6e6e6;
    style level fill:#ffffff,stroke:#e6e6e6;
    style recursion fill:#ffffff,stroke:#e6e6e6;

Ansible rescue: blocks are forbidden (tests/lint/ansible/structure/test_no_rescue_blocks.py); the rare justified block (# nocheck: rescue-<reason>) leaves its role-specific evidence in DIR_RESCUE_DIAGNOSTICS, which every capture level ships as local-dumps/.