mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 02:54:26 +00:00
kevinveenbirkenbachandClaude Fable 5
c51c0de9ba
Replace scripts/system/diagnostics/rescue.sh with utils/diagnostics/ container.py: one workflow call on failure captures the current level (containers with logs+inspect, swarm services, journal, host resources, in-play role dumps) and recurses itself through every DiD boundary via self-copy, from the runner down to the innermost runtime (RESCUE_MAX_DEPTH). compose and swarm now share an identical failure step; the swarm per-node collector wrapper is gone. All 17 script-only Ansible rescue: blocks are deleted and their bare block: wrappers inlined (plus a wordpress leftover); the three justified blocks (postgres pg_hba, xwiki extensions, keycloak cross-node recovery) keep their evidence tasks and now end in a fail task. Two lints enforce the state: rescue: is forbidden without a '# nocheck: rescue-<reason>' first-task marker, and a block without rescue/always/when or another scoping key must be inlined. sys-service-terminator hoists become to the include apply. Also move routine/08_cleanup.sh to utils/clean/teardown.sh and topology/ to utils/topology/ so routine/ holds only the sequenced flow. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
54 lines
2.9 KiB
Markdown
54 lines
2.9 KiB
Markdown
# Diagnostics: recursive container rescue snapshot
|
|
|
|
`container.py` is the single failure-diagnostics entry point. The CI
|
|
workflows (compose + swarm) call it once on the runner when a job fails;
|
|
it captures the current level and then recurses itself through every
|
|
Docker-in-Docker boundary, so one call collects evidence from the
|
|
outermost runner down to the innermost application container. It always
|
|
exits 1.
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
subgraph entry["🩺 One workflow call on failure()"]
|
|
wf["⚙️ CI step (compose + swarm identical)<br/>INFINITO_RESCUE_DIAGNOSTICS_DIR=/tmp/rescue-diagnostics/<app><br/>python3 utils/diagnostics/container.py <app> <context>"]:::infra
|
|
end
|
|
|
|
subgraph level["📸 Per-level capture (repeated on every depth)"]
|
|
direction TB
|
|
host["🖥️ Host facts<br/>meta, uname, df/free/uptime,<br/>journal, dmesg OOM evidence"]:::store
|
|
dumps["📂 local-dumps/<br/>in-play role evidence from<br/>DIR_RESCUE_DIAGNOSTICS (pg_hba, xwiki)"]:::store
|
|
rtc["🐳 Runtime state<br/>info, stats, ps -a,<br/>per-container logs + inspect"]:::store
|
|
svc["🐝 Swarm services<br/>ls, per-service ps + logs"]:::store
|
|
end
|
|
|
|
subgraph recursion["🔁 Recursion (RESCUE_MAX_DEPTH, default 3)"]
|
|
direction TB
|
|
probe["🔍 For every RUNNING container:<br/>has python3 + docker/container runtime?"]:::step
|
|
copy["📤 Copy itself in<br/>(stdin → /tmp/rescue-self.py)"]:::step
|
|
nested["🩺 Run nested capture<br/>RESCUE_DEPTH+1, own staging dir"]:::step
|
|
pull["📥 tar the nested snapshot back to<br/>containers/<name>/nested/ + clean up"]:::step
|
|
probe -->|yes| copy --> nested --> pull
|
|
probe -.->|no: plain app container,<br/>logs already captured| skipped["⏭️ skip"]:::skip
|
|
end
|
|
|
|
wf --> level --> recursion
|
|
nested -.->|same program,<br/>one level deeper| level
|
|
|
|
result["🗂️ Artifact tree (uploaded by CI)<br/><app>-<stamp>-<pid>/<br/>├── meta/system/journal/…<br/>├── local-dumps/…<br/>├── containers/<name>.log + .inspect.json<br/>├── containers/<did-node>/nested/<app>-…/ (recursion)<br/>└── services/<svc>.log + .ps.txt"]:::store
|
|
recursion --> result
|
|
|
|
classDef infra fill:#f3f7fb,stroke:#d6e2ee,color:#455a64;
|
|
classDef store fill:#f1f8f6,stroke:#d5e7e2,color:#455a64;
|
|
classDef step fill:#fbf6f0,stroke:#eee1cf,color:#455a64;
|
|
classDef skip fill:#f7f8f9,stroke:#e3e6e9,color:#455a64;
|
|
style entry fill:#ffffff,stroke:#e6e6e6;
|
|
style level fill:#ffffff,stroke:#e6e6e6;
|
|
style recursion fill:#ffffff,stroke:#e6e6e6;
|
|
```
|
|
|
|
Ansible `rescue:` blocks are forbidden
|
|
(`tests/lint/ansible/structure/test_no_rescue_blocks.py`); the rare
|
|
justified block (`# nocheck: rescue-<reason>`) leaves its role-specific
|
|
evidence in `DIR_RESCUE_DIAGNOSTICS`, which every capture level ships as
|
|
`local-dumps/`.
|