d14354e892 added both probes reading only /etc/dnsmasq.conf, but svc-net-tor writes its onion zone to /etc/dnsmasq.d/tor-onion.conf and leaves the main file with nothing but a conf-dir pointer. Every Tor host of run 33698844864 therefore captured resolve-inzone.txt as '[no output, exit 2]' and all 471 sampler lines as inzone=unconfigured: during the resolver outage that ended Gitea#0, nobody ever asked dnsmasq for an onion name, and two investigations went without the one measurement that would have settled them.
Both probes now follow the conf-dir pointer, main file first so a lab node with its zone in the main config answers as before. Each is exercised by a test that runs it against real files rather than comparing its text, which the rescue constant had no test for at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_APP_LOG_TAIL filtered with -size -20M. That is GNU syntax; busybox rejects the
suffix and aborts the whole find, not just the test:
$ docker run --rm busybox find /etc -type f -size -20M
find: invalid number '20M'
$ docker run --rm busybox find /etc -type f -size -20971520c
/etc/localtime
/etc/group
So every alpine-based container returned an empty capture. -20971520c is the same
bound in bytes, and c is the only suffix POSIX requires; both finds take it.
The 2>/dev/null goes with it, and that is the part worth naming: capture() at
container.py:76 already merges stdout and stderr, so the redirect is exactly what
swallowed `find: invalid number '20M'`. The collector reported success while finding
nothing.
Two investigations today paid for this. The dnf cause on LittleJS#0 could not be
settled because /var/log/dnf5.log was absent, and the discourse OIDC failure in
Nextcloud#7 had to be reconstructed from the reverse-proxy log because
/var/www/discourse/log/production.log was absent -- while two vendor mkmf.log files
from the same tree were present, proving the directory was traversed. Both times the
missing file read as a property of the situation; it was a property of the tool.
Nothing depends on the constant: grep over tests/, scripts/ and .github/ finds no
reference to _APP_LOG_TAIL or app-logs.
Introduced yesterday in 8434d35d82, so it has never worked on an alpine container.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The collector took `logs` from every container, which is stdout and stderr and
nothing else. An application that logs to a file left the bundle with a status code
and no trace of why.
That is what stopped the snipe-it investigation in run 33317483326. Its MCP adapter
reported `UNREACHABLE read call snipeit_get_license_summary: the provider answered
500`, and Laravel had written the exception to storage/logs inside the container,
where `logs` never looks. Three separate root causes were proposed from static
reading and all three were refuted -- the relation exists, inherited from SnipeModel;
the group permissions are stored exactly as decodePermissions expects; the controller
is identical at the deployed tag -- and the job is still unexplained.
The collector already execs into every container for resolv.conf, systemctl and
journalctl, so the gap was the application's own files, not the ability to reach them.
find drives the capture through -exec rather than a loop over $(find ...): the first
version split on whitespace and broke on paths containing spaces, which a probe
against a directory with two of them showed immediately.
The per-container and per-service captures fold into tables because the file sits at
the 500-line limit. Splitting it is not an option: this script copies itself into
every container it inspects and runs there standalone, so an import of a sibling
module resolves to nothing -- an attempt at that broke seventeen tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 32118850138 lost the :53 listener, so the sampler counts listeners.
Run 32538854067 lost DNS with the listener still bound: sockets-udp.txt
showed dnsmasq holding 0.0.0.0:53 while every forwarded query timed out,
and the sampler reported that hour as healthy.
Each tick now also records what a glibc caller gets back — for the zone
this resolver owns, derived from its own address= line, and for one name
it must forward — with the latency that caller paid. The rescue collector
probes the in-zone name too: probing external names alone cannot separate
"forwarding is gone" from "dnsmasq is gone".
The sampler now also runs on the swarm lab DNS nodes; it only ever ran in
the compose path, which is why the Magento artifact carried no sampler log
at all. Its location comes from group_vars FILE_DNS53_SAMPLER_LOG and is
required at every call site, so a run cannot quietly collect nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Run 32118850138 left a dnsmasq that was alive with zero sockets,
nothing logged, and a 6m39s death window between two host DNS
consumers. Capture each dnsmasq's fd table and rlimits in the rescue
snapshot (closed listeners vs exhausted fd table vs lost netlink
watch), and sample the :53 listener set every 10s inside the DiD
container so the next occurrence is bounded to one bucket.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A Tor node rewrites resolv.conf to the local dnsmasq alone, so the
rescue bundle of run 32118850138 could not prove whether the pinned
upstream (172.30.0.53) was alive while that dnsmasq sat silent on a
dead :53. Probe the daemon.json dns servers too, deduped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dnsmasq and apt-resolution investigations both stalled on facts the
snapshot never recorded.
The snapshot is taken by docker exec inside the DiD container and the
recursion only descends, so the runner's own docker - where the resolver
runs as ${INFINITO_RUNNER_PREFIX}-coredns - was structurally invisible.
dedicated.sh now runs a second pass on the CI host, with RESCUE_SEEN
seeded with the DiD id so it does not collect the first snapshot twice.
For the rest: a process table, so the MainPID that service-state reports
can be checked against something; the dnsmasq drop-in directory, because
this project's config lives at /etc/dnsmasq.d/tor-onion.conf and never in
the main file; nsswitch.conf and hosts, without which a getent verdict is
ambiguous; one probe per name plus, where the distro ships dig or
nslookup, one per nameserver, so a refusing resolver is distinguishable
from an unreachable mirror; the full firewall table set, since a dropped
query is recorded in filter and -t nat was actively hiding it; routes and
neighbours; os-release, because the matrix is distro-shaped and a bundle
could not say which leg it came from; the runtime's networks, which hold
the forwarder behind every container's 127.0.0.11; free inodes, which a
df -h reads as free space; and the runtime's disk, volume and image
inventory, the last with full references, since an engine was once picked
from a registry host embedded in an image name.
ps runs without args: the bundle is uploaded as an artifact and argv
carries whatever credentials a process was invoked with - a real proxy
password showed up in the first live run of this capture.
Two exec budgets are corrected while the per-container cost grows to
three calls. The per-container journal was borrowing the 600s meant for a
host boot journal, so one wedged container could cost 840s; container
journals span a deploy, not a boot. The cheap questions get 30s, and the
one collection site that had no outer bound gets the timeout 1500 the
others already carry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three captures read two sources into one file - devices.txt held
/proc/devices and /proc/misc, net-snmp.txt held snmp and netstat,
sockets-proc.txt held four socket tables - so a grep hit no longer named
a source. Derive the filename from the path instead: the twelve cat
captures become a list of paths, and source_name() turns each into its
own file. Nine names are unchanged; conntrack gains the two segments its
old name dropped, and it was pointing at /proc/net/conntrack while
reading /proc/net/stat/nf_conntrack.
The daemon and kill-marker journals sat under containers/ between the
per-container dumps with an underscore prefix to sort them to the top.
They are host journals; name them as such and put them beside the others.
Give every journalctl call its own budget. The boot journal is now the
largest file in the bundle by two orders of magnitude - 224 MB of 220 MB
total on a one-day host - and timing out writes the exception text into
the file instead of the log.
The nocheck marker on the probe read repairs the lint gate for the
capture test added in 58c06fc8d.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RESCUE_MAX_DEPTH did two jobs without saying so: it bounded real nesting,
and it terminated cycles. Real nesting terminates on its own - every level
is a container - so the cap only ever cost snapshots. Cycles do not
terminate: svc-runner (templates/compose.yml.j2:14) and svc-prx-openresty
(meta/volumes.yml:86) bind-mount the host's docker socket, so those
containers list the host's containers including themselves and would
descend into themselves without end.
Carry the ids already entered instead. Container ids are shared across a
socket and distinct across daemons, so this cuts exactly the cycle and
lets true nesting run to whatever depth it has. A container whose id will
not resolve is skipped rather than entered: without an identity the cycle
cannot be seen, and a missing nested snapshot costs less than a recursion
that never returns.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
getent exits 2 with neither stdout nor stderr when it cannot resolve a
name, so resolve-probe.txt landed in the bundle as zero bytes - the same
shape a capture takes when the collector never reached it. The one file
that answers 'could this host resolve anything' could not distinguish a
failure from its own absence. Record the exit status whenever a command
produced nothing.
write() swallowed OSError, which hides exactly the incident that produces
it: once the filesystem holding the bundle is full, every later file goes
missing without a reason. The job log is the only channel left at that
point, so announce it there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
journalctl -n 10000 cut the window a CI job actually spans. Measured on a
busy host the journal runs at ~56 000 lines per boot-hour, so the cap
covered roughly the last ten minutes of a ninety-minute job and the
incident had scrolled out before the snapshot was taken. Capture the boot
in full, and give err its own file: priority is journald metadata rather
than text, so grepping the warning dump recovers only a third of it.
systemctl list-units calls a forking unit active/running while its guessed
main process is gone, and the containers carry no ps, so the unit table
alone cannot say whether a daemon is still there. service-state.txt adds
MainPID, NRestarts and ExecMainStatus per unit. sockets-proc.txt reads the
kernel tables directly for the containers that ship no iproute2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two investigations of run 30781723102 stalled on files the rescue capture does
not take.
The zfs question - why the debian lab nodes cannot use a module their ubuntu
counterparts apply on ten nodes - needed /proc/modules to see whether the module
is loaded, /proc/misc to see whether it registered (a misc device never appears
in /proc/devices, which is what docker_dataroot.sh was searching), `ls -l
/dev/zfs` to tell a real node from the static one every privileged container
inherits, and `zfs version` for the libzfs version a whole refuted hypothesis
turned on.
The matrix-mdad question - whether ~31.8s per git clone is a connect stall or a
server-side queue - needs the TCP retransmission counters in /proc/net/snmp and
/proc/net/netstat and the conntrack drop counters. Neither can be distinguished
from per-clone timings alone.
Executed on a host with no zfs: a missing binary and a missing proc file each
land as text in the artifact rather than aborting the capture, which is the
best-effort contract run() already implements.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
container.py hardcoded /tmp/infinito-rescue-diagnostics, a second copy of
DIR_RESCUE_DIAGNOSTICS from group_vars/all/05_paths.yml. Move the value to
a generated env key so the paths SPOT stays single: a new handler seeds
INFINITO_RESCUE_LOCAL_DUMPS_DIR from read_group_path(), recurse() forwards
it into nested runtimes, and an unset key now prints why the in-play role
dumps are missing rather than silently collecting nothing.
Registers the handler in ORDERED_HANDLERS - the registry is explicit, not
discovered - and moves rescue_diagnostics_dir.py into the new rescue/
package so the two siblings satisfy the prefix-group lint.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A build that dies on "Temporary failure resolving deb.debian.org" leaves no
trace of why. The rescue snapshot carried journals, disks, containers and swarm
tasks, but nothing about name resolution, so the one question that decided the
case - whether the node's own dnsmasq answered at that second - could not be
settled from any artifact.
collect_host now also takes resolv.conf, daemon.json, the listening UDP and TCP
sockets, the NAT rules, and a resolve probe against the three hosts that killed
builds today. Each entry goes through the same best-effort capture as its
neighbours, so a node without ss or iptables-save records that fact rather than
disturbing the collection.
Because the collector copies itself into every runtime it finds, this lands per
level: runner, node, and the containers inside them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The provision include used a templated path, which the container-exec lint cannot follow, so its argv-form 'container exec' into the OpenProject container read as ungated and would fire on swarm workers (resolver exits 64). Switch to a static include the lint follows and carry IS_STACK_HOST on the include when, so workers skip the whole coupling as a unit. Also picks up the autoformat reflow of the diagnostics daemon-journal capture.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The repair services (rpr-container-soft/hard) emit a syslog 'infinito-kill' marker before they down/restart containers; the rescue collector captures containers/_kill-markers.txt (plus a 6h docker/containerd daemon journal) so a failure artifact names which service stopped containers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The snapshot directory lives under the dump base (both derive from INFINITO_RESCUE_DIAGNOSTICS_DIR), so copytree copied its growing destination into itself until ENAMETOOLONG; the suppressed error left a local-dumps/<stamp>/local-dumps chain that failed the artifact upload of otherwise green jobs (swapfile, wireguard-plain). The copy now ignores the snapshot's own subtree and every ancestor holding it, which also excludes stale sibling snapshots of the same app; a regression test drives real copytree over the production topology.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The gitlab dedicated-postgres failure exhausted all 167 non-superuser slots, but the rescue artifact could not show which clients held them. The collector now dumps pg_stat_activity (per-backend rows, query truncated to 120 chars) and a usename/datname/state connection histogram for every container with postgres in its name, via psql over the container's peer-authenticated socket. Runs only in the failure path and capture() never raises, so a non-postgres name match just records the exec error.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
journalctl -n 1000 covered ~2 minutes on a busy node, so the kernel
and unit events of a failure 30+ minutes earlier were already gone
from the rescue artifact (run 29448092737 nc0: the 22:27 kill window
was unrecoverable). Capture a 10000-line tail plus a 6h warning-level
journal, the full dmesg ring, and drop the 80-line cap on the OOM
filter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
container.py now captures, per container, systemctl status and journalctl, plus host-level systemctl list-units. The SPOT rescue diagnostic reaches the top-level systemd nfs-server container, so a nfs-ganesha startup failure is saved without a role-local dump.
nfs-ganesha now logs to syslog via an OPTIONS override (-L SYSLOG) so late runtime failures reach journald and are captured by the generalized diagnostic. LOGFILE= was a no-op because the packaged unit expands ${OPTIONS}, never $LOGFILE. The bespoke ganesha probe/dump block is removed, superseded by the generalized capture.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace scripts/system/diagnostics/rescue.sh with utils/diagnostics/
container.py: one workflow call on failure captures the current level
(containers with logs+inspect, swarm services, journal, host resources,
in-play role dumps) and recurses itself through every DiD boundary via
self-copy, from the runner down to the innermost runtime (RESCUE_MAX_DEPTH).
compose and swarm now share an identical failure step; the swarm per-node
collector wrapper is gone. All 17 script-only Ansible rescue: blocks are
deleted and their bare block: wrappers inlined (plus a wordpress leftover);
the three justified blocks (postgres pg_hba, xwiki extensions, keycloak
cross-node recovery) keep their evidence tasks and now end in a fail task.
Two lints enforce the state: rescue: is forbidden without a
'# nocheck: rescue-<reason>' first-task marker, and a block without
rescue/always/when or another scoping key must be inlined.
sys-service-terminator hoists become to the include apply. Also move
routine/08_cleanup.sh to utils/clean/teardown.sh and topology/ to
utils/topology/ so routine/ holds only the sequenced flow.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
