diagnostics.sh runs under a 900-second deadline that the sweep deadline
can shorten further. The package manager section came last, after the NFS,
ganesha backtrace and controller probes, so a cut-off run would have lost
exactly the log this collection exists for. It now runs first.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Six Fedora jobs of main runs 39494396920 and 39590304501 (Postmarks#0,
OpenBao#1, Jenkins#1, Magento#0, Matrix#2, OpenResty#0) died in dnf with
`Failed to download packages`. Ansible's dnf5 module reports nothing
more: `failures: []`. Its file logger writes the download errors, with
mirror, URL and status, at debug level to /var/log/dnf5.log, but no
rescue bundle collects that file, so the cause of five of them stays
unknown.
The swarm stack diagnostics now append the last 3000 lines of
/var/log/dnf5.log and, for dnf4 hosts, /var/log/dnf.librepo.log of every
node. A node without them records the `tail` error and the capture goes
on.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
On ubuntu-26.04 runners the swarm deploy fails at svc-storage-nfs-client's
probe-write, right after a successful probe-mount, on every client node:
`Destination directory /run/svc-storage-nfs-client/probe does not exist`.
The rescue bundles cannot say why: ganesha writes to
/var/log/ganesha/ganesha.log, which the swarm diagnostics never collected,
and its NFS_V4 component logged nothing below EVENT.
Debug runs now log NFS_V4 at DEBUG, and the diagnostics append the tail
of ganesha.log to the nfs-boundary section.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Penpot job 105068912535 and MediaWiki job 105068912565 both died on one lab node whose dnsmasq kept its :53 socket bound and rejected every query with 'ignoring query from non-local network', once for a container address and once for 127.0.0.1. Rejecting loopback as non-local only makes sense if --local-service is active and dnsmasq's own interface list went stale, but the bundle records neither the flag nor where it would come from.
Capture the process command line, /etc/default/dnsmasq, the unit including its drop-ins, and the config that actually landed on disk. Addresses and listening sockets were already collected and are not duplicated.
Read-only. Every call falls back to a text line when the file or the unit is absent, so a node without dnsmasq reports that rather than failing the collector.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
03_wait_converge built the DB service name from the engine type:
"${DB_DEP}_${DB_DEP}" -> mariadb_mariadb. That name is only correct when
the database is central. force_shared_db deliberately leaves a
manager-pinned role carrying a literal `shared: false` alone, so
web-app-magento keeps its embedded engine, and sys-svc-rdbms deploys it as
magento_database inside magento's own stack.
The name came from the engine, which is global, and was looked up with
`docker service ls` across the whole swarm instead of the role's stack, so
the `shared` flag never entered the check at any step. Both polarities were
wrong:
variant #1 - no consumer pulls the central provider in, so
mariadb_mariadb does not exist. Run 32630684501 job 97175522113 polled it
ninety times and failed the drill, while the rescue artifact shows
magento_database at 1/1 Running and the play at failed=0.
variant #0 - matomo and mailu declare a Jinja `shared`, get force-shared
and drag svc-db-mariadb in. Job 95662862942 then reported
`db(mariadb): 1/1 Running 2 hours ago` beside `magento: Running 21
minutes ago` - green on a database magento never touches.
resolve_db_service asks the deployed swarm which service carries this
role's database: <entity>_database first, else <dep>_<dep>. The two cannot
collide, because sys-svc-rdbms renders the _database sidecar only under
`lookup('database', application_id, 'local')`. Exact-line matching, not
docker's substring name filter, per 246558a13.
Only web-app-magento changes behaviour: it is the one role that is both
pinned by force_shared_db and seen by the DB_DEP grep. matrix and
bigbluebutton never reach the DB half - their entity key carries no image,
so skip_if_no_swarm_service exits first - and discourse resolves
postgres_postgres exactly as before, in every variant.
A role that declares a DB dep no service serves now says so and gates the
app half alone, instead of polling a name nobody deployed.
diagnostics.sh carried the same guess in its container filter.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dump listed the state path itself, so the volume directories showed up but
nothing inside them. Ownership questions about a specific file were therefore
unanswerable from the artifact alone, which is how a permission failure on a
database file cost two investigations without ever being settled.
Records mode, numeric owner and size one level in. Numeric because the
container uids have no names on the nfs server, and those uids are what the
questions are about.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role runs ganesha at -N NIV_EVENT, and roughly eighteen init steps between the CLIENT commits and the listener log below that level, so the silence in run 30699990912 marked the edge of visibility rather than the fault. Raise INIT and DISPATCH under MODE_DEBUG: nfs_Init_svc(), which binds 2049, logs under COMPONENT_DISPATCH, so INIT alone would have missed exactly the step in question.
Add a userspace backtrace beside the kernel stack dump, which only ever says futex_do_wait. eu-stack names the function holding the lock; elfutils goes into the lab image rather than the role, since a production NFS server should not carry a debugger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill's cryptsetup step died claiming a loop device and the artifact carried nothing to say why: no losetup -a, no /dev listing, and the backup host appears in no other collector loop. Record both views, what the kernel has attached and which nodes the container's /dev carries, so an exhausted pool and a missing device node are told apart at a glance.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The parenthetical claimed the capture names the address and errno of a failed bind. dnsmasq logs that at startup only, and a startup bind failure is fatal, so the half-bound state seen in the artifact cannot originate there. The dynamic bind path is measurably silent on success and untested on failure. The neighbouring markers in this collector carry no description either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The socket table already shows which listening address is missing, but nothing recorded why. A build that cannot resolve dies on the node whose dnsmasq never bound the advertised resolver, and the journal is the only place naming the address and errno. Captured whole rather than tailed, because a failed bind is logged at startup and the neighbouring captures keep only the tail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The collector carried a shell-side default for INFINITO_RESCUE_DIAGNOSTICS_DIR,
which the SPOT lint forbids because a second source drifts unnoticed - and this
one already had: the fallback spelled /tmp/rescue-diagnostics, the value of
INFINITO_RESCUE_DIAGNOSTICS_BASE, while the variable it stood in for resolves to
/tmp/infinito-rescue-diagnostics. Had it ever been used, the collector would have
written where no artifact collects.
A `:?` guard replaces it, matching the six the DR drill already carries. The
only caller sets the variable explicitly for both collectors, and _context.sh
sources the env layer regardless, so the guard should never fire.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Of six CI failures triaged today, three were explainable only from downloaded
artifacts and two not even there: the job log carried "not converged:
docker_docker" or a bare "rc=1", while this collector drowned that same log in
compose trees, env listings and ganesha thread stacks.
diagnostics.sh switches its stdout per section with exec, so every topic lands
in its own file under the rescue directory and the log keeps one line. An
unwritable output directory degrades to stdout rather than killing the
collector - a failed mkdir would otherwise abort the script at the first
redirection, which the caller swallows with `|| true`, leaving no diagnostics
at all and no sign of it. For the same reason the collector says so when it
ends up having captured nothing.
00_one.sh hands the same directory to both collectors. The variable was scoped
to the python one, so the shell collector wrote where no artifact picks it up.
The capture grows by exactly what today's investigations went without: per-node
resolv.conf, daemon.json, listening sockets, NAT rules and a resolve probe
against the three hosts that killed builds, plus df and docker system df.
rescue_index.sh runs last and lists what was collected together with the note
that it is downloadable. The index is the one thing worth printing.
A test pins the routing against a stub docker: one file per section, no bleed
between them, a single log line, and the degrade path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The artifact of the wedged run carried journal, unit status and mounts but no process data, so the hang could be localized to the export config commit and not pinned to a syscall. wchan and syscall per thread close that gap.
MainPID 0 is its own likely state given Restart=on-failure with the start-rate limit disabled, so that branch reports the unit counters instead. Every read keeps 2>&1 because /proc stack and syscall are EPERM on hardened kernels.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ganesha unregisters its RPC programs from inside nfs_Init_svc(), before
the sockets are ever put into listen(). That unregister reaches
libntirpc's rpcb_unset -> CLNT_DESTROY -> svc_vc_destroy_it, which spins
on nanosleep({0,0}) with no timeout -- and it only takes that path when
the AF_LOCAL connect to /var/run/rpcbind.sock succeeds. So a reachable
rpcbind is what wedges the server, not a missing one.
The wedged instance burned 5m30s of CPU while serving nothing, never
logged NFS SERVER INITIALIZED, and left the controller mount retrying
into ECONNREFUSED for 1706s before the deploy failed 20 minutes further
downstream. Masking and stopping both rpcbind units before ganesha starts
makes the spin unreachable: without a local socket rpcb_unset returns
false immediately. Wants=rpcbind.service is not Requires=, so a masked
unit is logged and ignored, and v4 registration is explicitly optional in
ganesha -- which is why this holds only while the export set stays
NFSv4-only.
The unit list is probed first so distros without rpcbind are skipped
rather than failing the play, and the probe distinguishes "no such unit"
from an unreachable manager so the mask can never be skipped silently.
The Type=forking unit reports started while the daemon is still spinning,
so a wait_for on 2049 turns a 28-minute mystery into a named failure, and
the diagnostics now carry rpcbind's journal plus what actually listens.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The discover job was named after the full distro list and the plan summary printed all five, but test-deploy-swarm.yml never set INFINITO_DISTROS, so every run fell back to the default.env distro and only debian was ever exercised. The job name has been advertising coverage that did not exist.
The workflow now passes the distros input into the env and hands the drill to the shared loop from 1bbe7a0eb. Bootstrap, setup, matrix deploy, seed, drain and assert move out of eleven separate steps into routine/00_one.sh, which is what the loop runs once per distro.
Collection and teardown move with them, into an EXIT trap: it replaces the removed 'if: always()' and 'if: failure()' step conditions and returns the original exit code so cleanup cannot mask the deploy failure. The always() teardown step stays as a second net for a step that times out as a whole.
Playwright reports and rescue diagnostics gain a distro segment, matching the layout the compose path already writes, because a second distro would otherwise overwrite the first one's evidence. The structure summary carries the distro in its heading for the same reason.
The README described the old shape and is corrected: the loop entries are listed, topology_summary.sh was missing, and 01_bootstrap.sh is no longer 'one CI step'.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The swarm deploy job now renders the same summary page as compose:
role runtimes, a swarm structure section (topology roles plus live node
list and service placement from the manager) and the Playwright table.
The matrix deploy tees its ansible output and prints the compose
matrix-deploy round/PASS markers, so the runtime parser segments swarm
runs per variant round too.
The runtime table unifies tasks and roles with a Type column (tasks
first), appends the recap total row and flags a mismatch between the
parsed sum and the recap total; rows are read from ROLES RECAP sections
only, which removes the double-counted TASKS RECAP task rows. Each role
row carries one column per host: executed, skipped or failed, with
ignored failures downgraded. The Playwright variant column drops the
redundant variant- prefix. Verified against the logs of runs
29656268771 (swarm) and 29620158441 (compose): every segment's parsed
sum matches the recap total within 0.05s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A wedged manager (act-swarm containerd stall) made the docker exec tar
block up to 900s; on a cancelled job that ate the runner's grace window
and the always() Cleanup step never ran, leaking the DiD cluster. Probe
the manager first (10s) and cap the copy at 120s with --kill-after so a
hung exec is force-reaped; teardown then always follows. Reports stay
collected before teardown, since they live inside the manager node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
27 jobs of run 29147179243 burned their full 360-min budget. Three
systemic layers, all addressed here:
- The CLI e2e task ran the staged test.sh unbounded, and test.sh blocks
in 'systemctl start' on a oneshot backup unit with no JobTimeout; on a
wedged dockerd or a stalled NFS mount both freeze silently for hours.
The task now runs the script under 'timeout TEST_E2E_CLI_TIMEOUT'
(60 min via the timeout lookup, so TIMEOUT_FACTOR scales it) and the
unit starts are bounded by BKP_TEST_HEALTH_TIMEOUT.
- The converge probe re-ran the full per-service diagnostics (service ps,
service logs, inspect, journalctl) on EVERY retry, so 600 nominal
2-second attempts cost 25-33s each and silently outlived the step
timeout. The block is gone (the workflow-level container.py rescue
snapshot already collects it recursively on failure) and every docker
call is timeout-bounded. Same removal in the swarm collect script,
which keeps only the NFS and render evidence container.py does not see.
- Playwright-collect, rescue diagnostics and Cleanup had no
timeout-minutes, and teardown.sh calls docker against D-state nodes,
so a failed job hung there for hours until the job cap killed it
without artifacts. Every step is bounded now, every teardown call is
timeout-wrapped, and the deploy steps (swarm 270, compose 300) leave
room for diagnostics and cleanup below the 360-min cap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
routine/08_cleanup.sh becomes utils/clean/teardown.sh and topology/
becomes utils/topology/ so routine/ holds only the sequenced flow; all
consumers (workflow steps, compose/swarm.yml env guards, relative
sources, README) follow. The shell rescue collector and its lint are
gone: utils/diagnostics/container.py covers compose and swarm
identically. extend_inventory gains the backup-host placements the DR
drill deploys.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the per-node docker-run bring-up with a single node image
(compose/swarm/Dockerfile bakes python3/dnsmasq/openssh/rsync/cryptsetup)
declared in compose/swarm.yml, one compose up, and one ansible play
(compose/swarm/playbook.yml over ansible_connection: docker) that waits for
systemd, captures IPs, wires lab DNS and installs the .deb per node. Reshape
the harness into topology/ (naming SPOT), routine/ (01_bootstrap + the
sequenced 02-08 steps) and utils/ (clean/, collect/, shared helpers); the
workflow, matrix.py and Makefile follow the new paths. swarm init/join was
already ansible-owned (svc-swarm-node), corrected in the README diagram.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
