kevinveenbirkenbachandClaude Opus 5 3748e3ee00 fix(swarm-drill): judge the units by their state, not by one call's exit code ...

Two places in the DR drill ruled on a systemd unit without looking at what the
unit was doing.

[8b/9] took the exit code of `systemctl try-restart nfs-ganesha` as the verdict.
The unit carries Restart=on-failure from the role's own drop-in, and that
drop-in also hangs a 20 second readiness probe on port 2049 as ExecStartPost.
Right after the backing FS is wiped and restored, that probe runs out while the
container still waits on its own stale NFS mount, so systemd kills the daemon it
just started and reports a failed start - then restarts it three seconds later
and succeeds. Measured in run 34699757369, jobs 103572236910 (Matrix#0) and
103638497583 (Gitea#0): 'Control process exited, code=exited, status=124',
'Failed to start', then 'Scheduled restart job, restart counter is at 1'. The
restart attempt may now fail; the assertion is a bounded wait on is-active, and
it aborts loudly with systemctl status when the unit never arrives. The
coherence probe that follows is unchanged and still has to pass.

[0/9] waited for in-flight backups with `systemctl is-active --quiet
'svc-bkp-*.service'` and did not wait. In job 103588089327 (Moodle#0) the volume
unit ran from 01:00:56 to 01:01:12 while that loop returned after 0.36s for four
nodes. The drill then attached to the running job instead of starting its own,
and [3b/9] refused the generation it found because it predated the drill by ten
seconds.

That second change is a hardening, not a proven fix: the reason the loop
returned early is still unknown. The obvious explanation does not hold - the
systemd manual says is-active "Returns an exit code 0 if at least one is
active", so a pattern matching one running unit among four idle ones should have
blocked. list-units --state=active expresses the same intent without relying on
that semantics. Closing the gap needs the call run on a node with exactly that
mix, which no local container offers.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

2026-09-13 15:02:46 +02:00

..

act

…

ci

fix(diagnostics): let the in-zone probes read the drop-in that holds the zone

2026-09-03 12:50:58 +02:00

local

…

swarm

fix(swarm-drill): judge the units by their state, not by one call's exit code

2026-09-13 15:02:46 +02:00

utils

…

distros.sh

fix(swarm-test): put the step guard on the same side of sudo as its payload

2026-09-01 12:36:10 +02:00

roundtrip.sh

…