mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 10:59:59 +00:00
fix(swarm): say what converged, not only what did not
stack_ready.sh reports on both of its unhappy exits -- the latched branch dumps
the offending services, the not-converged branch dumps task rows and state -- and
falls off the end of the file in silence when it succeeds.
That silence cost a whole investigation. Moodle#1 of run 33414875521 died with
rc 137 and empty stdout/stderr 1.6s after this gate returned ok, and the entire
44422-line job contains no docker service ps, no task-state dump and no
State.OOMKilled for moodle. Which task generation was live at the moment of the
exec had to be reconstructed from the file NAMES in the rescue tree, and the
proof only arrived from a separately downloaded bundle:
moodle_moodle.3.mwdxfc7z6m3q877dr3ow1f5qr finished = 20:05:17.418099896Z
install task end = 20:05:17.427801
The task ids and Status.Timestamp this prints are exactly what was missing.
Cost is one run per convergence, not per poll: ansible's until stops at the first
rc 0, so the success path executes once per deploy. Both docker calls inside
report_task_states are already timeout 15 and swallow their own failure, so a
slow manager delays nothing and breaks nothing.
The array guard mirrors the existing one at :90 -- under set -u, "${arr[@]}" on
an empty array aborts in bash before 4.4.
Deliberately NOT included: a gate for UpdateStatus.State == updating. The gate is
demonstrably blind to in-place updates -- it returned ok in 0.630s after a stack
deploy that reported changed, and across all nine deploys of PASS 2 it passed on
the first poll, while in PASS 1 (service creation) it retried 56 times over
2:25.116. But UpdateStatus is written by swarmkit's updater goroutine after the
raft commit, which is the same lag that makes the replica count stale, and the
old container still reported healthy 0.6s AFTER the gate returned. A previous
update also leaves the status at completed, which an early poll reads as fine.
That fix would narrow a window of unmeasured width across 40+ swarm roles rather
than close it; it needs the timing measured on a throwaway cluster first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
52ab6a3428
commit
455a733785
1 file changed
+7
@@ -135,3 +135,10 @@ if [ -n "$not_running" ]; then
|
||||
fi
|
||||
exit 1
|
||||
fi
|
||||
|
||||
printf '%s\n' "$services" | sed 's/^/ converged: /' >&2
|
||||
if [ ${#service_names[@]} -gt 0 ]; then
|
||||
for svc in "${service_names[@]}"; do
|
||||
report_task_states "$svc"
|
||||
done
|
||||
fi
|
||||
Reference in new issue
Block a user