mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 19:01:54 +00:00
The DR drill starts the backup units itself, but the deploy arms their calendar timers first, and a job that runs long enough crosses every backup window in group_vars/all/07_schedule.yml. web-app-decidim ran 23:05-02:24 and hit exactly that: the drill's own volume backup succeeded at 01:00:49, then the timer fired a second, unattended run at 01:03:18 while step [8b/9] was lazily unmounting /var/lib/infinito underneath it. The failure is silent by construction. baudolo writes the dump in two processes - 'pg_dump > x.tmp', then 'mv x.tmp x'. The redirect shell exits 0 because a shell never reports a failed writeback on a soft NFS mount, but the unmount discarded that mount instance together with the unflushed file, so the mv resolved through the fresh instance and got ENOENT. The unit died at 01:04:04, the same second the drill finished remounting, and 80 minutes later the CLI test read 'failed' and aborted the job before running its own backup. Disarming the timers makes the drill the only thing driving those units. Stopping a timer does not stop its service, and both the drill and roles/svc-bkp-volume-2-local/files/test/test.sh start their units explicitly, so nothing under test loses coverage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>