On swarm nodes /var/lib/infinito is the ganesha NFSv4.2 state share, so
every backup repository landed on it. ganesha 6.5 refuses user.* extended
attributes, and baudolo 7.1.x copies volumes with `rsync -aHAX`, so the
volume backup of OpenCloud (user.oc.* metadata) failed with
`fsetxattr(... "user.oc.id") failed: Operation not supported (95)`.
After the state mount the client role now binds the node's own
/var/lib/infinito-backup over /var/lib/infinito/backup. Every consumer
keeps its path; the data stays on local disk with its attributes, and the
backup host pulls each provider over ssh as before. The bind carries
_netdev: without it, systemd-fstab-generator puts the bind into
local-fs.target below its NFS parent in remote-fs.target, and the boot
transaction cycles. The DR drill remounts the bind after its lazy remount
of the state share, because `umount -l` on the parent detaches it.
Closes#1012
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Narrowing the meshed drill to its backup half bought budget by giving
up the recovery legs over the mesh. The budget overrun is worth measuring
before it is designed around, so the pre-drill pass and the full teardown and
recover come back. Reverts a27b093c4.
DRILL_STAGE=backup returns once the marker has travelled from the live
export through the deployed units to the LUKS device, before step 6 tears the
stack down. The loop device is closed on the way out, which the full path does
inline at step 7. Default stays full, so the direct arm is unchanged.
A round that drills has to deploy twice, so the caller needs the answer
before it deploys anything. Answering here keeps the rule deciding which
applications have something to restore in the script that already owns it,
rather than restating it in the caller.
The verdict is printed rather than signalled by exit status: a probe that
dies on the way to the decision must not read as 'no drill' and leave the
caller skipping the deploy the drill depends on.
Two places in the DR drill ruled on a systemd unit without looking at what the
unit was doing.
[8b/9] took the exit code of `systemctl try-restart nfs-ganesha` as the verdict.
The unit carries Restart=on-failure from the role's own drop-in, and that
drop-in also hangs a 20 second readiness probe on port 2049 as ExecStartPost.
Right after the backing FS is wiped and restored, that probe runs out while the
container still waits on its own stale NFS mount, so systemd kills the daemon it
just started and reports a failed start - then restarts it three seconds later
and succeeds. Measured in run 34699757369, jobs 103572236910 (Matrix#0) and
103638497583 (Gitea#0): 'Control process exited, code=exited, status=124',
'Failed to start', then 'Scheduled restart job, restart counter is at 1'. The
restart attempt may now fail; the assertion is a bounded wait on is-active, and
it aborts loudly with systemctl status when the unit never arrives. The
coherence probe that follows is unchanged and still has to pass.
[0/9] waited for in-flight backups with `systemctl is-active --quiet
'svc-bkp-*.service'` and did not wait. In job 103588089327 (Moodle#0) the volume
unit ran from 01:00:56 to 01:01:12 while that loop returned after 0.36s for four
nodes. The drill then attached to the running job instead of starting its own,
and [3b/9] refused the generation it found because it predated the drill by ten
seconds.
That second change is a hardening, not a proven fix: the reason the loop
returned early is still unknown. The obvious explanation does not hold - the
systemd manual says is-active "Returns an exit code 0 if at least one is
active", so a pattern matching one running unit among four idle ones should have
blocked. list-units --state=active expresses the same intent without relying on
that semantics. Closing the gap needs the call run on a node with exactly that
mix, which no local container offers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The coherence step wrapped a blocking 'systemctl try-restart' in a 120s
budget, one second short of the wedge-recovery cycle the unit performs
when the liburcu-bp startup wedge (f10a4a181) fires - observed in run
32657210518, where the Akaunting drill died on rc=124 while ganesha came
back healthy 4s later. 240s covers even the uncapped 123s cycle; with the
abort grace capped at 10s (64b39e227) one cycle is ~43s.
The '|| systemctl try-restart nfs-server' fallback also masked exactly
this failure: nfs-server.service is 'inactive dead' on a ganesha node,
try-restart on an inactive unit is a documented no-op exiting 0, so a
larger budget alone would have turned a SIGKILLed ganesha into a green
drill. Dispatching on the installed unit file keeps the kernel-NFS branch
and lets a real restart failure surface: a wedge leg now fails loud after
one ~43s cycle instead of passing on self-healing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The converge gate polls STACK_NAME_PRIMARY_SERVICE_KEY. Four roles never
create that name and each was given a static `workload: node-local` to make
the gate skip itself:
21fe9e984 web-svc-cdn ships only cdn_build, never cdn_cdn
2c6c8b763 web-app-matrix names its services matrix_synapse and friends
76508c946 svc-runner deploys node-local compose projects per node
cb52a72f2 bigbluebutton phantom service, and the DR drill must still run
Four hand-written claims for one observable fact. The claim is also static
where the truth is not: web-app-discourse renders `services: {}` only while
its sidecars are shared, and job 94965456921 of run 31864448027 polled
discourse_discourse ninety times for a stack that had none. Declaring the
role node-local would have blinded the gate in the variants where its
sidecars are local and a stack does exist.
has_swarm_service asks the deployed stack for the exact name instead. Every
caller runs behind `if rc == 0`, so an absent service is the role's shape and
not a broken deploy. Exact-line matching, not docker's name filter, which
matches substrings.
No coverage is lost: 03_wait_converge cannot pass without that service, and
05/06/07 plus the backup drill only run once it has. The four roles keep the
skip they were given; discourse and every future case get it without a
declaration. The teardown branch in the drill keeps its meaning - stack rm
where the service exists, compose stop per node where it does not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill starts the backup units itself, but the deploy arms their calendar
timers first, and a job that runs long enough crosses every backup window in
group_vars/all/07_schedule.yml. web-app-decidim ran 23:05-02:24 and hit exactly
that: the drill's own volume backup succeeded at 01:00:49, then the timer fired
a second, unattended run at 01:03:18 while step [8b/9] was lazily unmounting
/var/lib/infinito underneath it.
The failure is silent by construction. baudolo writes the dump in two
processes - 'pg_dump > x.tmp', then 'mv x.tmp x'. The redirect shell exits 0
because a shell never reports a failed writeback on a soft NFS mount, but the
unmount discarded that mount instance together with the unflushed file, so the
mv resolved through the fresh instance and got ENOENT. The unit died at
01:04:04, the same second the drill finished remounting, and 80 minutes later
the CLI test read 'failed' and aborted the job before running its own backup.
Disarming the timers makes the drill the only thing driving those units.
Stopping a timer does not stop its service, and both the drill and
roles/svc-bkp-volume-2-local/files/test/test.sh start their units explicitly,
so nothing under test loses coverage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three steps of the DR drill, one theme: every step now proves what it claims.
[9] restored into the volume's mountpoint while the stack was down - a bare directory,
because docker mounts a volume declaring its own backing store only while a container
holds it. The payload landed on the node disk and the real mount shadowed it on the next
deploy: success reported, nothing delivered, and roughly a gigabyte added to the very
device the drill later runs out of. It now deletes the marker server-side, binds the
backing store at the mountpoint, runs the recover CLI through it, and asserts the marker
on the NFS server's own disk. That also exercises the positive branch of the guard from
773a25ab6 - its refusal path is unit-tested, its pass-through was tested nowhere. When
the manager exposes no backing store the branch skips with a reason: docker volume
inspect is node-local, and with replicas: 1 the volume object usually lives elsewhere.
[3b] replaces a proof that used to happen by accident. The double capture 2efc73d3b
removed also showed, incidentally, that the volume unit really writes - the drill found
its marker in the volume repo too. The new step checks it directly: the newest
backup-docker-to-local generation must be at least as new as DRILL_START_TS (taken on
the manager, the same clock that names generations) and hold a non-empty file under a
volume subtree. Its failure message names the suspicion it exists for: an exclusion that
swallowed every volume. With no volume unit deployed it skips, tracked by
VOLUME_TRIGGERED alongside the existing SECRETS_TRIGGERED.
[5] prints du -sm per machine and repo when the space precondition fails, so the next red
run names what fills the pulled tree instead of leaving it to arithmetic.
Four adversarial review rounds found five defects in this block and confirmed each fix:
an NFS mount reconstructed from driver options the codebase never emits (it is a bind),
the recover CLI's second positional passed a volume name where it takes a host, a missing
volume object aborting the drill instead of skipping, and an umount ordered before the
return-code check, which under set -e would have killed the drill on a busy mount with no
diagnostic and left the bind behind.
None of this has run in a swarm yet. Every failure path is loud and names its cause.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Docker mounts a volume that declares its own backing store (Options set: nfs,
cifs, bind) only while a container holds it. recover.py resolved the volume's
Mountpoint and rsynced into it regardless, so with the consuming stack stopped -
the state its own docstring demands - the payload landed on the node's disk
under the bare mountpoint directory, and the next stack start mounted the real
backing store over it: a restore that reports success, delivers nothing, and
leaves an invisible, unreclaimable orphan behind.
That orphan had weight. In run 30859490335 the swarm drill's [9/9] volume leg
restored ~1 GiB of nextcloud_data onto the manager's 30 GiB loop device in
exactly the minute the device crossed SeaweedFS's 2 GiB minFreeSpace floor -
the last straw on a device already >=93% consumed (the sizing is ea14535f4) -
after which every authenticated Nextcloud request died on an unwritable object
store.
recover.py now probes the mountpoint with `mountpoint -q` (over ssh for
--docker-host) whenever the volume declares Options, and dies naming the
mechanism and the way out instead of writing a single byte. Plain local volumes
without Options keep the old path untouched - their _data directory on the node
disk is the legitimate target.
The drill skips its [9/9] volume leg for such volumes with the reason in the
log: its data restore already happens in [8/9] straight into the NFS export,
which is the restore path the role's README prescribes for NFS-backed volumes
on swarm. The leg never proved anything for them - it only filled the disk.
Three new unit tests pin the contract: unmounted refused, mounted accepted,
remote probed via ssh with the exact command line.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
bdb3b29a9 passed the watchdog floor to the drill as INFINITO_DISK_FLOOR_MB and
broke test-lint:
AssertionError: INFINITO_* keys referenced in code without a registry entry
INFINITO_DISK_FLOOR_MB
The prefix is reserved for the deployment env registry, where every key needs a
default in default.env or a handler that sets it. This value is neither: it is a
handoff from matrix.py to the script matrix.py starts, and it only exists for
the duration of that call. DRILL_EXTRAS, one line above, already goes the same
way under an unprefixed name.
So the key becomes DISK_FLOOR_MB and its declaration moves up beside
DRILL_EXTRAS in the `: "${VAR:?...}"` form the file uses for matrix handoffs,
leaving the INFINITO_* block below to the values that really come from .env.
Registering it in default.env would have satisfied the lint by writing 6144 down
a second time, which is the duplication bdb3b29a9 set out to remove; a nocheck
would have hidden the naming mistake rather than fixed it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill's space check required FREE >= PULLED, but the run is killed from
outside by matrix.py's watchdog as soon as the runner drops below 6G free. The
check knew nothing about that floor, so it waved through a drill that could not
finish: 19008M free against a 13961M pulled tree passes FREE >= PULLED, and the
sync then leaves 5047M, which is under the floor.
That is what happened to the arch swarm drill in run 30830377785. The device
sync reported the marker at 19:16:38.34, the watchdog fired at 19:16:39.73 on
its next 30s tick, and the step died with rc 75 thirteen minutes into work that
was already doomed when the precondition ran.
The requirement is PULLED + FLOOR, not FREE >= PULLED and not 2 * PULLED. FREE
is measured after the pull, so the first copy is already deducted; exactly one
further copy is ever added on top of it - the device copy during the sync, and
later the restore root, by which time base.sh has deleted the pulled tree. The
two extra copies are sequential, never simultaneous.
matrix.py owns the floor now and exports it as INFINITO_DISK_FLOOR_MB. It was
previously a bare `6 * 2**30` the shell had no way to see, which is why the two
sides could disagree about what "enough disk" means.
This makes the failure honest and immediate; it does not create disk. The arch
runner is short by about 1.1G, and closing that needs a capacity decision -
bigger runner, smaller test dataset, or reclaiming build cache before the drill.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard compared free space against USB_SIZE_MB = PULLED_MB*2+256, which is
the nominal size of an image 02_luks_device.sh creates with truncate - sparse,
so nothing ever allocates it. It also ran after that script, i.e. once the image
was already created, formatted and mounted, so the image's own footprint was
already subtracted from the df it read. It charged the pulled tree a second time
while that tree was counted as used, not free. In run 30781723102 it demanded
28108M against 16582M and refused a drill that fits: the peak is 14555M
additional, derived from rescue-diagnostics-swarm-web-app-nextcloud-0, and the
same run's debian job contains a full 9/9 completion of the same drill.
Rather than predicting the image's ext4 overhead - two earlier attempts at that
constant were wrong in both directions - both thresholds are now read:
df on DEV_MOUNT reports what the encrypted filesystem actually offers after
mkfs, and df on DIR_BACKUPS reports what the host still has for the copy the
sync writes into the sparse file and, later, the restore root. Both are compared
against PULLED_MB, a measured quantity. USB_SIZE_MB keeps sizing the image.
Every abort now tears the device down (umount, luksClose, rm -f); a refused
drill previously left the mapping and the image behind.
Exercised against the recorded numbers and six edge cases: 26624/16582/13926
passes where the old guard refused, an undersized device or host is refused, and
an empty or non-numeric df reading is reported as unreadable.
Both thresholds sit exactly at PULLED_MB, so second-order allocation inside the
image (extent trees, directory blocks) is uncovered - tens of MiB against a
13.9 GiB tree. The drill has not been run with this guard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three conflicts, each resolved by combining both sides rather than picking one.
compose.yml passes both env vars through to the container: swarm-nfs adds
INFINITO_SWARM_MGR_IP, this branch adds INFINITO_TOR_SOCKS_PORT, and neither
substitutes for the other.
daemon.json.j2 takes the data-root block from swarm-nfs and keeps HEAD's dns
key. The incoming side writes networks.internet.dns directly; lookup(
'container_dns') already returns that value as its clearnet entry and prepends
the docker0 bridge on a Tor node in compose mode, so the incoming form is a
strict subset and dropping it loses nothing.
The WordPress CA-trust addon spec gains skipUnlessServiceEnabled("sso") from
swarm-nfs alongside HEAD's gotoOnion and resolveTimeout, and asserts on the
Plugin Name the mu-plugin header declares - the mustuse screen renders that,
not the filename - with resolveTimeout applied to the literal the incoming
side left bare.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
losetup -f prints the device with a trailing (lost) marker when its node is missing, and both claim sites fed the whole string to mknod. The recovery path that materialises /dev/loop8 and beyond therefore never worked, which is why the pool appeared to stop at the eight preallocated nodes even though max_loop is 0. Strip the marker before the node is built.
Nothing released the data-root loop a node claims, so each distro round leaked four of them: across 140 job logs leg 1 always applies 5 and declines 0, leg 2 applies 3 and declines 2, leg 3 declines 4. The declining nodes fall back to overlayfs, which disables the btrfs snapshot mode the script exists to exercise, and the DR drill then finds no device at all.
Detach right after the mount instead. The mount holds the opener, so the kernel arms autoclear rather than detaching, and the loop plus its sparse image are reclaimed when the node container takes its mount namespace down. Verified in a privileged container: the mount survives the detach, AUTOCLEAR reads 1, the filesystem stays writable, btrfs subvolume create still works, and the loop is released on umount.
Drop the teardown block added earlier: it filtered on (lost), which marks a missing device node rather than a missing backing file, so it selected exactly the devices losetup -d cannot detach, ran unprivileged, and printed success unconditionally. Restore the (deleted) filter in the drill. Require mountpoint up front so a missing tool declines instead of killing the node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every node runs docker_dataroot.sh, which attaches a loop for its data root, and nothing ever detached them. Run 30699990912 shows the consequence: all eight kernel loop devices bound to docker-fs.img in two inode groups, i.e. two rounds' worth, leaving none for the DR drill's LUKS image.
Release the orphaned ones after the nodes are gone -- only then is a loop unambiguously ours to reclaim, rather than a concurrent run's. Match (lost) as well as (deleted): a backing file inside a removed container's overlay becomes unfindable rather than merely unlinked, which is why the existing cleanup in 02_luks_device.sh never fired.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill died with cryptsetup's 'loop device with autoclear flag is required', which names neither the resource nor the reason. Claim a loop for the image and release it again first, creating the /dev node when the container's tmpfs carries none, so cryptsetup finds what it needs and a genuine shortage stops the drill with a message that says so.
The cryptsetup calls themselves are unchanged and keep owning their own autoclear device, so nothing lingers once the mapper closes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve conflicts, resolved by intent rather than by side.
Six shared one shape: the incoming side adds the new rescue_index.sh call but
drops this branch's `# nocheck: shell-or-true` markers. The markers carry a
lint contract here, so the incoming structure was taken and the markers put
back — dedicated.sh, guide/one.sh, guide/host_deploy.sh, swarm/routine/00_one.sh
and trigger_units.sh, where the incoming version additionally writes the full
journal to a file instead of only echoing a tail.
web-app-matrix keeps HEAD's _synapse_extra_args accumulator (the superset: CA
mount plus the onion case) but adopts the incoming improvement inside it — the
inlined regex_replace gives way to lookup('subnet_gateway').
handlers/__init__.py drops rescue_diagnostics_dir from the old import group
(the incoming side moved it under .infinito.rescue, keeping both would import
it twice) and keeps tor_socks_port, a HEAD-only handler ORDERED_HANDLERS still
needs.
svc-bkp-volume-2-local takes the incoming side throughout: both branches landed
the same feature, but swarm-nfs evolved it — the knob moved from the global
STORAGE_BACKUP_SNAPSHOT_MODE to services.volume-2-local.snapshot_mode, plus the
subvolume helper, 04_snapshot_subject.yml, meta/packages.yml and baudolo 3.2.2.
Three files that left behind were removed: files/baudolo_snapshot.py and the two
worktree tests at their pre-refactor paths, all byte-identical to their moved
counterparts.
Six follow-on failures the merge produced, fixed at the root:
- STORAGE_BACKUP_SNAPSHOT_MODE lost its last reader with the config-lookup
design and is gone from group_vars.
- utils/github/variant_bundle_limits.py sat next to variant_bundles.py, which
the prefix lint forbids; the pair now lives in utils/github/variant/ as
bundles.py and bundle_limits.py, with the -m entry point, the ruff
per-file-ignore, the design-doc link and the mirroring test path updated.
- The incoming WordPress addon spec navigated with a raw page.goto and carried
three bare timeout literals; both route through gotoOnion and resolveTimeout
now, which is what makes navigation survive onion transport.
- Six unmarked `|| true` got a concrete reason each: docker.socket is absent on
distros that ship docker without it, the loop device is already detached when
the probe failed before attaching, and modprobe zfs may fail because the
/dev/zfs check decides.
make quality green across all four targets. The cold-domains budget test that
had to be waived here passes on its own now — 19d56f07e raises it to 100s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The nextcloud drill watched /dev/root go from 40G used to 124G used with 21G
free on a 145G disk shared by five nested nodes, the NFS export, both backup
repos and a 14.2GB pull. Four staging trees survived the whole run - grep for
`rm -rf` found nothing after the secrets leg - so the drill kept every copy it
made until the containers went away.
DR_RESTORE_STAGE, DR_VOL_STAGE, DR_SEC_STAGE and RESTORE_ROOT are released
after their last reader, each checked individually against the line that reads
it. All four sit on the success path: `set -e` aborts before them on any
failure, so a drill that fails still hands its evidence to the rescue
collector.
The device step now measures free space before it starts and refuses with both
numbers rather than running into ENOSPC inside a nested systemd unit. The
threshold is USB_SIZE_MB, which the script already computes, so no new constant
enters - and it states the real ledger: the drill holds the pulled tree twice,
once for the device sync and again while the restore root exists alongside the
image. It sits after 02_luks_device.sh because that is where the only stale-loop
reaper runs; a guard placed earlier would abort forever on a leaked loop device
and the reap that would clear it would never happen.
Reading free space is split from evaluating it. Under pipefail a df that
rejects --output fails the command substitution and set -e kills the drill
before the check can speak - which is how the guard managed to produce exactly
the silence it exists to prevent. Driving the body against six df behaviours
(roomy, tight, no header, option unsupported, empty, non-numeric) is what
surfaced that.
This does not address why the observed run failed. It died in step [4/9] with
21G still free, so this guard would not have fired on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill's remote pull exits 1 after 65 minutes and nothing says why. The
unit loops over two providers and only counts errors, so which one failed is
unrecoverable; the pull runs `rsync -abP`, whose --progress writes one journal
line per file - 77137 of them on a nextcloud pull - and the drill captured
`journalctl -n 60`. The entire diagnostic was therefore rsync filenames.
The unit now names the host it could not pull from. rsync keeps --partial and
trades --progress for --info=stats2, turning the flood into a closing balance.
trigger_units.sh writes the complete journal of a failed unit into the rescue
dump directory the artifact ships as local-dumps, and prints only the lines
from warning upwards.
That directory arrives as an argument instead of a second copy of the paths
SPOT: the drill reads INFINITO_RESCUE_DIAGNOSTICS_DIR once, which the env
handler derives from group_vars/all/05_paths.yml, and hands it to every call
site. A test pins that contract, and it earned its place immediately - it found
two call sites that a grep for the file name cannot see, and both were the
units this change exists for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mapper name is host-global and nothing closes it once a drill succeeds:
the only teardowns sit inside the recover step and in the post-luksOpen
error path. A preserved cluster therefore hands the next round a live
'usbdrill' and cryptsetup refuses with 'Device usbdrill already exists'.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The LUKS step of the DR drill died on `The file /dev/mapper/usbdrill does
not exist and no size was specified.` -- from mkfs, not from cryptsetup,
because luksOpen had exited 0. The node did have a udev, and it had
created the mapper entry: as a symlink to ../dm-1. Modern udev no longer
mknods, it only manages symlinks, and the /dev/dm-N target it points at is
a devtmpfs node that a container's tmpfs /dev never receives. Without a
udev libdevmapper creates the node itself, which is why the debian nodes
passed -- their image ships no udevd at all, while arch cannot install
systemd without one.
Masking udevd in the node image removes the whole class rather than one
call site. DM_DISABLE_UDEV rides on the node containers instead of the
script because the drill opens the volume a second time through the
recover CLI in a separate docker exec, which would not inherit an export.
The block-device check then fails loudly with both prerequisites named,
and closes the mapping first so an abort cannot leak it onto the runner --
device-mapper is not namespaced and every distro round shares the VM.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The node container has no udev, so LOOP_CTL_GET_FREE hands out a minor whose
/dev node never appears; it also cannot load kernel modules, so those nodes
stay dead unless the host carries the loop driver. Materialize the nodes,
verify one can actually be attached, and expose the host prerequisite as a
bootstrap step instead of failing three hours in with a cryptsetup message.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill configured svc-bkp-remote-2-local with a fixed manager plus NFS provider list for every app, while the manager only runs a backup repository when the app's include closure pulls svc-bkp-volume-2-local or svc-bkp-secrets-2-local in. For an app whose container_backup consumer is statically off, such as svc-dns-unbound, user-backup never ran on the manager, so the pull was rejected with 'Permission denied (publickey,password)' and the fail-if-any wrapper took the whole unit down at drill step 4/9. Four roles reach that step today: svc-dns-unbound, svc-db-redis, svc-registry-cache and svc-registry-docker.
backup_repos.py expresses the placement once and matrix derives the provider list from it per round. derive_includes accepts the round's variant map because INFINITO_APP_VARIANTS only reaches the subprocess callers. The extras file keeps the device paths the drill reads back and loses the provider list nothing read from it, and the drill banners stop claiming backup legs that were never installed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Back to the pre-swarm-nfs-fix baseline:
- 5c2c8f531 + 985ec7eab: the swarm writer-quiesce DR drill. Deep-dive
attribution of CI run 29715516546 found it the dominant failure source
(17 of 23 jobs): [9b/9] handed-off generation mismatch, [6/9] quiesce
stack-rm network race, [4/9] remote-2-local pull stall, compose discourse
restore duplicate key.
- d6f980574: openproject swarm ingress to web:8080. Carried a leaking
proxy_internal_port lookup (no per-app default) that broke the seaweedfs
dependency front-proxy ('Key web not found'); the bypass-the-Apache-proxy
approach was never verified against downloads/X-Sendfile.
- 1dd153bad: xwiki replicas:1 + mem 3g. Did not resolve the xwiki superadmin
failure (still red in run 29715516546), so it is unverified.
Re-approach each from a clean baseline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up corrections to b449f8157 (swarm DR drill):
- export test.env in 06/07 via set -a so the child db restore/probe
scripts inherit BKP_TEST_DATABASES_CSV and BKP_TEST_RESTORE_BIN
instead of aborting at their :? guards before any restore runs
- move db_probe.sh/db_restore.sh into a db/ subfolder (probe.sh,
restore.sh) and update all callers, so they stop tripping the
prefix-grouped-files structure lint
- define TEST_E2E_SWARM_DRILL in group_vars/all and drop the now
redundant default() from test.env.j2 (satisfies the undefined-var
and spot-default lints together)
- fix a mermaid parse error in the drill README: a ; in a
sequenceDiagram note was read as a statement separator
make quality-high green (test + lint phases, 15 targets, 0 failures).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Defer Swarm SQL replay to the disposable DR drill so shared writers such as Discourse are stopped before the destructive restore. Verify the selected generation with before/after database probes while preserving the Compose restore path.
dedicated.sh streams rescue diagnostics via docker exec tar since docker cp cannot see the systemd container's mount namespace and silently extracted nothing. recover_drill.sh guards on docker availability: non-stack role deploys now drill the nfs and secrets legs instead of hard-failing on the impossible volume leg; stack jobs still drill all three. 04_stack_rm_wait tolerates an already-removed stack and re-issues the rm once services are gone.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The drill entry gate treated 'deploys no swarm service' as 'has nothing
to restore', but the backup chain operates on NFS volumes and deployed
backup units, not on scheduler objects; bigbluebutton's seven
NFS-backed volumes went entirely unproven. Gate the drill solely on the
NFS-flagged volume and make only the full-disaster step
workload-aware: stack roles keep docker stack rm, node-local roles get
their compose project (label com.docker.compose.project=<entity>, the
compose handler's --project convention) stopped on every node so no
writer races the export wipe and restore. Declare
workload: node-local on bigbluebutton so the phantom-service
converge/chaos gates skip while play, e2e and now the drill keep
running; svc-runner stays skipped via its missing NFS volume. Document
the nine drill steps host-by-host in the README.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The marker discovery matched any .dr-drill-marker under files/, so
SRC_REL could resolve to a parent tree and step 8 restored the wrong
subtree into the live export as root, leaving n8n_data owned root:root
and n8n (uid 1000) crash-looping on EACCES. Scope both repo layouts to
the primary volume's own marker and declare uid/gid 1000 on the n8n
data volume so nfs_prep enforces ownership on every deploy instead of
relying on Docker's copy-on-first-use.
The drill also stacked four copies of the backup tree on shared space
(live export, pulled tree, fully-allocated LUKS image, final restore
into a tmpfs /tmp) and ENOSPCed on large apps. Restore to /var/tmp,
drop the pulled tree before the device recover, make the loop image
sparse and free it after the marker verify, and stage only the volume
generation instead of the whole machine tree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Step 8 rewrites the export subtree directly on the ganesha server's
backing filesystem while the server keeps running and the nodes keep
their client mounts, so ganesha's mdcache and the client dentry caches
still describe the pre-wipe tree and update-pass tasks are Rejected
with readdirent ENOENT for the whole converge window. Step 8b restarts
ganesha, lazily remounts every swarm node and gates on a real write
probe through the manager's mount, each command timeout-bounded so a
wedged DinD node fails loudly instead of hanging the job.
The drill's tar extracts remapped ownership by name on the receiving
host like the rsync legs did; extract with --numeric-owner. Drill paths
now come from the INFINITO_DIR_* SPOT env vars.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A prior wedged drill run can leave loop devices bound (backing file deleted, or bound to the image path); a full loop pool makes cryptsetup luksFormat abort with "loop device with autoclear flag is required". Detach the orphaned and stale loops first so cryptsetup can autoclear-attach a fresh one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The drill (backup/base.sh) no longer runs its own step-10 redeploy; it recovers device -> nfs export -> volume -> secrets and writes a sentinel. The matrix update pass (which runs next anyway) boots the recovered stack, and verify_recovered_marker.sh asserts the live marker after it. Saves one full stack redeploy per drilled app; the marker-gone check before recovery stays in 05_wipe_export.sh.
Also derive recover_drill.sh REPO from the script location instead of an INFINITO_REPO_ROOT shell-side default (lint: no INFINITO_* / absolute-path defaults in .sh).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add cli.administration.recover: one uniform type/source/target command wrapping each svc-bkp-* role files/recover.py (nfs, volume, secrets, device) plus full for whole-tree recovery. Colon-encoded sources are classified by shape; device supports an on-device subpath, a generation, and remote pull over ssh.
Replace the per-host swarm DR-drill scripts (03/06/07/08_recover_*.sh) with the recover CLI in backup/base.sh. Pass the device on-device subpath relative (the CLI reads absolute segments as the restore-root) and load the applications-free .deploy.yml extras twin on the step-10 redeploy so host_vars credentials survive the -e override.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The DR drill's simulated USB was a fixed 512M loop image (~449M usable
after LUKS2 + ext4), but the deployed local-2-device unit syncs the
whole pulled backup root, which after the remote pull holds two
machine-id trees (the manager's docker+secrets repos and the NFS
export repo). For an app the size of akaunting that overflows the
image mid-rsync and the drill fails with ENOSPC at step [5/9]. base.sh
now measures the pulled tree (du -sm) and passes 2x its size + headroom
(floor 2G) to 02_luks_device.sh, which sizes the loop image from that
argument; fallocate keeps the image sparse so the larger floor is free.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
routine/08_cleanup.sh becomes utils/clean/teardown.sh and topology/
becomes utils/topology/ so routine/ holds only the sequenced flow; all
consumers (workflow steps, compose/swarm.yml env guards, relative
sources, README) follow. The shell rescue collector and its lint are
gone: utils/diagnostics/container.py covers compose and swarm
identically. extend_inventory gains the backup-host placements the DR
drill deploys.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
extend_inventory maps svc-bkp-remote-2-local + svc-bkp-local-2-device onto
the backup host and write_extras configures them (backup_providers =
manager + NFS server IPs; device mount/target/source), so the matrix
deploy installs the real systemd units there. The drill now exercises
them: 01_ssh_trust.sh only installs the pull identity, the remote-2-local
unit pulls every provider (replacing both manual pull calls), and after
02_luks_device.sh simulates the USB plug the local-2-device unit performs
the device sync; the device recover passes the configured target subpath
via --device-target. Unit absence on the backup host is now a hard
failure since the roles deploy with the round.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add routine/backup/: base.sh drives a marker seeded into the live NFS
volume and the manager secrets through the whole chain (volume+secrets+nfs
backup units -> remote-2-local pull -> local-2-device LUKS 'USB') and back
via every recover.py (device -> local root -> nfs export, docker volume and
host secrets), asserting each marker survives. Each per-host step is one
in-node routine run over docker exec. extend_inventory maps the volume/
secrets/nfs backup roles onto their hosts so the units install; write_extras
generates the pull keypair; the INFINITO_SWARM_BACKUP_* keys are added to
default.env + the passthrough handler.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
