01_bootstrap.sh pins the five swarm nodes into the shared cache network
at 200 + INFINITO_INSTANCE * 5. That band fits a /24 only for instances
0 to 10; from instance 11 on the script aborted with "pushes the node
band past ...254", so no worktree past slot 10 could start a swarm
cluster while the package cache runs. Five nodes per slot cannot fit
255 slots in one /24 with any fixed formula.
Instances whose band fits keep their pinned addresses, so the primary
checkout and CI behave as before. Above that, the nodes join the cache
network without --ip and Docker assigns them from its pool (the upper
half of the subnet), the same way every worktree stack container joins
it. Nothing in the repository reads the nodes' cache-network addresses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On swarm nodes /var/lib/infinito is the ganesha NFSv4.2 state share, so
every backup repository landed on it. ganesha 6.5 refuses user.* extended
attributes, and baudolo 7.1.x copies volumes with `rsync -aHAX`, so the
volume backup of OpenCloud (user.oc.* metadata) failed with
`fsetxattr(... "user.oc.id") failed: Operation not supported (95)`.
After the state mount the client role now binds the node's own
/var/lib/infinito-backup over /var/lib/infinito/backup. Every consumer
keeps its path; the data stays on local disk with its attributes, and the
backup host pulls each provider over ssh as before. The bind carries
_netdev: without it, systemd-fstab-generator puts the bind into
local-fs.target below its NFS parent in remote-fs.target, and the boot
transaction cycles. The DR drill remounts the bind after its lazy remount
of the state share, because `umount -l` on the parent detaches it.
Closes#1012
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
ubuntu-latest is rolling over to ubuntu-26.04, whose sudo is sudo-rs. It
ignores `-E` ("preserving the entire environment is not supported") and
hands the child only the explicit `env` assignments, so the swarm matrix
died at import with `KeyError: 'INFINITO_SRC_DIR'` before any inventory
existed (PR #1009, Nextcloud#0). Jobs that landed on ubuntu-24.04 kept
passing.
Every `sudo -E` now passes the caller's exported names through
`--preserve-env="$(compgen -e | paste -sd, -)"`, which both classic sudo
and sudo-rs honour. A lint rejects `sudo -E` and bare `--preserve-env`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The workspace jobs pulled every image anonymously from Docker Hub and
two of them failed on its rate limit in run 39059128678 (#990): the
fedora swarm drill on busybox:1.38 on all three nodes, the centos
compose drill while building tor-smtp_custom. The regular deploy jobs
pull through the GHCR mirror, but the mirror was tied to being a real
GitHub runner: Profile.image_mirror_enabled() returned
runs_on_github(), and the swarm routine generated mirrors.yml only when
cli.meta.runtime reported github, which is act inside make
swarm-zombie.
The mirror is now its own capability, INFINITO_GHCR_MIRROR (next to
INFINITO_GHCR_MIRROR_PREFIX), written as a run fact like
INFINITO_CACHE_STACK: true on a real GitHub runner,
false everywhere else, and a non-empty exported value wins for the run.
Profile.image_mirror_enabled() reads the declaration and keeps
runs_on_github() as the fallback, and 02_provision_inventory.sh gates
on the same value instead of the runtime name. make swarm-zombie
forwards it into act through ACT_ENV, since act ignores the host env.
call-test-workspace.yml declares INFINITO_GHCR_MIRROR=true and passes
INFINITO_GHCR_MIRROR_PREFIX into the workspace container. Nothing else
of the workspace changes: is_ci, runs_on_github, the cache stack and
the docker root stay local-like. The step that loads the rescue paths
now removes the .env it generated on the runner, because on debian
rows load.sh sourced it inside the container and the run inherited
INFINITO_RUNNING_ON_GITHUB=true.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The three-arm comparison on web-app-gitea#0 put the meshed arm at 414ms
per task over the direct arm, and the tunnel at 4% of that. What it could not
split is the rest: the arms differ both in connection plugin (docker against
ssh) and in escalation (root with no prompt against administrator with a
password sudo on every task).
A fixed set of module calls, timed either side of the transport switch with
become off and on, gives four numbers that separate the two. It costs about a
minute where a fourth deploy arm would have cost three hours, and it needs no
root login and no sudoers change.
INFINITO_SWARM_TRANSPORT_BENCH guards it, so a normal round times nothing.
The arm existed to answer one question: how much of the meshed arm's
cost is the connection plugin and how much is the tunnel. Runs 37592061960
(docker), 37753506661 (SSH on the underlay) and 37621556682 (SSH over
WireGuard) answered it on web-app-gitea#0, debian/ext4:
pass 1 5964s / 8511s / 8832s
pass 2 5319s / 7472s / 7705s
Leaving ansible_connection: docker costs +4700s, the tunnel +554s - 89.5%
against 10.5%. The answer is recorded; the scaffolding does not need to ship.
Reverts d792519725, 1b28f87b37, 89b8a8c8d1, 0470491e1a, 529c5ace43,
dd3d10a9a2, c20dd0e5f5, 458d8956e1, 63ade1e03b, d281555945 and 4adfb6e3d2.
95cfc8b7b7 is already superseded by bbfd82f89, which sets the profiling limit
upstream.
Narrowing the meshed drill to its backup half bought budget by giving
up the recovery legs over the mesh. The budget overrun is worth measuring
before it is designed around, so the pre-drill pass and the full teardown and
recover come back. Reverts a27b093c4.
DRILL_STAGE=backup returns once the marker has travelled from the live
export through the deployed units to the LUKS device, before step 6 tears the
stack down. The loop device is closed on the way out, which the full path does
inline at step 7. Default stays full, so the direct arm is unchanged.
A round that drills has to deploy twice, so the caller needs the answer
before it deploys anything. Answering here keeps the rule deciding which
applications have something to restore in the script that already owns it,
rather than restating it in the caller.
The verdict is printed rather than signalled by exit status: a probe that
dies on the way to the decision must not read as 'no drill' and leave the
caller skipping the deploy the drill depends on.
SWARM_REQUIRED_SERVICES no longer force-keeps wireguard: with the axis off the
role is absent from the inventory, so there is no disabled-service conflict to
work around.
The drill strips the services it cannot run without from the incoming disable
list, and wireguard was not among them. For an app whose own graph does not
reach it, the provision step disabled the service while extend_inventory had
already placed svc-net-wireguard on every swarm host, and the services
disabler rejected the pair:
service 'wireguard' is disabled, but provider role 'svc-net-wireguard' is
still active in .../devices.yml
svc-swarm-node declares the service required for exactly this reason, so the
drill has no more business disabling it than it has disabling node.
Two places in the DR drill ruled on a systemd unit without looking at what the
unit was doing.
[8b/9] took the exit code of `systemctl try-restart nfs-ganesha` as the verdict.
The unit carries Restart=on-failure from the role's own drop-in, and that
drop-in also hangs a 20 second readiness probe on port 2049 as ExecStartPost.
Right after the backing FS is wiped and restored, that probe runs out while the
container still waits on its own stale NFS mount, so systemd kills the daemon it
just started and reports a failed start - then restarts it three seconds later
and succeeds. Measured in run 34699757369, jobs 103572236910 (Matrix#0) and
103638497583 (Gitea#0): 'Control process exited, code=exited, status=124',
'Failed to start', then 'Scheduled restart job, restart counter is at 1'. The
restart attempt may now fail; the assertion is a bounded wait on is-active, and
it aborts loudly with systemctl status when the unit never arrives. The
coherence probe that follows is unchanged and still has to pass.
[0/9] waited for in-flight backups with `systemctl is-active --quiet
'svc-bkp-*.service'` and did not wait. In job 103588089327 (Moodle#0) the volume
unit ran from 01:00:56 to 01:01:12 while that loop returned after 0.36s for four
nodes. The drill then attached to the running job instead of starting its own,
and [3b/9] refused the generation it found because it predated the drill by ten
seconds.
That second change is a hardening, not a proven fix: the reason the loop
returned early is still unknown. The obvious explanation does not hold - the
systemd manual says is-active "Returns an exit code 0 if at least one is
active", so a pattern matching one running unit among four idle ones should have
blocked. list-units --state=active expresses the same intent without relying on
that semantics. Closing the gap needs the call run on a node with exactly that
mix, which no local container offers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
From triage of run 33923190220. Each fix is independently complete.
penpot: ansible templates delegate_to before the task's merged when, and AnsibleUndefinedVariable escapes get_delegated_vars_and_hostname. PENPOT_BACKEND_HOST_NODE is set only by the resolve include, so the bootstrap task hard-failed on every host that include skipped instead of skipping with it. A block-level when cannot prevent this; move the tasks into a dynamic include.
matrix: the MDAD reconfigure looked for volume matrix_matrix_mdad_state (real name matrix_mdad_state) and container matrix-mdad (real name matrix). Both if-bodies were skipped while the task still reported changed, so a rotated S3 secret never reached synapse and every media write got SignatureDoesNotMatch. Move the shell into its own script and pass the real names through the environment. Also add no_log to the vars.yml render, whose diff printed 16 secrets in cleartext.
n8n: when the settings file and N8N_ENCRYPTION_KEY disagree, oclif's findCommand throws for every command, so 'n8n export:credentials' is unreachable in exactly the state that would need it. The re-key is one-way; pin it rotatable: false.
swarm-drill: distros.sh sees only rc=124 and cannot tell a wedged deploy from one the sweep deadline cut short. Say so where the governor caps it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
0b1565c493 went in with --no-verify and without the autoformat pass, so the
trailing nocheck comment in run_within_deadline() sat sixteen columns right
of the comment column the next line uses. The quality gate's autoformat step
realigns it on every pass; this lands that realignment so the tree stops
dirtying itself after each gate.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The EXIT trap handed out four fixed budgets, 1500 + 900 + 900 + 900, against
the 900s reserve the governor sets aside for it. Its first step alone was
allowed to run longer than the whole reserve, so the trap regularly outlived
the deadline and the outer `timeout -k 60` escalated to SIGKILL. Nextcloud#3
and Nextcloud#11 in run 33574056480 died that way, both at 20460s to the
second, with the diagnostics still running in their own process group and the
teardown never reached.
Each step now takes the smaller of its nominal budget and the time actually
left, recomputed before it runs, and leaves behind what every later step needs.
The worst case where each one exhausts its cap fits the reserve exactly: 480 +
120 + 30 + 120 + 30 + 120. Two steps that carried no timeout at all join the
accounting, because one unbounded step defeats the whole arrangement.
The cap only engages near the deadline. A deploy that fails on its own with
hours of budget left sees the nominal budgets unchanged, so the depth is
traded away only in the case where the exit code already names the cause.
What that case gains is a truthful ending. The job now reports the deploy's own
rc rather than 137, the cluster is torn down instead of leaked into the next
distro, and the reports are collected. A step cut short by its cap says so on
stdout and returns 124 to the caller, so a half-written rescue bundle can no
longer read as a complete one.
The drill is not made faster by any of this. A round too slow for
INFINITO_CI_DISTRO_BUDGET_SECONDS still fails, now as a clean 124.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b8a6205396 put INFINITO_SWARM_TEARDOWN_RESERVE_SECONDS into default.env and
made 00_one.sh require it with a :? guard, but wired it into no handler.
default.env is only the source of the value; a key reaches the generated .env
solely through a handler's setdefault, so make dotenv dropped it and the
swarm routine aborted on its own guard.
The two nocheck markers cover the neighbouring key on those lines:
INFINITO_CI_DISTRO_DEADLINE_EPOCH is computed by distros.sh from the sweep
budget at run time, so a static default would be a stale timestamp. They have
to sit on the same line; above the statement the marker does not apply.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Moodle#0 of run 33445831564 was cancelled by the 355-minute job cap with almost no
artifacts. The cap did not end a deploy; it reaped a zombie.
00_one.sh wrapped `timeout` around `sudo`, so the guard ran as the unprivileged
runner and the payload as root. An unprivileged SIGKILL cannot reach a root process,
and the same holds for distros.sh's own `timeout -k` one level up. Probed both
shapes: the payload lands in a pgid the outer governor never addresses either way, so
--foreground would not have helped -- the privilege wall is what carries.
The log shows the consequence directly. The governor fired on schedule:
06:18:27.884 distros.sh: line 103: 15914 Killed timeout -k 60 "${remaining}s" "$@"
06:18:27.894 [ERROR] Run failed for distro=ubuntu: rc=137 after 20460s
and then ansible output kept arriving in the same tee'd log -- a TASK at 06:18:27.929,
another at 06:26:00.189, elapsed still advancing -- until
`##[error]The operation was canceled.` The orphan held the step's pipe open, so tee
never saw EOF and the step never returned.
954b9f1ae6 added the TERM trap precisely so a timeout kill would still collect
diagnostics, and its reasoning ("GNU timeout signals the whole process group") is
correct. It simply never gets signalled on the swarm path, because sudo stands
between. That is why this cell carries only inventory- and role-runtimes- while its
compose twin also has rescue-diagnostics- and playwright-. This completes that fix
rather than replacing it.
The ladder was also inverted: INFINITO_CI_DISTRO_BUDGET_SECONDS=20400 (340 min)
against INFINITO_SWARM_STEP_TIMEOUT_MINUTES=350, with default.env:97 asserting the
order as if correct. For the trap to run, the step guard must fire BEFORE the sweep
governor. Rather than re-tune two constants that start their clocks at different
points -- matrix start trails drill start by 1m49s to 2m04s, measured -- 00_one.sh now
derives its ceiling from the governor's deadline minus a teardown reserve, and keeps
its own configured value when that is smaller.
Not fixed here, and it must be said: Moodle#0 does not fit on a hosted runner at all.
Its pass-2 pacing tracks pass-1 within 5%, putting the pass end at roughly 362 minutes
after job start, past both the 355 cap and GitHub's own 360 ceiling. Raising the cap
would be masking and is arithmetically impossible anyway. What this buys is an honest
red cell at ~323 minutes WITH its diagnostics, instead of a silent cancellation with
none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both Workspace jobs of run 33445831564 die at
roles/sys-svc-container/handlers/main.yml:9, on the first registry pull a nested
node's dockerd performs after a restart:
docker: Error response from daemon: Get "https://registry-1.docker.io/v2/":
tls: failed to verify certificate: x509: certificate signed by unknown authority
The image name is a red herring. rpardini/docker-registry-proxy SSL-bumps, and the
certificate it presents for registry-1.docker.io and for ghcr.io carries the same
issuer, CN=DockerMirrorBox Intermediate IA -- measured against the live proxy through
the make helper, with three controls: direct fetch on public roots returns 401, the
same fetch through the proxy on public roots only returns "self-signed certificate in
certificate chain", and --cacert /dev/null errors, proving curl honours the flag.
That last control mattered: curl consults its compiled-in CApath as well, where
update-ca-certificates has hash-linked the proxy CA, so without --capath /nonexistent
the negative control passes silently.
So a mirror-name fix would have MOVED the error from docker.io to ghcr.io rather than
removing it. What is missing is trust, and the trust anchor came from
INFINITO_CACHE_REGISTRY_CA_HOST_PATH -- a variable act does not forward, and whose
`make dotenv` rewrites the shared .env back to default.env. It can therefore name a
directory the proxy that actually answers never signed with.
The bootstrap now reads the anchor off that proxy's own /ca bind. And it stops
limping: with the frontend up and no /ca bind exposed, the run aborts naming the
reason instead of continuing untrusting and surfacing three minutes later as a
cryptic x509.
Locally the patch is a measured no-op -- the inspected value is byte-identical to the
.env value, and the render with and without it is identical -- so it changes behaviour
only where the variable is wrong, which is act.
Not proven: that /mnt/cache/registry/ca/ca.crt on the hosted runner is the root the
live proxy presents. It follows from rpardini's design and from the sibling leg that
works, but both host directories are absent from every artifact and that runner is
gone. The line "==> registry-cache CA source: /mnt/cache/registry/ca" in the next
workspace run settles it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The coherence step wrapped a blocking 'systemctl try-restart' in a 120s
budget, one second short of the wedge-recovery cycle the unit performs
when the liburcu-bp startup wedge (f10a4a181) fires - observed in run
32657210518, where the Akaunting drill died on rc=124 while ganesha came
back healthy 4s later. 240s covers even the uncapped 123s cycle; with the
abort grace capped at 10s (64b39e227) one cycle is ~43s.
The '|| systemctl try-restart nfs-server' fallback also masked exactly
this failure: nfs-server.service is 'inactive dead' on a ganesha node,
try-restart on an inactive unit is a documented no-op exiting 0, so a
larger budget alone would have turned a SIGKILLed ganesha into a green
drill. Dispatching on the installed unit file keeps the kernel-NFS branch
and lets a real restart failure surface: a wedge leg now fails loud after
one ~43s cycle instead of passing on self-healing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
03_wait_converge built the DB service name from the engine type:
"${DB_DEP}_${DB_DEP}" -> mariadb_mariadb. That name is only correct when
the database is central. force_shared_db deliberately leaves a
manager-pinned role carrying a literal `shared: false` alone, so
web-app-magento keeps its embedded engine, and sys-svc-rdbms deploys it as
magento_database inside magento's own stack.
The name came from the engine, which is global, and was looked up with
`docker service ls` across the whole swarm instead of the role's stack, so
the `shared` flag never entered the check at any step. Both polarities were
wrong:
variant #1 - no consumer pulls the central provider in, so
mariadb_mariadb does not exist. Run 32630684501 job 97175522113 polled it
ninety times and failed the drill, while the rescue artifact shows
magento_database at 1/1 Running and the play at failed=0.
variant #0 - matomo and mailu declare a Jinja `shared`, get force-shared
and drag svc-db-mariadb in. Job 95662862942 then reported
`db(mariadb): 1/1 Running 2 hours ago` beside `magento: Running 21
minutes ago` - green on a database magento never touches.
resolve_db_service asks the deployed swarm which service carries this
role's database: <entity>_database first, else <dep>_<dep>. The two cannot
collide, because sys-svc-rdbms renders the _database sidecar only under
`lookup('database', application_id, 'local')`. Exact-line matching, not
docker's substring name filter, per 246558a13.
Only web-app-magento changes behaviour: it is the one role that is both
pinned by force_shared_db and seen by the DB_DEP grep. matrix and
bigbluebutton never reach the DB half - their entity key carries no image,
so skip_if_no_swarm_service exits first - and discourse resolves
postgres_postgres exactly as before, in every variant.
A role that declares a DB dep no service serves now says so and gates the
app half alone, instead of polling a name nobody deployed.
diagnostics.sh carried the same guess in its container filter.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
01_bootstrap.sh ran apt-get update with no timeout, so a stalled mirror
connection hung until the drill's global budget killed it - twice in run
32284541284 (jobs 96178507396 funkwhale, 96178507443 friendica), each
after hours of silence. The journal pins it exactly: sudo[462047] opened
19:51:36 and closed 00:07:30, 4h15m54s in a single apt-get update, and
the session only ended because timeout fired. Healthy runs of the same
line in the same jobs took 3s, 21s and 41s.
The cost is not one red job. distros.sh:130 is the only ceiling and it
wraps the whole per-distro sequence in the remaining global budget, so
one stalled fetch also takes the untried distros with it: funkwhale ran
2 of 5, friendica 3 of 5, both ending at budget remaining=-19s/-20s.
-qq hid the stall and || true cannot catch a hang.
Same failure class and same remedy as ba8a72309, which bounded
scripts/install/apt.sh; this closes the sibling call site it left open.
Constants are taken verbatim from there rather than re-derived. timeout
runs under sudo so SIGTERM reaches apt-get instead of the sudo wrapper -
the 4h session above is what the other order looks like. -E stays so the
package-cache proxy variables survive.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The converge gate polls STACK_NAME_PRIMARY_SERVICE_KEY. Four roles never
create that name and each was given a static `workload: node-local` to make
the gate skip itself:
21fe9e984 web-svc-cdn ships only cdn_build, never cdn_cdn
2c6c8b763 web-app-matrix names its services matrix_synapse and friends
76508c946 svc-runner deploys node-local compose projects per node
cb52a72f2 bigbluebutton phantom service, and the DR drill must still run
Four hand-written claims for one observable fact. The claim is also static
where the truth is not: web-app-discourse renders `services: {}` only while
its sidecars are shared, and job 94965456921 of run 31864448027 polled
discourse_discourse ninety times for a stack that had none. Declaring the
role node-local would have blinded the gate in the variants where its
sidecars are local and a stack does exist.
has_swarm_service asks the deployed stack for the exact name instead. Every
caller runs behind `if rc == 0`, so an absent service is the role's shape and
not a broken deploy. Exact-line matching, not docker's name filter, which
matches substrings.
No coverage is lost: 03_wait_converge cannot pass without that service, and
05/06/07 plus the backup drill only run once it has. The four roles keep the
skip they were given; discourse and every future case get it without a
declaration. The teardown branch in the drill keeps its meaning - stack rm
where the service exists, compose stop per node where it does not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill starts the backup units itself, but the deploy arms their calendar
timers first, and a job that runs long enough crosses every backup window in
group_vars/all/07_schedule.yml. web-app-decidim ran 23:05-02:24 and hit exactly
that: the drill's own volume backup succeeded at 01:00:49, then the timer fired
a second, unattended run at 01:03:18 while step [8b/9] was lazily unmounting
/var/lib/infinito underneath it.
The failure is silent by construction. baudolo writes the dump in two
processes - 'pg_dump > x.tmp', then 'mv x.tmp x'. The redirect shell exits 0
because a shell never reports a failed writeback on a soft NFS mount, but the
unmount discarded that mount instance together with the unflushed file, so the
mv resolved through the fresh instance and got ENOENT. The unit died at
01:04:04, the same second the drill finished remounting, and 80 minutes later
the CLI test read 'failed' and aborted the job before running its own backup.
Disarming the timers makes the drill the only thing driving those units.
Stopping a timer does not stop its service, and both the drill and
roles/svc-bkp-volume-2-local/files/test/test.sh start their units explicitly,
so nothing under test loses coverage.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
docker's name filter matches substrings, so every stack whose sidecar is
named after its app answered with two containers and two services. The
drill took the first, which after a reschedule was the MCP sidecar: it
declares no healthcheck and does not listen on the app port, so the step
waited sixty seconds on the wrong process. Selection now goes through the
swarm service label and an exact name match, both behind one helper.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill probed for a fixed 60s. A container without a shell can only be
observed through its docker health status, and mattermost declares
start_period 30s with interval 1m, so that status cannot report healthy
before 90s. The step was unreachable for such an app however fast it
started. The budget now comes from the container's own healthcheck and
stays at 60s for every container that declares none.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The quiesce guard spelled the dev container as the literal
infinito_nexus_debian, so it only ever matched a checkout whose .env was
generated for debian without a project prefix. It missed the self-hosted
runner form infinito_<project>_nexus_<distro>, any custom.env pin, and any
.env generated for another distro — and it missed them silently, because a
non-matching grep just skips the block and leaves the competing onion
publisher running.
INFINITO_CONTAINER is the SPOT for that name: compose.yml uses it as
container_name, and scripts/meta/env/load.sh (already sourced at line 19)
exports it. Proven against the container running on the author's host: the
literal does not match it, the variable does.
The outer `docker ps | grep -qx` pre-check is gone with it — `docker exec`
on an absent container already exits non-zero with stderr suppressed, and
the `if` absorbs that even under `set -euo pipefail`.
The variable joins the line-21 assertion so a broken env fails by name
instead of dying inside a pipeline subshell, which would reproduce exactly
the silent-skip class this commit removes. compose.yml already requires the
same variable with `:?`, so this is not a new demand on the environment.
tests/lint/repository/test_no_hardcoded_container_name.py pins it: run
against the pre-fix file it flags all three lines, and it stays green on
the distro-agnostic spellings in stack_refresh.sh and worktree/up.sh.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Docker hands out the lowest free address, so a node attached without one lands
in the same band where the cache services pin themselves and can take an
address its owner then cannot claim. That happened: a node held the
registry-cache address and every swarm cycle died at the image pull.
Each node now takes a fixed address, offset per instance because worktrees
share the primary's cache network while clearing only their own cluster, so a
surviving cluster elsewhere no longer collides with a fresh one.
A failed attach is classified rather than swallowed. The nodes already carry
extra_hosts pointing at the cache addresses, so a node that silently fails to
attach reaches neither apt nor the registry and reports it as network
flakiness. Only the markers the docker_network_noop filter already treats as
benign are tolerated; anything else stops the bootstrap.
The subnet is asserted rather than assumed, since the address arithmetic only
holds for a x.y.z.0/24.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The routine only carried an EXIT trap, so a run terminated by the enclosing
`timeout` in distros.sh:128 left no rescue diagnostics, no playwright reports
and no teardown — the exact case where the artifacts matter most.
Add a TERM trap and make the handler take the status as an argument, so EXIT
keeps using `$?` while TERM passes 143 explicitly. The `_teardown_done` latch
is required because the EXIT trap still fires after the TERM trap; without it
the whole collect-and-teardown block, including its 1500s and 900s timeouts,
would run twice.
This works because GNU timeout signals the whole process group: bash defers a
trap while it waits on a foreground child, so a bare kill to the script PID
alone would not surface it until that child exited. Reproduced under the real
shape (timeout -k): TEARDOWN rc=143, exactly once, after 2.01s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The swarm DR stage aborted with "no backup unit installed on <mgr> or
<nfs-server>" on all five workspace distros. routine/backup/base.sh triggers
svc-bkp-volume-2-local / svc-bkp-secrets-2-local on the manager and
svc-bkp-nfs-2-local on the export host; all three trigger_units calls returned
rc=2 (no unit matches), so _triggered stayed 0.
The units were never installed because the roles were never in the closure.
utils/tests/swarm/README.md pins the contract: derive_includes pulls
svc-bkp-volume-2-local in via the `container_backup` consumer and
svc-bkp-nfs-2-local via `nfs_backup`. derive_includes._disabled_provider_roles
maps those same keys back to those roles and `found -= disabled_roles` prunes
them, so any caller that lists them in `disable` silently removes what the
drill then requires.
scripts/tests/environment/12_swarm.sh and 13_roundtrip.sh both do exactly that:
`variable_services <app> node nfs-server` keeps three keys and disables every
other group-conditional one, which sweeps up container_backup and nfs_backup as
collateral. Reproduced offline byte-identically to the CI line:
$ python3 -m cli.meta.roles.applications.dynamic_services \
--exclude mariadb node nfs-server
asset,...,container_backup,...,nfs_backup,...
Filtering in those two callers would leave `make swarm-zombie app=X disable=...`
and any future caller exposed, so the filter goes to the single entry point every
swarm drill passes through: the CI matrix reaches routine/00_one.sh via
distros.sh, and swarm-zombie reaches the same script through act. Both `disable`
consumers -- the provision-time services_disabler and derive_includes in the
matrix subprocess -- read the env var this script re-exports.
Measured before/after on the derive_includes closure:
APP_ID=svc-db-mariadb -> {} / {svc-bkp-secrets-2-local, svc-bkp-volume-2-local}
APP_ID=svc-storage-nfs-server-> {} / {svc-bkp-nfs-2-local}
node and nfs-server join the set for the same reason db87a1a1c added them to the
callers: they build the lab cluster itself.
Token matching is exact, so nfs_backup2 and mynode survive; `IFS=', ' read -a`
avoids the pathname expansion an unquoted split would have allowed. The CI swarm
workflow passes no `disable` at all, so the filter is a no-op there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role README claimed the backup machinery "operates transparently against the mount
paths" and demoted the source-of-truth rule to advice in a prose box - both written
before 2efc73d3b made that rule the code's behaviour. It now states what is enforced:
this role skips export-backed volumes, svc-bkp-nfs-2-local is their single capture, and
manager-pinned roles, roles forcing another compose_mode_force and nfs: false opt-outs
stay covered here. It also names the limit the code actually has - the exclusion checks
that some host is in the svc-bkp-nfs-2-local inventory group, not that the role ever ran
successfully - because a README promising more than the code delivers is worse than one
promising less.
Its Schema section carried the role's timer flow; it now carries the mechanism the
change set is about: meta/volumes.yml -> swarm_nfs_backed() as the single predicate ->
its two consumers (the compose rewrite and the backup exclusion) -> the inventory-group
gate -> the export host's single capture -> the restore path with its mountpoint probe
-> the drill's [3b] guard against over-exclusion.
The two swarm READMEs drew the drill with eight steps and pre-773a25ab6 semantics. Both
now draw the nine that exist, including what makes the proof a proof: the pulled tree is
wiped before the device recover, so the device holds the only copy left; the live export
is wiped with the marker proven gone before it is restored; and [9] deletes the marker
server-side before restoring the volume through its bound backing store.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three steps of the DR drill, one theme: every step now proves what it claims.
[9] restored into the volume's mountpoint while the stack was down - a bare directory,
because docker mounts a volume declaring its own backing store only while a container
holds it. The payload landed on the node disk and the real mount shadowed it on the next
deploy: success reported, nothing delivered, and roughly a gigabyte added to the very
device the drill later runs out of. It now deletes the marker server-side, binds the
backing store at the mountpoint, runs the recover CLI through it, and asserts the marker
on the NFS server's own disk. That also exercises the positive branch of the guard from
773a25ab6 - its refusal path is unit-tested, its pass-through was tested nowhere. When
the manager exposes no backing store the branch skips with a reason: docker volume
inspect is node-local, and with replicas: 1 the volume object usually lives elsewhere.
[3b] replaces a proof that used to happen by accident. The double capture 2efc73d3b
removed also showed, incidentally, that the volume unit really writes - the drill found
its marker in the volume repo too. The new step checks it directly: the newest
backup-docker-to-local generation must be at least as new as DRILL_START_TS (taken on
the manager, the same clock that names generations) and hold a non-empty file under a
volume subtree. Its failure message names the suspicion it exists for: an exclusion that
swallowed every volume. With no volume unit deployed it skips, tracked by
VOLUME_TRIGGERED alongside the existing SECRETS_TRIGGERED.
[5] prints du -sm per machine and repo when the space precondition fails, so the next red
run names what fills the pulled tree instead of leaving it to arithmetic.
Four adversarial review rounds found five defects in this block and confirmed each fix:
an NFS mount reconstructed from driver options the codebase never emits (it is a bind),
the recover CLI's second positional passed a volume name where it takes a host, a missing
volume object aborting the drill instead of skipping, and an umount ordered before the
return-code check, which under set -e would have killed the drill on a busy mount with no
diagnostic and left the bind behind.
None of this has run in a swarm yet. Every failure path is loud and names its cause.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pick was made once per matrix entry, in a workflow step ahead of the drill,
and every distro of that entry then ran on it. Coverage followed: one entry
exercised one filesystem, and the swarm never reached zfs at all, because
resolve.sh's userland_for() kept it out of any entry containing arch, centos or
fedora - which is every automatic sweep. The draw now happens inside the loop in
distros.sh, once per distro, so a five-distro entry covers zfs with 87% and all
three kinds with 62% probability instead of one kind with certainty.
The workflow input changes meaning with it. It was a single kind, offered as a
choice in the manual entry; it is now a space-separated allow-list. Empty draws
from all three, and naming kinds narrows the draw to them and marks the pick
required, so a host that cannot deliver one of them fails rather than quietly
running on something else. A name that is not a filesystem this project puts a
data root on is rejected outright instead of reaching the applying step.
For the draw to mean anything the images have to carry the userlands, so
scripts/install/zfs.sh becomes scripts/install/filesystem.sh and installs ext4
and btrfs alongside zfs. That is not symmetric work: btrfs-progs does not exist
in the EL family at all, because RHEL dropped btrfs, and a plain
`dnf install btrfs-progs` there fails the whole transaction and takes e2fsprogs
with it. EPEL carries it, so the EL family goes through EPEL from the start -
split by distro rather than by a failed first attempt, since Fedora has no
epel-release to fall back to and routing a transient failure there would turn it
into a fatal one. verify() now demands all four binaries, so a packaging
regression fails the build instead of leaving a hole the drill discovers later.
Measured on all five bases: debian trixie, ubuntu 26.04, arch, fedora 44 and
centos 10 each end with mkfs.ext4, mkfs.btrfs, zfs and zpool present and ldd
fully resolved.
Two defects only became reachable once a single job could ask for different
filesystems in turn:
* docker_dataroot.sh answered "already prepared" and returned for any mount it
found, so from the second distro on, every draw would have been silently
ignored on a host that survives the sweep - the runner in compose and host
mode. It now compares what is mounted against what was asked for and
dismantles the old root when they differ. Everything docker cached there
goes with it; that is what covering more than one filesystem per entry
costs.
* reclaim_stale_pool ran only when the new pick was zfs, so a zfs to btrfs
transition left the previous distro's pool imported on the shared kernel,
pinning a loop device over an image its dead container had already unlinked.
It now runs on every pick, which needed the /dev/zfs handling lifted out of
the zfs branch into zfs_device_ready().
The restore_docker trap is armed before anything can stop docker, rather than
after, so a failure while dismantling a data root cannot leave a host without a
daemon. resolve.sh splits its allow-list into an array before testing it, so a
whitespace-only input is the empty list it looks like rather than a narrowed one
that yields an empty pick marked required. pick.sh clears both keys when it
declines to draw: they are declared in default.env, so a generated .env carries
whatever was exported when it was made, and a caller gating its apply on the
value would otherwise repoint the developer's own docker in the very case the
skip exists to prevent.
Verified by execution rather than by reading: a privileged container ran
docker_dataroot.sh with real loop devices and mkfs through ext4 -> ext4 -> btrfs
-> ext4, confirming the recognition, the replacement in both directions and that
no loop leaks across it; a second harness with a recording zpool stub confirmed
that a btrfs pick does reclaim a stale pool, that no pool means no destroy, and
that a host without /dev/zfs still completes an ext4 pick without touching zpool.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Docker mounts a volume that declares its own backing store (Options set: nfs,
cifs, bind) only while a container holds it. recover.py resolved the volume's
Mountpoint and rsynced into it regardless, so with the consuming stack stopped -
the state its own docstring demands - the payload landed on the node's disk
under the bare mountpoint directory, and the next stack start mounted the real
backing store over it: a restore that reports success, delivers nothing, and
leaves an invisible, unreclaimable orphan behind.
That orphan had weight. In run 30859490335 the swarm drill's [9/9] volume leg
restored ~1 GiB of nextcloud_data onto the manager's 30 GiB loop device in
exactly the minute the device crossed SeaweedFS's 2 GiB minFreeSpace floor -
the last straw on a device already >=93% consumed (the sizing is ea14535f4) -
after which every authenticated Nextcloud request died on an unwritable object
store.
recover.py now probes the mountpoint with `mountpoint -q` (over ssh for
--docker-host) whenever the volume declares Options, and dies naming the
mechanism and the way out instead of writing a single byte. Plain local volumes
without Options keep the old path untouched - their _data directory on the node
disk is the legitimate target.
The drill skips its [9/9] volume leg for such volumes with the reason in the
log: its data restore already happens in [8/9] straight into the NFS export,
which is the restore path the role's README prescribes for NFS-backed volumes
on swarm. The leg never proved anything for them - it only filled the disk.
Three new unit tests pin the contract: unmounted refused, mounted accepted,
remote probed via ssh with the exact command line.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
bdb3b29a9 passed the watchdog floor to the drill as INFINITO_DISK_FLOOR_MB and
broke test-lint:
AssertionError: INFINITO_* keys referenced in code without a registry entry
INFINITO_DISK_FLOOR_MB
The prefix is reserved for the deployment env registry, where every key needs a
default in default.env or a handler that sets it. This value is neither: it is a
handoff from matrix.py to the script matrix.py starts, and it only exists for
the duration of that call. DRILL_EXTRAS, one line above, already goes the same
way under an unprefixed name.
So the key becomes DISK_FLOOR_MB and its declaration moves up beside
DRILL_EXTRAS in the `: "${VAR:?...}"` form the file uses for matrix handoffs,
leaving the INFINITO_* block below to the values that really come from .env.
Registering it in default.env would have satisfied the lint by writing 6144 down
a second time, which is the duplication bdb3b29a9 set out to remove; a nocheck
would have hidden the naming mistake rather than fixed it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill's space check required FREE >= PULLED, but the run is killed from
outside by matrix.py's watchdog as soon as the runner drops below 6G free. The
check knew nothing about that floor, so it waved through a drill that could not
finish: 19008M free against a 13961M pulled tree passes FREE >= PULLED, and the
sync then leaves 5047M, which is under the floor.
That is what happened to the arch swarm drill in run 30830377785. The device
sync reported the marker at 19:16:38.34, the watchdog fired at 19:16:39.73 on
its next 30s tick, and the step died with rc 75 thirteen minutes into work that
was already doomed when the precondition ran.
The requirement is PULLED + FLOOR, not FREE >= PULLED and not 2 * PULLED. FREE
is measured after the pull, so the first copy is already deducted; exactly one
further copy is ever added on top of it - the device copy during the sync, and
later the restore root, by which time base.sh has deleted the pulled tree. The
two extra copies are sequential, never simultaneous.
matrix.py owns the floor now and exports it as INFINITO_DISK_FLOOR_MB. It was
previously a bare `6 * 2**30` the shell had no way to see, which is why the two
sides could disagree about what "enough disk" means.
This makes the failure honest and immediate; it does not create disk. The arch
runner is short by about 1.1G, and closing that needs a capacity decision -
bigger runner, smaller test dataset, or reclaiming build cache before the drill.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The guard compared free space against USB_SIZE_MB = PULLED_MB*2+256, which is
the nominal size of an image 02_luks_device.sh creates with truncate - sparse,
so nothing ever allocates it. It also ran after that script, i.e. once the image
was already created, formatted and mounted, so the image's own footprint was
already subtracted from the df it read. It charged the pulled tree a second time
while that tree was counted as used, not free. In run 30781723102 it demanded
28108M against 16582M and refused a drill that fits: the peak is 14555M
additional, derived from rescue-diagnostics-swarm-web-app-nextcloud-0, and the
same run's debian job contains a full 9/9 completion of the same drill.
Rather than predicting the image's ext4 overhead - two earlier attempts at that
constant were wrong in both directions - both thresholds are now read:
df on DEV_MOUNT reports what the encrypted filesystem actually offers after
mkfs, and df on DIR_BACKUPS reports what the host still has for the copy the
sync writes into the sparse file and, later, the restore root. Both are compared
against PULLED_MB, a measured quantity. USB_SIZE_MB keeps sizing the image.
Every abort now tears the device down (umount, luksClose, rm -f); a refused
drill previously left the mapping and the image behind.
Exercised against the recorded numbers and six edge cases: 26624/16582/13926
passes where the old guard refused, an undersized device or host is refused, and
an empty or non-numeric df reading is reported as unreadable.
Both thresholds sit exactly at PULLED_MB, so second-order allocation inside the
image (extent trees, directory blocks) is uncovered - tens of MiB against a
13.9 GiB tree. The drill has not been run with this guard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three conflicts, each resolved by combining both sides rather than picking one.
compose.yml passes both env vars through to the container: swarm-nfs adds
INFINITO_SWARM_MGR_IP, this branch adds INFINITO_TOR_SOCKS_PORT, and neither
substitutes for the other.
daemon.json.j2 takes the data-root block from swarm-nfs and keeps HEAD's dns
key. The incoming side writes networks.internet.dns directly; lookup(
'container_dns') already returns that value as its clearnet entry and prepends
the docker0 bridge on a Tor node in compose mode, so the incoming form is a
strict subset and dropping it loses nothing.
The WordPress CA-trust addon spec gains skipUnlessServiceEnabled("sso") from
swarm-nfs alongside HEAD's gotoOnion and resolveTimeout, and asserts on the
Plugin Name the mu-plugin header declares - the mustuse screen renders that,
not the filename - with resolveTimeout applied to the literal the incoming
side left bare.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The routine flowchart marked a terminator-driven remote-2-local start on bkp-01, failing for lack of the ssh trust the drill installs later. Both halves are wrong: the lab deploys the app stack and the backup host with --skip-backup, so MODE_BACKUP is false, svc-bkp-remote-2-local never sets system_service_force_flush_final, and the unit never enters system_service_run_list.
The drill is the only caller that starts the pull. The unit journal of the failed run holds exactly one invocation, at 20:58:30 - one line after 01_ssh_trust.sh installed the key, and that first rsync moved 5.8 GB. A terminator-started pull would have run at ~20:55:4x, before the key existed.
The box also loses its trigger colouring, since nothing fires there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
losetup -f prints the device with a trailing (lost) marker when its node is missing, and both claim sites fed the whole string to mknod. The recovery path that materialises /dev/loop8 and beyond therefore never worked, which is why the pool appeared to stop at the eight preallocated nodes even though max_loop is 0. Strip the marker before the node is built.
Nothing released the data-root loop a node claims, so each distro round leaked four of them: across 140 job logs leg 1 always applies 5 and declines 0, leg 2 applies 3 and declines 2, leg 3 declines 4. The declining nodes fall back to overlayfs, which disables the btrfs snapshot mode the script exists to exercise, and the DR drill then finds no device at all.
Detach right after the mount instead. The mount holds the opener, so the kernel arms autoclear rather than detaching, and the loop plus its sparse image are reclaimed when the node container takes its mount namespace down. Verified in a privileged container: the mount survives the detach, AUTOCLEAR reads 1, the filesystem stays writable, btrfs subvolume create still works, and the loop is released on umount.
Drop the teardown block added earlier: it filtered on (lost), which marks a missing device node rather than a missing backing file, so it selected exactly the devices losetup -d cannot detach, ran unprivileged, and printed success unconditionally. Restore the (deleted) filter in the drill. Require mountpoint up front so a missing tool declines instead of killing the node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every node runs docker_dataroot.sh, which attaches a loop for its data root, and nothing ever detached them. Run 30699990912 shows the consequence: all eight kernel loop devices bound to docker-fs.img in two inode groups, i.e. two rounds' worth, leaving none for the DR drill's LUKS image.
Release the orphaned ones after the nodes are gone -- only then is a loop unambiguously ours to reclaim, rather than a concurrent run's. Match (lost) as well as (deleted): a backing file inside a removed container's overlay becomes unfindable rather than merely unlinked, which is why the existing cleanup in 02_luks_device.sh never fired.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill died with cryptsetup's 'loop device with autoclear flag is required', which names neither the resource nor the reason. Claim a loop for the image and release it again first, creating the /dev node when the container's tmpfs carries none, so cryptsetup finds what it needs and a genuine shortage stops the drill with a message that says so.
The cryptsetup calls themselves are unchanged and keep owning their own autoclear device, so nothing lingers once the mapper closes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve conflicts, resolved by intent rather than by side.
Six shared one shape: the incoming side adds the new rescue_index.sh call but
drops this branch's `# nocheck: shell-or-true` markers. The markers carry a
lint contract here, so the incoming structure was taken and the markers put
back — dedicated.sh, guide/one.sh, guide/host_deploy.sh, swarm/routine/00_one.sh
and trigger_units.sh, where the incoming version additionally writes the full
journal to a file instead of only echoing a tail.
web-app-matrix keeps HEAD's _synapse_extra_args accumulator (the superset: CA
mount plus the onion case) but adopts the incoming improvement inside it — the
inlined regex_replace gives way to lookup('subnet_gateway').
handlers/__init__.py drops rescue_diagnostics_dir from the old import group
(the incoming side moved it under .infinito.rescue, keeping both would import
it twice) and keeps tor_socks_port, a HEAD-only handler ORDERED_HANDLERS still
needs.
svc-bkp-volume-2-local takes the incoming side throughout: both branches landed
the same feature, but swarm-nfs evolved it — the knob moved from the global
STORAGE_BACKUP_SNAPSHOT_MODE to services.volume-2-local.snapshot_mode, plus the
subvolume helper, 04_snapshot_subject.yml, meta/packages.yml and baudolo 3.2.2.
Three files that left behind were removed: files/baudolo_snapshot.py and the two
worktree tests at their pre-refactor paths, all byte-identical to their moved
counterparts.
Six follow-on failures the merge produced, fixed at the root:
- STORAGE_BACKUP_SNAPSHOT_MODE lost its last reader with the config-lookup
design and is gone from group_vars.
- utils/github/variant_bundle_limits.py sat next to variant_bundles.py, which
the prefix lint forbids; the pair now lives in utils/github/variant/ as
bundles.py and bundle_limits.py, with the -m entry point, the ruff
per-file-ignore, the design-doc link and the mirroring test path updated.
- The incoming WordPress addon spec navigated with a raw page.goto and carried
three bare timeout literals; both route through gotoOnion and resolveTimeout
now, which is what makes navigation survive onion transport.
- Six unmarked `|| true` got a concrete reason each: docker.socket is absent on
distros that ship docker without it, the loop device is already detached when
the probe failed before attaching, and modprobe zfs may fail because the
/dev/zfs check decides.
make quality green across all four targets. The cold-domains budget test that
had to be waived here passes on its own now — 19d56f07e raises it to 100s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The nextcloud drill watched /dev/root go from 40G used to 124G used with 21G
free on a 145G disk shared by five nested nodes, the NFS export, both backup
repos and a 14.2GB pull. Four staging trees survived the whole run - grep for
`rm -rf` found nothing after the secrets leg - so the drill kept every copy it
made until the containers went away.
DR_RESTORE_STAGE, DR_VOL_STAGE, DR_SEC_STAGE and RESTORE_ROOT are released
after their last reader, each checked individually against the line that reads
it. All four sit on the success path: `set -e` aborts before them on any
failure, so a drill that fails still hands its evidence to the rescue
collector.
The device step now measures free space before it starts and refuses with both
numbers rather than running into ENOSPC inside a nested systemd unit. The
threshold is USB_SIZE_MB, which the script already computes, so no new constant
enters - and it states the real ledger: the drill holds the pulled tree twice,
once for the device sync and again while the restore root exists alongside the
image. It sits after 02_luks_device.sh because that is where the only stale-loop
reaper runs; a guard placed earlier would abort forever on a leaked loop device
and the reap that would clear it would never happen.
Reading free space is split from evaluating it. Under pipefail a df that
rejects --output fails the command substitution and set -e kills the drill
before the check can speak - which is how the guard managed to produce exactly
the silence it exists to prevent. Driving the body against six df behaviours
(roomy, tight, no header, option unsupported, empty, non-numeric) is what
surfaced that.
This does not address why the observed run failed. It died in step [4/9] with
21G still free, so this guard would not have fired on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill's remote pull exits 1 after 65 minutes and nothing says why. The
unit loops over two providers and only counts errors, so which one failed is
unrecoverable; the pull runs `rsync -abP`, whose --progress writes one journal
line per file - 77137 of them on a nextcloud pull - and the drill captured
`journalctl -n 60`. The entire diagnostic was therefore rsync filenames.
The unit now names the host it could not pull from. rsync keeps --partial and
trades --progress for --info=stats2, turning the flood into a closing balance.
trigger_units.sh writes the complete journal of a failed unit into the rescue
dump directory the artifact ships as local-dumps, and prints only the lines
from warning upwards.
That directory arrives as an argument instead of a second copy of the paths
SPOT: the drill reads INFINITO_RESCUE_DIAGNOSTICS_DIR once, which the env
handler derives from group_vars/all/05_paths.yml, and hands it to every call
site. A test pins that contract, and it earned its place immediately - it found
two call sites that a grep for the file name cannot see, and both were the
units this change exists for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
