Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The restore cycle stops every subject of the host, replays into the database
services alone, and starts the rest again at the very end. Any exit in
between skips that last step, so the host keeps its subjects down - and
nothing recorded it.
The next deploy then failed on a symptom of the teardown rather than on
whatever ended the drill. It cost two runs: once 'rake db:migrate' against a
database the cycle had emptied but not yet replayed, once
Redis::CannotConnectError because the cycle never restarted redis. Both times
the message pointed away from the cause, and fixing what it named would have
fixed nothing.
A trap armed before the teardown now records the teardown and the exit code,
and mode=update refuses to run while that record stands. purge=true and
mode=reinstall pass, because both rebuild the state the drill destroyed; a
completed cycle clears the record itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
wait_service_terminated guards two opposite moments with one rule. The call
before the forced backup only has to wait out a run still in flight, while
the calls after a systemctl start read the verdict on the code under test.
Both treated 'failed' as fatal, so a unit left failed by an earlier deploy
ended the drill in the pre-flight - after 0.13s, reporting a state three
hours old as if it were this run's result.
That made the drill blind from its first failure onward: it could never start
another backup, so every defect behind the first one stayed unreachable.
The pre-flight now clears such a state and carries on. After a start of its
own, 'failed' stays fatal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The payload check accepted a files/ tree OR an sql dump, so a database whose dump never happened and which was copied as raw engine files passed it. That is exactly what occurred: Discourse's postgres landed as PG_VERSION/base/pg_wal and the drill reported green, while restoring that replays a torn cluster rather than a dump.
The generation's manifest is now the authority on which volumes held a database and which of those carry a dump; generations written before baudolo recorded that are judged by the engine's on-disk footprint instead. Demonstrated against both: the manifest branch catches a volume the footprint heuristic cannot see, and leaves a non-database volume alone.
seed/marker.py reads dumps and therefore needs baudolo, but ran under a bare python3, which Ansible's PATH resolves to the sealed project venv while sys-pip-install installs into the system interpreter - so it died at import and the entire restore half of the drill never ran. It now uses BKP_TEST_PYTHON, the interpreter the role installed into.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A recent refactor imported baudolo into utils.recovery for three layout strings, and cli.administration.recover.__main__ imports full unconditionally - so the whole recovery CLI, --help included, died at import on every host without baudolo. That is most of them: recover device|nfs|secrets runs on swarm nodes, the NFS server and bare rescue hosts, while only the backup node carries svc-bkp-volume-2-local. utils/recovery had been deliberately stdlib-only for exactly that reason.
baudolo 5.0.0 writes a manifest into every generation stating the layout it used, so the names are now asked of the generation being read rather than of the installed package. A tree written by any version is walked with the names that version used, and utils.recovery.layout answers only for generations older than the manifest, which cannot describe themselves. That also retires two of core's guesses: the '<entity>_database' naming convention now yields to the engine the run actually recorded.
A repo whose manifest cannot be read is skipped rather than guessed at, but no longer takes the other repos with it - recovering two of three beats recovering none because one manifest is corrupt. The exit code reports the gap.
The pin lint compared two declared version strings and was therefore green while a different distribution was installed. It now checks the installed distribution and the layout literals against baudolo's own BackupPaths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
seed/token.py sat on sys.path beside the scripts that ran it, so any import chain reaching tokenize picked up the drill's seeder instead of the stdlib's token module - dataclasses -> inspect -> tokenize -> token is enough to trigger it. marker.py names what the file does and collides with nothing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
utils/recovery restated three things baudolo owns: the generation layout (suffixes, sql/ and files/ directories), the \connect parser for cluster dumps, and the databases.csv reader. Each was a second implementation of a contract whose author is one pip install away - the connect parser had even received the same quoted-identifier fix twice, once per repo.
The layout constants are now derived from BackupPaths, the inventory comes from dump_inventory, and the CSV goes through read_rows. dump_inventory is stricter than the parser it replaces: it also reads CREATE DATABASE lines, so a database a stream creates but never reconnects to is found - a new test pins that. recover/full.py and the drill's token.py take their globs from the same constants instead of spelling them out.
The dependency this creates is declared: backup-docker-to-local==4.0.0 in the base dependencies, because the deploy CLI must import from the base set alone. The role installs the same version, and a lint holds the two pins together - they drifted apart is exactly the failure this whole commit removes, so it must not come back one level up.
The generation-layout tests now write their fixtures through BackupPaths itself; before, the fixture hand-built the paths with the same assumption the test claimed to verify.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill removed each volume and re-created it. Docker refuses to remove a volume any container still references, and a container outside compose is only stopped here, not removed, so discourse's launcher container aborted the run at that point: after the host was already down and part of the volumes were gone. Re-creating a removed volume also drops its driver options, handing an NFS- or bind-backed volume back as a plain local one.
Volumes are emptied in place instead, from inside a throwaway container, and the emptiness is proven in the same call. Nothing has to be re-created, so a launcher container starts again exactly as it was built.
The engine that receives a replay is started by the service name on its own compose label. The previous fixed name existed for no engine on the host: the central Postgres serves project postgres from volume postgres_data, bigbluebutton serves project bigbluebutton from service postgres. Neither is called database, so the drill died at 'up -d database' after the teardown and before any restore.
A cluster dump yields one marker per database it recreates, or a replay that brought back only part of an instance would still pass. The drill runs once, in the sync pass; repeating it in the async pass would only tear the host down twice for the same answer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ASYNC pass fires a second backup run when it finds only one generation.
Generation directories are named by the second, so on a host with little to
copy both runs landed in the same one: baudolo reused the finished generation,
rsync --delete overwrote it, and the stamp guard failed the unit afterwards.
CI job 95119330633 died exactly there.
Wait out the second before the extra run, and take baudolo 3.4.3, which claims
the generation directory exclusively and aborts before the first write instead
of reporting the damage after it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three conflicts, each resolved by combining both sides rather than picking one.
compose.yml passes both env vars through to the container: swarm-nfs adds
INFINITO_SWARM_MGR_IP, this branch adds INFINITO_TOR_SOCKS_PORT, and neither
substitutes for the other.
daemon.json.j2 takes the data-root block from swarm-nfs and keeps HEAD's dns
key. The incoming side writes networks.internet.dns directly; lookup(
'container_dns') already returns that value as its clearnet entry and prepends
the docker0 bridge on a Tor node in compose mode, so the incoming form is a
strict subset and dropping it loses nothing.
The WordPress CA-trust addon spec gains skipUnlessServiceEnabled("sso") from
swarm-nfs alongside HEAD's gotoOnion and resolveTimeout, and asserts on the
Plugin Name the mu-plugin header declares - the mustuse screen renders that,
not the filename - with resolveTimeout applied to the literal the incoming
side left bare.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An untyped inspect searches every docker object kind, and a role's overlay network carries the same entity name as its main container, so the call can resolve the network and report a missing State key instead of naming the container. That is how the openresty assert died on every swarm job of run 30709070050.
Extend container-exec-resolver to cover inspect, stripping raw markers and quoted spans first so the flag walker survives Go template format strings. Add container-inspect-type, scoped to roles/ because the collision follows the entity naming convention and the outer act namespace carries no role networks. Sweep 40 call sites; suppress the delegate_to-resolved CIDs and the compose co-location probes with a stated reason.
Move the five container rules into tests/lint/ansible/shell/container/ and drop the container_ prefix from their filenames. Register both markers in the suppression docs, where the resolver marker was missing entirely.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve conflicts, resolved by intent rather than by side.
Six shared one shape: the incoming side adds the new rescue_index.sh call but
drops this branch's `# nocheck: shell-or-true` markers. The markers carry a
lint contract here, so the incoming structure was taken and the markers put
back — dedicated.sh, guide/one.sh, guide/host_deploy.sh, swarm/routine/00_one.sh
and trigger_units.sh, where the incoming version additionally writes the full
journal to a file instead of only echoing a tail.
web-app-matrix keeps HEAD's _synapse_extra_args accumulator (the superset: CA
mount plus the onion case) but adopts the incoming improvement inside it — the
inlined regex_replace gives way to lookup('subnet_gateway').
handlers/__init__.py drops rescue_diagnostics_dir from the old import group
(the incoming side moved it under .infinito.rescue, keeping both would import
it twice) and keeps tor_socks_port, a HEAD-only handler ORDERED_HANDLERS still
needs.
svc-bkp-volume-2-local takes the incoming side throughout: both branches landed
the same feature, but swarm-nfs evolved it — the knob moved from the global
STORAGE_BACKUP_SNAPSHOT_MODE to services.volume-2-local.snapshot_mode, plus the
subvolume helper, 04_snapshot_subject.yml, meta/packages.yml and baudolo 3.2.2.
Three files that left behind were removed: files/baudolo_snapshot.py and the two
worktree tests at their pre-refactor paths, all byte-identical to their moved
counterparts.
Six follow-on failures the merge produced, fixed at the root:
- STORAGE_BACKUP_SNAPSHOT_MODE lost its last reader with the config-lookup
design and is gone from group_vars.
- utils/github/variant_bundle_limits.py sat next to variant_bundles.py, which
the prefix lint forbids; the pair now lives in utils/github/variant/ as
bundles.py and bundle_limits.py, with the -m entry point, the ruff
per-file-ignore, the design-doc link and the mirroring test path updated.
- The incoming WordPress addon spec navigated with a raw page.goto and carried
three bare timeout literals; both route through gotoOnion and resolveTimeout
now, which is what makes navigation survive onion transport.
- Six unmarked `|| true` got a concrete reason each: docker.socket is absent on
distros that ship docker without it, the loop device is already detached when
the probe failed before attaching, and modprobe zfs may fail because the
/dev/zfs check decides.
make quality green across all four targets. The cold-domains budget test that
had to be waived here passes on its own now — 19d56f07e raises it to 100s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The launcher declined on every btrfs host for a reason no operator could
guess: `/var/lib/docker` is created by mkdir, so it is an ordinary
directory inside the enclosing subvolume, and `btrfs subvolume snapshot`
needs a subvolume root as its source. Documenting that as a manual
prerequisite would have left the feature switched on and never firing.
ensure_subject_subvolume.sh converts the data root on the deploy that
introduces the mode. The copy uses `cp --reflink=always`: both trees live
on the same btrfs, so extents are shared instead of duplicated and the
conversion costs seconds and no extra space no matter how large the data
root is - where an rsync copy would have meant hours of downtime and a
free-space requirement of 110%. The original tree is kept until the daemon
has answered on the converted root, so a failed copy, a mismatched
verification pass or a daemon that will not start all put it back and
start docker on it again.
The mode itself moves from group_vars into meta/services.yml as
`snapshot_mode`, read through `lookup('config', ...)`, so a single host can
override it from the inventory. Falsey spellings normalise to `never` and
anything else fails a deploy-time assert, because YAML reads a bare
off/no/false as a boolean and `--mode False` would kill the unit nightly.
btrfs-progs is installed wherever a btrfs mount exists, with centos opted
out: centos stream ships neither the kernel module nor the package, and
demanding it there would fail the package-availability test for the whole
RedHat family.
The pin moves to 3.2.2, which is not cosmetic: that release carves the
snapshot inside its subject, and the reaper here looks for leftovers in
exactly that place. Against 3.2.1 it would search beside the data root and
never find one.
Two drills run before anything else in test.sh: subvolume_probe.sh proves
the conversion carries every path, payload and symlink across and that the
result can actually be snapshotted; snapshot_probe.sh pins the launcher's
decision, including the refusals that keep a host on the live copy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The launcher decides per run whether baudolo may capture the volumes from
a filesystem snapshot. That decision reads mountinfo, inode numbers and
`btrfs subvolume list`, so a unit test can only mock it - and the one
defect that shipped in this area (baudolo 3.2.0 dropping the trailing
separator) survived precisely because the production path was never
driven end to end.
snapshot_probe.sh builds a throwaway btrfs on a loop device under
mktemp -d, puts a stub on PATH in place of the docker daemon, and asserts
both halves of the contract: a snapshot is taken where it would be
faithful, and where it would not be, `auto` degrades to the live copy
while a stated kind fails the run. It covers the plain case, a volume
declaring its own backing store, a nested subvolume, a `.baudolo-*`
leftover of a killed run, and a plain directory as the data root. The
host's real data root is never touched and a trap releases the mount, the
loop device and the directory.
It runs first in test.sh, before the backup unit is even waited on,
because it depends on nothing a backup produces. Where root or the btrfs
tooling is missing it reports SKIP after checking that baudolo-snapshot
resolves on PATH and that `--mode never` hands argv through unchanged -
on such a host the launcher declines anyway, so there is nothing else to
assert.
The README documents all four drills with a diagram each, and how to run
the snapshot probe from a host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Conflicts: keep both new passthrough env vars; adopt the swarm-nfs
ollama manager-ops split while porting our model-pull retry hardening
into it. Strip three narration comments the stricter comment lint
rejects.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The single-app compose job deploys nothing containerized (all svc-db/svc-prx skipped by the whitelist, DB seed off), so docker volume ls is empty and baudolo stamps an empty backup generation; verify_backup.sh then failed deterministically on 'no volume payload'. The test only ever passed piggybacked on app hosts with real volumes. Probe the host's backup subjects after the forced run: with zero volumes and zero compose containers, assert positively that the newest generation is empty (an empty generation is the correct outcome on a bare host) and skip the restore drill, mirroring the existing non-stack-host gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Back to the pre-swarm-nfs-fix baseline:
- 5c2c8f531 + 985ec7eab: the swarm writer-quiesce DR drill. Deep-dive
attribution of CI run 29715516546 found it the dominant failure source
(17 of 23 jobs): [9b/9] handed-off generation mismatch, [6/9] quiesce
stack-rm network race, [4/9] remote-2-local pull stall, compose discourse
restore duplicate key.
- d6f980574: openproject swarm ingress to web:8080. Carried a leaking
proxy_internal_port lookup (no per-app default) that broke the seaweedfs
dependency front-proxy ('Key web not found'); the bypass-the-Apache-proxy
approach was never verified against downloads/X-Sendfile.
- 1dd153bad: xwiki replicas:1 + mem 3g. Did not resolve the xwiki superadmin
failure (still red in run 29715516546), so it is unverified.
Re-approach each from a clean baseline.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up corrections to b449f8157 (swarm DR drill):
- export test.env in 06/07 via set -a so the child db restore/probe
scripts inherit BKP_TEST_DATABASES_CSV and BKP_TEST_RESTORE_BIN
instead of aborting at their :? guards before any restore runs
- move db_probe.sh/db_restore.sh into a db/ subfolder (probe.sh,
restore.sh) and update all callers, so they stop tripping the
prefix-grouped-files structure lint
- define TEST_E2E_SWARM_DRILL in group_vars/all and drop the now
redundant default() from test.env.j2 (satisfies the undefined-var
and spot-default lints together)
- fix a mermaid parse error in the drill README: a ; in a
sequenceDiagram note was read as a statement separator
make quality-high green (test + lint phases, 15 targets, 0 failures).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Defer Swarm SQL replay to the disposable DR drill so shared writers such as Discourse are stopped before the destructive restore. Verify the selected generation with before/after database probes while preserving the Compose restore path.
The pre-state backup runs in service-loader preload before the app volume exists, so the test could verify against an empty generation. Trigger a fresh backup unit run before the verify and restore cycle so a populated generation is present.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the find_dock_val_by_bkp_entr filter with three role-local
lookups that read the live merged applications config:
backup_database_containers (backup.database_routine -> service names),
backup_image (backup.no_stop_required/disabled -> exact deployed image
refs via the image lookup, swarm registry prefix and custom builds
included), and backup_hard_restart_projects
(backup.project_hard_restart -> compose project names, declared on
mailu instead of hardcoded). Pin baudolo to 3.0.0, whose --images-*
flags match .Config.Image exactly and whose hard-restart flag is
--hard-restart-projects.
The restore drill now starts all compose projects in parallel per
retry round; the previous serial alphabetical loop deadlocked when a
project's migrate service waited on a datastore project that sorted
later (mastodon -> postgres), killing the CLI test at the 3600s
timeout. Volume restore rsyncs into the mountpoint directly, dropping
the rsync helper image remnants from the test env.
BREAKING CHANGE: deploys require backup-docker-to-local 3.0.0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
27 jobs of run 29147179243 burned their full 360-min budget. Three
systemic layers, all addressed here:
- The CLI e2e task ran the staged test.sh unbounded, and test.sh blocks
in 'systemctl start' on a oneshot backup unit with no JobTimeout; on a
wedged dockerd or a stalled NFS mount both freeze silently for hours.
The task now runs the script under 'timeout TEST_E2E_CLI_TIMEOUT'
(60 min via the timeout lookup, so TIMEOUT_FACTOR scales it) and the
unit starts are bounded by BKP_TEST_HEALTH_TIMEOUT.
- The converge probe re-ran the full per-service diagnostics (service ps,
service logs, inspect, journalctl) on EVERY retry, so 600 nominal
2-second attempts cost 25-33s each and silently outlived the step
timeout. The block is gone (the workflow-level container.py rescue
snapshot already collects it recursively on failure) and every docker
call is timeout-bounded. Same removal in the swarm collect script,
which keeps only the NFS and render evidence container.py does not see.
- Playwright-collect, rescue diagnostics and Cleanup had no
timeout-minutes, and teardown.sh calls docker against D-state nodes,
so a failed job hung there for hours until the job cap killed it
without artifacts. Every step is bounded now, every teardown call is
timeout-wrapped, and the deploy steps (swarm 270, compose 300) leave
room for diagnostics and cleanup below the 360-min cap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The container and NFS pre-deploy snapshots ran two near-identical role
scripts and two constructor tasks. Merge them into one shared script under
scripts/system/backup/ whose content check handles a file (databases.csv)
or a directory (export base), and drive both from a single looped
constructor task keyed by role, unit, and source. Each iteration keeps its
own group_names gate, its own timeout, and reports changed only when it did
not skip.
The shared script also reaches the fresh-host skip loudly: systemctl
list-unit-files exits non-zero when the version glob matches nothing, so the
listing now sits in an if instead of a bare command substitution that
aborted the snapshot under set -euo pipefail (the compose test-development
failure). The volume e2e test starts the unit through an if and dumps the
unit journal on failure so the real start error surfaces.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename the role, its test tree, the primary entity key
(volume-2-local), the derived vars (BKP_VOLUME_2_LOCAL_*,
SYS_SERVICE_BACKUP_VOLUME_2_LOCAL), the run-once fact, and every
group_names consumer flag, include path, doc, and lint reference. The
container_backup service key and the on-disk backup-docker-to-local
tree keep their names: both are contracts with existing consumer
entries, snapshot trees, and the baudolo pull tooling.
Operators must rename the svc-bkp-container-2-local inventory group and
disable the previous versioned units on already-provisioned hosts; the
pre-deploy snapshot glob derives its base from the new role name and
will not find them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
