baudolo 7.1.x writes volume generations with `rsync -aHAX`, so the
producing host keeps `user.*` extended attributes (OpenCloud keeps its
storage metadata in `user.oc.*`). Every later copy dropped them again,
because none of the other rsync calls passed `-X` (nor `-A`/`-H`):
- NFS export snapshots (svc-bkp-nfs-2-local `script.sh`),
- the backup host's pull (svc-bkp-remote-2-local `pull_specific_host.py`),
- the device sync (svc-bkp-local-2-device `script.py`),
- every recover into a live target (`utils/recovery/base.py`, used by the
volume, NFS and device recover scripts),
- the `recover full` and `recover remote` tree pulls.
A disaster recovery through the documented chain therefore restored
OpenCloud without `user.oc.*`, silently: the copies succeeded and the
dirval stamp does not cover attributes.
All of these now run `rsync -aHAX` (`-aHAXP` for the device sync). A
target filesystem without xattr support makes rsync exit non-zero as soon
as a source file carries attributes, so the hop fails loudly instead of
dropping them; there is no per-device opt-out. The single-image pull in
`recover remote` keeps `-a`. The provider's ssh-wrapper needs no change:
its glob accepts any server flags and its character guard allows them.
Two tests write a `user.oc.id` attribute and assert it survives the NFS
snapshot and the recover; with the old `-a` both fail with
`OSError: [Errno 61] No data available`.
Closes#1021
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
engine_by_key merges the generation manifest's volume-to-engine entries and
its docstring promises they win, but replay() called it without the
generation it was already holding, so only repository-derived names resolved.
A dump whose names the application model does not claim aborted the drill:
matrix variant 2 seeds five mautrix bridge databases into the central
postgres, their dumps sit in postgres_data under names that are no entity,
and both Matrix#2 CI jobs died on
no database service in the repository claims volume 'postgres_data'
or database 'mautrix_meta_bridge', so its engine is unknown
tearing the stack down mid-cycle and cascading into four more CLI tests.
The drill's marker seeder has passed the generation since dd07275e8, so one
run was resolving engines from two different maps; replay now reads the same
one. A generation without a manifest contributes nothing, exactly as before,
and a corrupt manifest already fails earlier in dumps_of.
The new test pins the shape: a dump only the manifest can place is replayed.
Before the change it raised with the CI message verbatim.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
assert_no_consumers refuses a replay while the compose project still runs
containers, because a booting consumer recreates the schema the pre-clean
just dropped. The restore cycle brings exactly one container up on purpose:
the engine the dump reaches through docker exec. The cluster path already
excluded it; the per-dump path did not.
For an application whose database carries its own name - discourse's database
is 'discourse' inside project 'discourse' - the running container is
discourse-database, so the engine reported itself as the blocker and the
drill could never replay a single-database dump.
The engine is resolved from the dump's volume, which the replay does anyway
a few lines later, so an engine that is down fails the same way as before,
only earlier.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
json.loads(path.read_text(...)) plus its marker ran to 155 characters, so ruff
format split the inner call across three lines and left the comment sitting
after the closing parenthesis, where the lint no longer reads it as belonging
to the file read. Splitting the read out of the parse keeps the call and its
marker together on one line within the limit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The task name broke the 80-char convention; naming the consumer alone says as much, since the enclosing role already names the application.
The manifest read opts out of the shared read cache rather than joining it. utils.cache.files pulls in PROJECT_ROOT and the gitignore reader, which would break the import-free rule utils/recovery/manifest.py exists under, and a process-wide cache is wrong for this file in any case: a backup run writes a new generation that the same process then reads back, so the cached copy would be the stale one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A recent refactor imported baudolo into utils.recovery for three layout strings, and cli.administration.recover.__main__ imports full unconditionally - so the whole recovery CLI, --help included, died at import on every host without baudolo. That is most of them: recover device|nfs|secrets runs on swarm nodes, the NFS server and bare rescue hosts, while only the backup node carries svc-bkp-volume-2-local. utils/recovery had been deliberately stdlib-only for exactly that reason.
baudolo 5.0.0 writes a manifest into every generation stating the layout it used, so the names are now asked of the generation being read rather than of the installed package. A tree written by any version is walked with the names that version used, and utils.recovery.layout answers only for generations older than the manifest, which cannot describe themselves. That also retires two of core's guesses: the '<entity>_database' naming convention now yields to the engine the run actually recorded.
A repo whose manifest cannot be read is skipped rather than guessed at, but no longer takes the other repos with it - recovering two of three beats recovering none because one manifest is corrupt. The exit code reports the gap.
The pin lint compared two declared version strings and was therefore green while a different distribution was installed. It now checks the installed distribution and the layout literals against baudolo's own BackupPaths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
utils/recovery restated three things baudolo owns: the generation layout (suffixes, sql/ and files/ directories), the \connect parser for cluster dumps, and the databases.csv reader. Each was a second implementation of a contract whose author is one pip install away - the connect parser had even received the same quoted-identifier fix twice, once per repo.
The layout constants are now derived from BackupPaths, the inventory comes from dump_inventory, and the CSV goes through read_rows. dump_inventory is stricter than the parser it replaces: it also reads CREATE DATABASE lines, so a database a stream creates but never reconnects to is found - a new test pins that. recover/full.py and the drill's token.py take their globs from the same constants instead of spelling them out.
The dependency this creates is declared: backup-docker-to-local==4.0.0 in the base dependencies, because the deploy CLI must import from the base set alone. The role installs the same version, and a lint holds the two pins together - they drifted apart is exactly the failure this whole commit removes, so it must not come back one level up.
The generation-layout tests now write their fixtures through BackupPaths itself; before, the fixture hand-built the paths with the same assumption the test claimed to verify.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A databases.csv row with database = '*' makes the backup dump a whole instance with pg_dumpall. The recover chain had no path for that file: it sorted cluster dumps out and printed SKIP, so a host whose engine is dumped that way could not be restored at all, and the drill refused to certify it.
Cluster dumps now go back through 'baudolo-restore cluster' with the instance's superuser. Three details decide the shape. A cluster row states '*' where a single-database row names a database, so credentials_of would file every instance under one key and keep the first, hence a lookup keyed by instance. Which applications must be down is stated by the dump's own connect lines and nowhere else, and reading them needs a real identifier parser: a quoted name may hold spaces and psql options precede the target. And the consumers of a cluster sit in the compose project of its engine, not in projects named after each database, so both are checked before anything is dropped.
The version gate moves out. baudolo compares the dump's header against the running server before its own pre-clean, so the copy here was a second implementation of the same rule, free to drift from the one that actually gates the replay. The test asserts the delegation negatively: the replay must not query the engine version itself. The pin follows, and baudolo-restore joins the command assertion.
The docker probing splits into utils/recovery/docker.py, which keeps the replay module under the size limit and gives the two halves one job each. Calls go through the module rather than through imported names, or a test patching the new module would not reach them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A backup generation stores databases as sql dumps, not as a file tree
(--dump-only-sql), and the recover chain only ever restored trees:
`git grep sql HEAD -- cli/administration/recover utils/recovery` was
empty. An operator running `recover full` after a real loss got their
files back, their databases untouched, and an exit code of 0.
Adds the missing leg as a fifth recoverer, planned right after the
volume steps so the dumps land once the volumes are back. Three
properties make it a restore rather than an attempt:
- It refuses while a consumer of that database runs. `--empty`
pre-cleans the schema in one psql session and replays in the next; a
booting consumer recreates it in between and the dump's own CREATE
TABLE fails. That is what turned CI red with `relation
"ar_internal_metadata" already exists`.
- The engine is read, not guessed: the service key in the consuming
role's meta/services.yml, the same value that fed the backup seed.
The two previous call sites each sniffed it from the container image
and could disagree.
- The dump's own header states which server it came out of, and that is
compared against the running engine before the pre-clean drops
anything. Forward across a major version is the upgrade path and
stays allowed; backward is refused. The version deliberately does not
travel in databases.csv: that file is written at seed time and would
state the version deployed back then, and a generation copied to
another host carries its header along while a csv row stays behind.
Mind the header formats, they are not symmetric - postgres puts
`Dumped from database version` on line 7, while mariadb's line 2 opens
with mariadb-dump's own version and only the tab-separated `Server
version` line names the server. Both are pinned as test fixtures,
captured from postgres:17-alpine and mariadb:11 themselves.
baudolo is untouched: nothing beyond the credentials it already carries
is read from databases.csv, and no upstream format changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every cross-host hop in the backup/recover chain remapped file owners
by NAME on the receiving host (rsync -a default; GNU tar extract as
root prefers name lookup), so a restored uid-1000-owned 0600 config
(n8n) came back owned by a foreign uid and the post-recovery redeploy
crash-looped on EACCES. Pass --numeric-ids to the capture, remote-pull,
device-sync and restore rsyncs and --numeric-owner to the drill's tar
extracts.
The drill additionally wiped and rewrote the export subtree directly on
the ganesha server's backing filesystem while the server kept running
and the nodes kept their client mounts: ganesha's mdcache and the
client dentry caches still described the pre-wipe tree, so update-pass
tasks were Rejected with readdirent ENOENT for the whole converge
window (moodle, espocrm multi-replica). Step 8b now restarts ganesha,
remounts every node and probes a real write through the manager's
mount, each bounded by timeout so a wedged DinD node fails loudly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add cli.administration.recover: one uniform type/source/target command wrapping each svc-bkp-* role files/recover.py (nfs, volume, secrets, device) plus full for whole-tree recovery. Colon-encoded sources are classified by shape; device supports an on-device subpath, a generation, and remote pull over ssh.
Replace the per-host swarm DR-drill scripts (03/06/07/08_recover_*.sh) with the recover CLI in backup/base.sh. Pass the device on-device subpath relative (the CLI reads absolute segments as the restore-root) and load the applications-free .deploy.yml extras twin on the step-10 redeploy so host_vars credentials survive the -e override.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The nfs backup rsyncs the whole export, and the shared backup root
lives inside it (every node's DIR_BACKUPS is the mounted state dir), so
each generation embedded a copy of all previous backups and grew
quadratically; the giant listings also flushed real errors out of
captured journals. The unit now passes the backup root relative to the
export as an anchored rsync exclude, and recover.py protects the live
root from --delete while never replaying embedded stale copies from old
generations (DirectoryRecovery gains rsync_extra_args for that).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add utils/recovery/base.py DirectoryRecovery: a service-backed pre-recover
(start the role's backup unit first) + rsync -a --delete mirror. Each
svc-bkp-* role gets a files/recover.py subclass (volume resolves the docker
mountpoint, nfs mirrors the export, local-2-device opens the newest LUKS
snapshot across machine hashes) or a documented one-way opt-out
(remote-2-local recover.py.nocheck). A lint guard enforces every svc-bkp-*
role ships a recover.py and a ## Recover README section; docs/architecture/
backup.md documents the end-to-end schema. Unit tests cover the base flow,
the per-role class logic and every recover.py main() CLI wiring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
