baudolo 7.1.x writes volume generations with `rsync -aHAX`, so the
producing host keeps `user.*` extended attributes (OpenCloud keeps its
storage metadata in `user.oc.*`). Every later copy dropped them again,
because none of the other rsync calls passed `-X` (nor `-A`/`-H`):
- NFS export snapshots (svc-bkp-nfs-2-local `script.sh`),
- the backup host's pull (svc-bkp-remote-2-local `pull_specific_host.py`),
- the device sync (svc-bkp-local-2-device `script.py`),
- every recover into a live target (`utils/recovery/base.py`, used by the
volume, NFS and device recover scripts),
- the `recover full` and `recover remote` tree pulls.
A disaster recovery through the documented chain therefore restored
OpenCloud without `user.oc.*`, silently: the copies succeeded and the
dirval stamp does not cover attributes.
All of these now run `rsync -aHAX` (`-aHAXP` for the device sync). A
target filesystem without xattr support makes rsync exit non-zero as soon
as a source file carries attributes, so the hop fails loudly instead of
dropping them; there is no per-device opt-out. The single-image pull in
`recover remote` keeps `-a`. The provider's ssh-wrapper needs no change:
its glob accepts any server flags and its character guard allows them.
Two tests write a `user.oc.id` attribute and assert it survives the NFS
snapshot and the recover; with the old `-a` both fail with
`OSError: [Errno 61] No data available`.
Closes#1021
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
CodeQL reports cmd and sort_spec as possibly uninitialised after the except branch that calls parser.error, because it does not model the argparse error method as NoReturn. It is: usage is printed and the process exits with status 2. An explicit return 2 on that branch states the exit status the call already produces, so the analyser and a reader see the same path.
The retry loop in the registry client counted two attempts in a variable the body never read; the underscore prefix was the convention for that, but CodeQL only honours the bare name.
No path changes behaviour; the three alerts stop pointing at correct code so the scanner stays readable for the ones that are not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A recent refactor imported baudolo into utils.recovery for three layout strings, and cli.administration.recover.__main__ imports full unconditionally - so the whole recovery CLI, --help included, died at import on every host without baudolo. That is most of them: recover device|nfs|secrets runs on swarm nodes, the NFS server and bare rescue hosts, while only the backup node carries svc-bkp-volume-2-local. utils/recovery had been deliberately stdlib-only for exactly that reason.
baudolo 5.0.0 writes a manifest into every generation stating the layout it used, so the names are now asked of the generation being read rather than of the installed package. A tree written by any version is walked with the names that version used, and utils.recovery.layout answers only for generations older than the manifest, which cannot describe themselves. That also retires two of core's guesses: the '<entity>_database' naming convention now yields to the engine the run actually recorded.
A repo whose manifest cannot be read is skipped rather than guessed at, but no longer takes the other repos with it - recovering two of three beats recovering none because one manifest is corrupt. The exit code reports the gap.
The pin lint compared two declared version strings and was therefore green while a different distribution was installed. It now checks the installed distribution and the layout literals against baudolo's own BackupPaths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
utils/recovery restated three things baudolo owns: the generation layout (suffixes, sql/ and files/ directories), the \connect parser for cluster dumps, and the databases.csv reader. Each was a second implementation of a contract whose author is one pip install away - the connect parser had even received the same quoted-identifier fix twice, once per repo.
The layout constants are now derived from BackupPaths, the inventory comes from dump_inventory, and the CSV goes through read_rows. dump_inventory is stricter than the parser it replaces: it also reads CREATE DATABASE lines, so a database a stream creates but never reconnects to is found - a new test pins that. recover/full.py and the drill's token.py take their globs from the same constants instead of spelling them out.
The dependency this creates is declared: backup-docker-to-local==4.0.0 in the base dependencies, because the deploy CLI must import from the base set alone. The role installs the same version, and a lint holds the two pins together - they drifted apart is exactly the failure this whole commit removes, so it must not come back one level up.
The generation-layout tests now write their fixtures through BackupPaths itself; before, the fixture hand-built the paths with the same assumption the test claimed to verify.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A backup generation stores databases as sql dumps, not as a file tree
(--dump-only-sql), and the recover chain only ever restored trees:
`git grep sql HEAD -- cli/administration/recover utils/recovery` was
empty. An operator running `recover full` after a real loss got their
files back, their databases untouched, and an exit code of 0.
Adds the missing leg as a fifth recoverer, planned right after the
volume steps so the dumps land once the volumes are back. Three
properties make it a restore rather than an attempt:
- It refuses while a consumer of that database runs. `--empty`
pre-cleans the schema in one psql session and replays in the next; a
booting consumer recreates it in between and the dump's own CREATE
TABLE fails. That is what turned CI red with `relation
"ar_internal_metadata" already exists`.
- The engine is read, not guessed: the service key in the consuming
role's meta/services.yml, the same value that fed the backup seed.
The two previous call sites each sniffed it from the container image
and could disagree.
- The dump's own header states which server it came out of, and that is
compared against the running engine before the pre-clean drops
anything. Forward across a major version is the upgrade path and
stays allowed; backward is refused. The version deliberately does not
travel in databases.csv: that file is written at seed time and would
state the version deployed back then, and a generation copied to
another host carries its header along while a csv row stays behind.
Mind the header formats, they are not symmetric - postgres puts
`Dumped from database version` on line 7, while mariadb's line 2 opens
with mariadb-dump's own version and only the tab-separated `Server
version` line names the server. Both are pinned as test fixtures,
captured from postgres:17-alpine and mariadb:11 themselves.
baudolo is untouched: nothing beyond the credentials it already carries
is read from databases.csv, and no upstream format changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add DIR_BACKUPS to utils.paths and INFINITO_DIR_BACKUPS /
INFINITO_DIR_SECRETS env handlers, derive the recover-console constants
(including NFS_EXPORT_STATE) from utils.paths and the nfs export SPOT,
and switch every remaining self-built /var/lib/infinito subpath - the
secrets recover targets, the swarm extras backup source, the users
token-store candidate chain, the CI recover drill, the swarm NFS
diagnostics and the playwright reports dir - to those constants. Tests
consume the env constants directly; narration comments in the touched
files are dropped per the comment lint.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add cli.administration.recover: one uniform type/source/target command wrapping each svc-bkp-* role files/recover.py (nfs, volume, secrets, device) plus full for whole-tree recovery. Colon-encoded sources are classified by shape; device supports an on-device subpath, a generation, and remote pull over ssh.
Replace the per-host swarm DR-drill scripts (03/06/07/08_recover_*.sh) with the recover CLI in backup/base.sh. Pass the device on-device subpath relative (the CLI reads absolute segments as the restore-root) and load the applications-free .deploy.yml extras twin on the step-10 redeploy so host_vars credentials survive the -e override.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5.5
Alejandro Roman