engine_by_key merges the generation manifest's volume-to-engine entries and
its docstring promises they win, but replay() called it without the
generation it was already holding, so only repository-derived names resolved.
A dump whose names the application model does not claim aborted the drill:
matrix variant 2 seeds five mautrix bridge databases into the central
postgres, their dumps sit in postgres_data under names that are no entity,
and both Matrix#2 CI jobs died on
no database service in the repository claims volume 'postgres_data'
or database 'mautrix_meta_bridge', so its engine is unknown
tearing the stack down mid-cycle and cascading into four more CLI tests.
The drill's marker seeder has passed the generation since dd07275e8, so one
run was resolving engines from two different maps; replay now reads the same
one. A generation without a manifest contributes nothing, exactly as before,
and a corrupt manifest already fails earlier in dumps_of.
The new test pins the shape: a dump only the manifest can place is replayed.
Before the change it raised with the CI message verbatim.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
assert_no_consumers refuses a replay while the compose project still runs
containers, because a booting consumer recreates the schema the pre-clean
just dropped. The restore cycle brings exactly one container up on purpose:
the engine the dump reaches through docker exec. The cluster path already
excluded it; the per-dump path did not.
For an application whose database carries its own name - discourse's database
is 'discourse' inside project 'discourse' - the running container is
discourse-database, so the engine reported itself as the blocker and the
drill could never replay a single-database dump.
The engine is resolved from the dump's volume, which the replay does anyway
a few lines later, so an engine that is down fails the same way as before,
only earlier.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A recent refactor imported baudolo into utils.recovery for three layout strings, and cli.administration.recover.__main__ imports full unconditionally - so the whole recovery CLI, --help included, died at import on every host without baudolo. That is most of them: recover device|nfs|secrets runs on swarm nodes, the NFS server and bare rescue hosts, while only the backup node carries svc-bkp-volume-2-local. utils/recovery had been deliberately stdlib-only for exactly that reason.
baudolo 5.0.0 writes a manifest into every generation stating the layout it used, so the names are now asked of the generation being read rather than of the installed package. A tree written by any version is walked with the names that version used, and utils.recovery.layout answers only for generations older than the manifest, which cannot describe themselves. That also retires two of core's guesses: the '<entity>_database' naming convention now yields to the engine the run actually recorded.
A repo whose manifest cannot be read is skipped rather than guessed at, but no longer takes the other repos with it - recovering two of three beats recovering none because one manifest is corrupt. The exit code reports the gap.
The pin lint compared two declared version strings and was therefore green while a different distribution was installed. It now checks the installed distribution and the layout literals against baudolo's own BackupPaths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
utils/recovery restated three things baudolo owns: the generation layout (suffixes, sql/ and files/ directories), the \connect parser for cluster dumps, and the databases.csv reader. Each was a second implementation of a contract whose author is one pip install away - the connect parser had even received the same quoted-identifier fix twice, once per repo.
The layout constants are now derived from BackupPaths, the inventory comes from dump_inventory, and the CSV goes through read_rows. dump_inventory is stricter than the parser it replaces: it also reads CREATE DATABASE lines, so a database a stream creates but never reconnects to is found - a new test pins that. recover/full.py and the drill's token.py take their globs from the same constants instead of spelling them out.
The dependency this creates is declared: backup-docker-to-local==4.0.0 in the base dependencies, because the deploy CLI must import from the base set alone. The role installs the same version, and a lint holds the two pins together - they drifted apart is exactly the failure this whole commit removes, so it must not come back one level up.
The generation-layout tests now write their fixtures through BackupPaths itself; before, the fixture hand-built the paths with the same assumption the test claimed to verify.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A databases.csv row with database = '*' makes the backup dump a whole instance with pg_dumpall. The recover chain had no path for that file: it sorted cluster dumps out and printed SKIP, so a host whose engine is dumped that way could not be restored at all, and the drill refused to certify it.
Cluster dumps now go back through 'baudolo-restore cluster' with the instance's superuser. Three details decide the shape. A cluster row states '*' where a single-database row names a database, so credentials_of would file every instance under one key and keep the first, hence a lookup keyed by instance. Which applications must be down is stated by the dump's own connect lines and nowhere else, and reading them needs a real identifier parser: a quoted name may hold spaces and psql options precede the target. And the consumers of a cluster sit in the compose project of its engine, not in projects named after each database, so both are checked before anything is dropped.
The version gate moves out. baudolo compares the dump's header against the running server before its own pre-clean, so the copy here was a second implementation of the same rule, free to drift from the one that actually gates the replay. The test asserts the delegation negatively: the replay must not query the engine version itself. The pin follows, and baudolo-restore joins the command assertion.
The docker probing splits into utils/recovery/docker.py, which keeps the replay module under the size limit and gives the two halves one job each. Calls go through the module rather than through imported names, or a test patching the new module would not reach them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A backup generation stores databases as sql dumps, not as a file tree
(--dump-only-sql), and the recover chain only ever restored trees:
`git grep sql HEAD -- cli/administration/recover utils/recovery` was
empty. An operator running `recover full` after a real loss got their
files back, their databases untouched, and an exit code of 0.
Adds the missing leg as a fifth recoverer, planned right after the
volume steps so the dumps land once the volumes are back. Three
properties make it a restore rather than an attempt:
- It refuses while a consumer of that database runs. `--empty`
pre-cleans the schema in one psql session and replays in the next; a
booting consumer recreates it in between and the dump's own CREATE
TABLE fails. That is what turned CI red with `relation
"ar_internal_metadata" already exists`.
- The engine is read, not guessed: the service key in the consuming
role's meta/services.yml, the same value that fed the backup seed.
The two previous call sites each sniffed it from the container image
and could disagree.
- The dump's own header states which server it came out of, and that is
compared against the running engine before the pre-clean drops
anything. Forward across a major version is the upgrade path and
stays allowed; backward is refused. The version deliberately does not
travel in databases.csv: that file is written at seed time and would
state the version deployed back then, and a generation copied to
another host carries its header along while a csv row stays behind.
Mind the header formats, they are not symmetric - postgres puts
`Dumped from database version` on line 7, while mariadb's line 2 opens
with mariadb-dump's own version and only the tab-separated `Server
version` line names the server. Both are pinned as test fixtures,
captured from postgres:17-alpine and mariadb:11 themselves.
baudolo is untouched: nothing beyond the credentials it already carries
is read from databases.csv, and no upstream format changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5