The --empty pre-clean picked its candidates by owner, on the stated
assumption that extension members are superuser-owned and would never be
selected. That holds only when a superuser installed the extension. A role
that installs one itself owns its functions, so they were listed for a
one-by-one DROP that postgres refuses:
cannot drop function vector_in(cstring,oid,integer)
because extension vector requires it
Under ON_ERROR_STOP that ends the whole restore, which is how a discourse
generation - it declares the vector extension - became unreplayable.
Membership now comes from pg_depend rather than from ownership. Each branch
carries its oid and classid so one NOT EXISTS covers all seven instead of
seven separate predicates, and the schema branch gets the same guard because
an extension can own a schema too. Skipping the members is enough: the dump's
CREATE EXTENSION IF NOT EXISTS finds the surviving extension either way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The package carried no ruff configuration at all, so it ran on the defaults (E4+E7+E9+F) while infinito-nexus-core, its only consumer, holds itself to a far wider selection. Measured against that selection this tree had 224 findings. It now has none.
The selector list is core's verbatim so both repositories answer to one bar. target-version stays py39 rather than core's py311, because requires-python still declares >=3.9 and pyupgrade would otherwise propose syntax the declared minimum cannot run. Every ignore carries its reason: S603/S607 in particular, since running docker and dump binaries from PATH in list form is this tool's whole job and is already injection-safe.
Two conversions are judgement rather than mechanics. os.path.join(dir, '') was the rsync idiom for a trailing separator, which Path drops, so it becomes an explicit os.sep. os.path.abspath stays where Path.resolve() would follow symlinks and let a symlinked volume test as inside the snapshot subject.
BREAKING CHANGE: VersionMismatch is renamed VersionMismatchError.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A finished generation cannot show whether a volume held a database, nor whether a dump was produced for it: under --only-sql a failed dump falls back to a file copy, and the resulting files/ tree looks like any other copy. The run knows both and threw the knowledge away as a printed warning, leaving every reader to guess from file names.
Each generation now carries a manifest.json stating its layout and, per volume, database / dumped / engine. baudolo.generation is the single place those names are spelled; restore/paths.py, backup/db.py and backup/volume.py stop repeating them. It is deliberately import-free so a consumer can read the manifest with nothing but json, on hosts where this package is not installed.
BREAKING CHANGE: BackupException is renamed BackupError. The rename is atomic across the ten modules that define or import it, three of which also carry the manifest change, so it lands in this commit rather than a separate one that could not import.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two defaults could not be right. --repo-name fell back to the literal 'backup-docker-to-local' while its help promised the git repo folder name, which nothing ever derived. --databases-csv pointed inside the installed package directory, where credentials must not live; when it applied, load_databases_df read a missing file as empty and the run finished without a single dump and without an error. Both are required now, --repo-name in the restore CLI too. The file itself may still be absent - babadcb's tolerance is untouched, only the path must be named.
--everything is withdrawn. Its one effect was to ignore --images-no-stop-required, which is what leaving that list empty already does, and its branch was the default path minus the requires_stop check. No caller, no test, and help and README described it differently.
--dump-only-sql becomes --only-sql, and --only-files joins it as the opposite half: no dumps at all, every volume as files. They form a mutually exclusive group. A host that only copies files has no business holding database passwords, so --databases-csv is not required there and is never read.
The smallest valid argv turned out to be written four times across the test tree; it now lives once. Withdrawn flags are listed in one place and proven to exit 2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The --empty pre-clean is a catalog-wide sweep: it drops every non-template database and every non-pg_ role of the instance. On a dedicated instance that is exactly right, because the dump recreates all of it. On a shared one it destroys databases the dump does not carry, with nothing to restore them from - and no test ever executed that sweep, because the e2e dropped the cluster by hand first and left the pre-clean with zero rows to generate.
Scoping the sweep to the dump's own inventory looks like the fix and is worse. A surviving database that owns or merely grants to one of the dump's roles pins that role in pg_shdepend; DROP OWNED BY only reaches the control database the pre-clean is connected to, so DROP ROLE fails - after phase 1 has already dropped the dump's databases. ON_ERROR_STOP aborts, the replay never starts, and the instance is left half emptied.
So the instance is checked instead. --empty now refuses when the instance holds a database the dump does not carry, names it, and touches nothing. The sweep stays as it was, safe behind that refusal. Reading the dump's inventory needs a real identifier parser: a quoted name may hold spaces, and psql options precede the target of a \\connect line.
The e2e no longer drops the cluster itself, so --empty has to do it and the replay has to put it back; a second pass then adds a foreign database and requires the refusal to leave both it and the restored data alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Writing into a volume's mountpoint only restores it when the mountpoint is the storage. A volume with driver options - NFS, a bind device, tmpfs - keeps the same /var/lib/docker/volumes/<name>/_data path, but docker mounts the real backing store over it on demand and unmounts it again when the last consumer stops. Restoring while nothing holds it lands in the empty directory underneath, is hidden by the next mount, and rsync reports success.
The declaration decides, not the mount table: the driver and its options are true at every moment, where the mount table is only true while a container happens to hold the volume.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A restore with --empty destroys before it replays: the pre-clean drops the schema in one session and the dump goes in the next, with no rollback across the two. A dump the engine cannot parse therefore does not fail harmlessly, it leaves an emptied database behind. The version each side is on decides that up front, so the refusal lands before the first session opens.
Both engines state their origin in the dump's own header and spell it differently. Postgres names the source server; MariaDB opens with mariadb-dump's own version and names the server further down, so matching the first number would read the tool on one engine and the server on the other. A pg_dumpall stream carries no version line of its own at all - the first belongs to the first database's embedded pg_dump output, arbitrarily far down - hence the deep scan.
Restoring forward across a major version stays allowed; that is the upgrade path. Only backward is refused, with --no-version-check as the way out.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ruff was never wired into this repository: no target, no CI step, no pin.
It reported 45 findings across sources and tests, so nothing enforced
what the codebase already mostly followed.
Adds `make ruff` (check + format --check), `make ruff-fix`, and `make
lint` as its alias, and makes `make test` run lint as a fourth parallel
spur. The CI workflow calls `make test`, so it is covered there too. The
linter is pinned in a `lint` extra: a ruff minor bump changes which rules
fire, and with the suite gating on a clean run an unpinned linter would
fail it on an unrelated day.
The 45 findings are fixed rather than configured away. Three needed a
decision instead of the mechanical fix:
- The generation timestamp keeps its local wall clock (DTZ005 waived).
Generations sort by that name, and UTC would order new ones before the
existing ones wherever the offset is positive - "newest generation" is
what every restore path selects on.
- The per-volume `copy` closure now binds volume_name and vol_dir as
default arguments (B023). It only worked because it is called inside
the same iteration.
- The two CLI top-level handlers keep their blind except (BLE001
waived): turning any failure into exit 1 is what a CLI boundary is
for. The two in run.py did not need it and were narrowed to what they
actually catch.
Also drops the comments that restate the code: the section banners in
restore/__main__.py, the filename repeated as line 1 of nine test files,
step narration above the statement it narrates, and a block in app.py
documenting parameters that had moved to another module. What names a
trip-wire stays - the snapshot destination rule, the mysql-binary
absence in MariaDB 11 images, the session-scoped FOREIGN_KEY_CHECKS, the
spooled temp file for multi-GB dumps, and the negative control that
loses its discriminating power if it ever passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A databases.csv row asking for every database of an instance
(database = '*') makes the backup side write <instance>.cluster.backup.sql
via pg_dumpall, and nothing could read it back: the restore CLI knew
files, postgres and mariadb. That dump was stored and unrestorable - a
format whose producer had no consumer.
Adds `baudolo-restore cluster`. Three properties of a cluster stream
shape it, and each one bit during development:
- It recreates databases, and CREATE DATABASE cannot run inside a
transaction block. So unlike the single-database replay this one must
NOT be wrapped in --single-transaction. The unit tests now pin both
contracts against each other.
- It recreates every role including the one the replay connects as, and
the pre-clean cannot drop the role holding its own session. That
single CREATE ROLE is filtered out of the stream while its ALTER ROLE
is kept, because that is what carries the attributes and the password.
Found by running it: the first replay died on `role "postgres"
already exists`.
- --empty means more than for one database: the cluster's databases go
first, then DROP OWNED BY releases what a role still holds in the
control database, then the roles themselves. The order is pinned by a
phase column because \gexec would otherwise emit them interleaved, and
a role cannot be dropped while it still owns a database.
Without --empty the replay stops at the first object that already
exists. Recreating a cluster over a populated one is a decision, not a
default.
The e2e test drills the real thing: two databases and their owning role
are dropped outright and have to come back with their payload and their
ownership intact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dump replay ran statement-by-statement with autocommit. When restore --empty runs against a LIVE database, the pre-clean drops a table, the replay recreates it and autocommits, and a concurrent writer (discourse's mini_scheduler upserting scheduler_stats(id=1)) inserts the same primary key into the empty table before the dump's COPY loads it. The COPY then aborts with a duplicate-key violation under ON_ERROR_STOP and the whole restore fails. Running the replay with --single-transaction keeps the recreated table invisible to other sessions until commit, so the writer can never insert the racing row.
The --empty pre-clean stays multi-statement (\gexec, one DROP per statement): running every DROP in one transaction exhausts max_locks_per_transaction on large schemas (e.g. gitlab).
Extract the pre-clean SQL from the inline string into restore/db/empty_preclean.sql (loaded via dirname(__file__)) and declare it as package-data so it ships in the wheel. Add a unit test guarding the single-transaction/multi-statement split and an e2e that reproduces the live-writer race.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A schema shipping a custom text search dictionary (taiga's english_stem_nostop) survived the pre-clean and aborted the dump replay with a duplicate pg_ts_dict_dictname_index violation under ON_ERROR_STOP. The discovery SELECT now also enumerates user-owned pg_ts_config and pg_ts_dict entries. The string-assertion unit test is replaced by real scenario data in the e2e: the seeded schema now contains an overloaded f()/f(int) pair and the nostop dictionary plus configuration, and the restored database is queried to prove each survives the backup, pre-clean and replay cycle exactly once.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A signature-less DROP FUNCTION aborts with 'function name is not unique' as soon as a schema overloads a name, killing the whole --empty replay under ON_ERROR_STOP. The function branch now emits name(identity args) via pg_get_function_identity_arguments; because that compound must not be identifier-quoted as a whole, quoting moves from the outer format into each branch, guarded by a unit test that pins the per-branch %I quoting.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One DO-block ran every DROP in a single transaction and exhausted
max_locks_per_transaction on large schemas (gitlab); emit one DROP per
row and execute via \gexec instead. Also drop user-owned non-public
schemas so a dump that CREATE SCHEMAs (discourse) does not abort on the
existing schema under ON_ERROR_STOP.
Covered by a new DinD e2e test restoring --empty against a fully
populated database with a non-public schema and every object class.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Restore now resolves the target volume's mountpoint via docker volume
inspect and rsyncs into it directly, mirroring how the backup path
already reads the mountpoint; the alpine-rsync container and the
--rsync-image flag are gone. The e2e harness mounts /var/lib/docker
read-write in the test container so the direct restore can write, the
same way baudolo runs as root on a real host.
BREAKING CHANGE: the restore 'files' subcommand no longer accepts
--rsync-image; the caller must have write access to the docker volume
root (root on the host), which is baudolo's normal privilege.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The drop_sql loop covered relations, routines, sequences and types but
not pg_collation, so a dump's CREATE COLLATION (OpenProject's ICU
public.versions_name) aborted the ON_ERROR_STOP replay with 'collation
already exists'. Add the fifth UNION ALL branch; DROP COLLATION IF
EXISTS public.<name> CASCADE rides the existing loop, and the loop
already drops every user table, so CASCADE fallout is absorbed by the
IF EXISTS no-ops.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Backup: a container that vanishes between the docker ps listing and the
swarm-task inspect (--rm one-shots, task-history GC) no longer aborts the
whole backup run; it counts as not stoppable and is skipped.
Restore: the postgres replay streams the dump through a spooled temp file
instead of buffering it three times in memory (multi-GB dumps OOMed the
restore mid-replay), and the superuser-only line filter is COPY-aware: data
rows inside COPY ... FROM stdin blocks pass through untouched, so a row
that happens to start with COMMENT ON EXTENSION or ALTER DEFAULT PRIVILEGES
is no longer silently dropped.
The e2e runner talks to the DinD daemon through docker exec instead of a
host-published tcp://127.0.0.1:2375: port publishing is unreachable from
sandboxed runners and from hosts with broken loopback publishing, and the
unencrypted root API port disappears from the host. The debug tmp dump
shrinks to tar plus docker cp against the DinD container itself.
New coverage: an e2e reproducing the swarm flake end to end (service task
on the volume, nothing whitelisted: the backup must succeed, the very same
task container must keep running, and the service must never replace a
task), unit tests for the COPY-aware filter, the swarm-task probe including
the vanished-container path, filter_stoppable ordering, and the one-session
FOREIGN_KEY_CHECKS drop assembly. Full suite: 35 unit, 9 integration,
30 e2e green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Restore fixes, both hit by the infinito svc-bkp e2e drill:
- mariadb --empty dropped tables one docker exec at a time with SET
FOREIGN_KEY_CHECKS=0 issued in a separate client session, so the
session-scoped toggle never applied and any FK-referenced parent table
(mailu.users) died with ERROR 1451. Issue the toggle and all DROPs in
one session.
- postgres --empty now drops only current_user-owned objects (extension
members like pg_trgm's set_limit are superuser-owned) with IF EXISTS
absorbing CASCADE fallout, and the replay skips superuser-only dump
lines (COMMENT ON EXTENSION, ALTER DEFAULT PRIVILEGES) that abort an
app-user psql run under ON_ERROR_STOP.
Backup fixes:
- pg_dump now runs with --no-owner --no-privileges so future dumps are
replayable by the owning app user in the first place.
- Swarm task containers are never stopped or started manually: the
orchestrator replaces a stopped task and a later docker start fails on
the detached overlay network. filter_stoppable skips them visibly and
the whitelist stop check ignores them.
Validated end to end against a live infinito compose stack: the full
svc-bkp-volume-2-local drill (verify, restore cycle, sql replay for
mailu, keycloak and one more db) passes with these patches applied.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Avoid passing raw bytes/str via stdin to subprocess.run(), which caused
"'bytes' object has no attribute 'fileno'" and
"stdin and input arguments may not both be used" errors.
If stdin is bytes or str, pass it via input= instead; otherwise forward
stdin unchanged. This fixes Postgres restore failures in E2E tests
without changing productive restore logic.
https://chatgpt.com/share/694ed70d-9e04-800f-8dec-edf08e6e2082
- add `make clean` and run it before `test-e2e` to avoid stale artifacts
- restore: do not hardcode `mysql` for --empty; use detected mariadb/mysql client
- e2e: improve subprocess error output for easier CI debugging
- e2e: adjust MariaDB readiness checks for socket-only root on MariaDB 11
- e2e: add `wait_for_mariadb_sql` and run SQL readiness checks for dedicated TCP user
- e2e: update MariaDB full/no-copy tests to use dedicated user over TCP and consistent credentials
https://chatgpt.com/share/694ecfb8-f8a8-800f-a1c9-a5f410d4ba02
The file restore command previously assumed that the target volume name
was identical to the volume name used in the backup path. This caused
restores to fail (exit code 2) when restoring data from one volume backup
into a different target volume.
Introduce an optional --source-volume argument for `baudolo-restore files`
to explicitly specify the backup source volume while keeping the target
volume unchanged.
- Default behavior remains fully backward-compatible
- Enables restoring backups from volume A into volume B
- Fixes E2E test scenario restoring into a new volume
Tests:
- Update E2E file restore test to use --source-volume
https://chatgpt.com/share/694ec70f-1d7c-800f-b221-9d22e4b0775e
