18 Commits

Author SHA1 Message Date
ec3d1a5046 Release version 3.1.4 2026-07-31 11:40:03 +02:00
37b735cf7b fix(backup): verify the post-stop volume pass by content
Each volume is copied twice into the same destination: once hot with the
container running, once cold after it is stopped. The cold pass is meant to
correct the hot one, but it uses rsync's quick check, which compares size and
whole-second mtime.

A pre-allocated 16 MiB WAL segment never changes size. When its last pre-stop
write and postgres' shutdown checkpoint fall in the same whole second, the cold
pass skips it while still replacing global/pg_control, whose previous write was
seconds earlier. The generation then holds a post-shutdown pg_control pointing at
a checkpoint LSN whose WAL bytes are still zero. Restoring it gives

    invalid record length at 0/3CB7A20: expected at least 24, got 0
    invalid checkpoint record
    PANIC: could not locate a valid checkpoint record at 0/3CB7A20

and postgres crash-loops until the restore test gives up after 1200s. Observed in
infinito-nexus-core CI run 30586213932, debian leg; the centos leg of the same
run restored cleanly, and the two generations differ by exactly one 16 MiB WAL
segment.

authoritative is a required keyword rather than an optional flag, so each of the
four call sites states which pass it is. Set, it adds --checksum, so the cold
pass compares content. The hot passes keep the quick check.

Reproduced with real rsync in both directions: without --checksum the WAL stays
zero while pg_control is post-shutdown; with it the pair is consistent. Hardlink
dedup against --link-dest is unaffected, measured at 20/20 with generation size
unchanged.

Rejected alternatives, both measured rather than argued: --ignore-times destroys
dedup (20/20 to 0/20 hardlinks, generation 36 to 72 MiB), and --modify-window=-1
makes correctness depend on the destination filesystem preserving sub-second
mtimes, which silently degrades on NFS or ext3.

COST. The quick check scales with file count; --checksum scales with bytes, read
on both sides, and it runs while the container is stopped. Measured warm-cache on
215 MiB: 0.10s to 0.39s, a factor of 3.9. Derived for disk-bound runs, the extra
stop time stays under a minute up to roughly 4 GB on spinning disk, 15 GB on a
SATA SSD, 60 GB on NVMe. At 1 TB it is 17 minutes on NVMe and over three hours on
spinning disk; at 10 TB it is hours to days. No size threshold is built in - a
guessed one would drop the guarantee exactly where an unrestorable backup costs
most. Volumes at that scale need filesystem snapshots or pg_basebackup rather
than two rsync passes over a live tree, which is a limit of this approach and not
of this change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-31 11:32:37 +02:00
9dc57c3235 Release version 3.1.3 2026-07-20 20:01:26 +02:00
8409843ff9 fix(restore): wrap the postgres dump replay in a single transaction
The dump replay ran statement-by-statement with autocommit. When restore --empty runs against a LIVE database, the pre-clean drops a table, the replay recreates it and autocommits, and a concurrent writer (discourse's mini_scheduler upserting scheduler_stats(id=1)) inserts the same primary key into the empty table before the dump's COPY loads it. The COPY then aborts with a duplicate-key violation under ON_ERROR_STOP and the whole restore fails. Running the replay with --single-transaction keeps the recreated table invisible to other sessions until commit, so the writer can never insert the racing row.

The --empty pre-clean stays multi-statement (\gexec, one DROP per statement): running every DROP in one transaction exhausts max_locks_per_transaction on large schemas (e.g. gitlab).

Extract the pre-clean SQL from the inline string into restore/db/empty_preclean.sql (loaded via dirname(__file__)) and declare it as package-data so it ships in the wheel. Add a unit test guarding the single-transaction/multi-statement split and an e2e that reproduces the live-writer race.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 20:00:31 +02:00
d2ba2eb5ae Release version 3.1.2 2026-07-18 00:39:10 +02:00
6a016d7a58 chore(claude): ignore local runtime state under .claude, keep settings.json
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-18 00:38:38 +02:00
d1d5445b1d chore(claude): pause for confirmation before CHANGELOG or pyproject edits
Release metadata (version bump and changelog entry) stays a manual step; agent edits to CHANGELOG.md and pyproject.toml now require explicit operator approval. The local .mcp.json is ignored.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-18 00:37:28 +02:00
bd267cc280 fix(restore): drop text search objects in the postgres --empty pre-clean
A schema shipping a custom text search dictionary (taiga's english_stem_nostop) survived the pre-clean and aborted the dump replay with a duplicate pg_ts_dict_dictname_index violation under ON_ERROR_STOP. The discovery SELECT now also enumerates user-owned pg_ts_config and pg_ts_dict entries. The string-assertion unit test is replaced by real scenario data in the e2e: the seeded schema now contains an overloaded f()/f(int) pair and the nostop dictionary plus configuration, and the restored database is queried to prove each survives the backup, pre-clean and replay cycle exactly once.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-18 00:34:53 +02:00
53460242d8 Release version 3.1.1 2026-07-17 17:28:54 +02:00
01a00dd791 fix(restore): drop overloaded functions by identity signature in the --empty pre-clean
A signature-less DROP FUNCTION aborts with 'function name is not unique' as soon as a schema overloads a name, killing the whole --empty replay under ON_ERROR_STOP. The function branch now emits name(identity args) via pg_get_function_identity_arguments; because that compound must not be identifier-quoted as a whole, quoting moves from the outer format into each branch, guarded by a unit test that pins the per-branch %I quoting.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 17:27:55 +02:00
779f297c85 Release version 3.1.0 2026-07-15 03:36:59 +02:00
35a4c355fe feat(cli): make --database-containers and --images-no-stop-required optional
Both default to an empty list so a pure file backup needs no dummy
arguments; an empty stop whitelist keeps the conservative stop-all
behavior.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 03:34:53 +02:00
b0ae1aba54 build(make): run the three test suites concurrently and allow e2e subsets
make test now runs clean and build once, then unit, integration and e2e
via a -j3 sub-make over run-only targets so a second clean cannot race
the build. scripts/test-e2e.sh accepts E2E_TEST_PATTERN to run a subset
of the e2e suite.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 03:34:37 +02:00
57d75b1e13 fix(restore): run --empty pre-clean per statement and drop non-public schemas
One DO-block ran every DROP in a single transaction and exhausted
max_locks_per_transaction on large schemas (gitlab); emit one DROP per
row and execute via \gexec instead. Also drop user-owned non-public
schemas so a dump that CREATE SCHEMAs (discourse) does not abort on the
existing schema under ON_ERROR_STOP.

Covered by a new DinD e2e test restoring --empty against a fully
populated database with a non-public schema and every object class.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 03:34:20 +02:00
bf5f6db7c3 Release version 3.0.0 2026-07-12 18:29:49 +02:00
b4d7e7f396 Merge branch 'feat/exact-image-match'
# Conflicts:
#	tests/e2e/helpers.py
#	tests/e2e/test_e2e_dump_only_fallback_to_files.py
#	tests/e2e/test_e2e_images_no_backup_required_early_skip.py
2026-07-12 18:27:05 +02:00
f9776ac47a feat(backup)!: exact --images-* matching and --hard-restart-projects
Match --images-no-stop-required and --images-no-backup-required against
the container's exact .Config.Image instead of a substring, so callers
pass full repo:tag references (registry prefix included) and near-miss
image names no longer flip the stop/skip decision. Rename the opt-in
--hard-compose-restart flag to --hard-restart-projects.

The e2e suite pins a SPOT for the DB images and in-container data dirs
(postgres:alpine at /var/lib/postgresql, mariadb:latest at
/var/lib/mysql) and passes exact image refs to the --images-* flags.

BREAKING CHANGE: --images-* now require exact image references, not
substrings; --hard-compose-restart is renamed to --hard-restart-projects.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 18:19:20 +02:00
8c1a6cc465 refactor(restore)!: restore volume files via direct host rsync
Drop the alpine-rsync helper image: resolve the volume mountpoint with
docker volume inspect and rsync the backup straight into it. Removes the
--rsync-image flag and the E2E_RSYNC_IMAGE pre-pull; the e2e test
container now mounts /var/lib/docker rw so the direct restore can write.

BREAKING CHANGE: the restore 'files' subcommand no longer accepts
--rsync-image; rsync must be available on the host running baudolo.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 18:18:57 +02:00
29 changed files with 697 additions and 160 deletions

3
.claude/.gitignore vendored Normal file
View File

@@ -0,0 +1,3 @@
*
!.gitignore
!settings.json

10
.claude/settings.json Normal file
View File

@@ -0,0 +1,10 @@
{
"permissions": {
"ask": [
"Edit(CHANGELOG.md)",
"Write(CHANGELOG.md)",
"Edit(pyproject.toml)",
"Write(pyproject.toml)"
]
}
}

1
.gitignore vendored
View File

@@ -3,3 +3,4 @@ artifacts/
*.egg-info
dist/
build/
.mcp.json

View File

@@ -1,5 +1,106 @@
# Changelog
## [3.1.4] - 2026-07-31
- Backup: the post-stop volume pass now compares by content (*--checksum*), so
it can no longer skip a file the hot pass had copied from a live source. Each
volume is rsynced twice into the same destination — once with the container
running, once after it is stopped — and the second pass used rsync's quick
check (size plus whole-second mtime). A pre-allocated 16 MiB WAL segment never
changes size, so when its last pre-stop write and postgres' shutdown
checkpoint fell in the same whole second, the cold pass skipped it while still
replacing *global/pg_control*, whose previous write was seconds earlier. The
generation then paired a post-shutdown *pg_control* with a WAL segment still
zeroed at the recorded checkpoint LSN, and restoring it crash-looped postgres
with "invalid record length … expected at least 24, got 0" followed by "PANIC:
could not locate a valid checkpoint record". *backup_volume* takes
*authoritative* as a required keyword rather than an optional flag, so each of
the four call sites states which pass it is; the hot passes keep the quick
check. *--ignore-times* was rejected because it destroys the *--link-dest*
hardlink dedup (measured 20/20 to 0/20, generation size doubled), and
*--modify-window=-1* because it makes correctness depend on the destination
filesystem preserving sub-second mtimes, which degrades silently on NFS or
ext3.
- Cost: the quick check scales with file count, *--checksum* with bytes read on
both sides, and it runs while the container is stopped. The extra stop time
stays under a minute up to roughly 4 GB on spinning disk, 15 GB on a SATA SSD
and 60 GB on NVMe; at 1 TB it is 17 minutes on NVMe and over three hours on
spinning disk. No size threshold is built in, since a guessed one would drop
the guarantee exactly where an unrestorable backup costs most — volumes at
that scale want filesystem snapshots or *pg_basebackup* rather than two rsync
passes over a live tree.
## [3.1.3] - 2026-07-20
- Restore: the postgres dump replay now runs under *--single-transaction*,
so a concurrent writer on a live database can no longer interleave a row
between the replay's table re-create and its *COPY* and trip a "duplicate
key value violates unique constraint" abort under ON_ERROR_STOP. This is
the discourse restore-drill race (*mini_scheduler* upserting
*scheduler_stats(id=1)* mid-restore) that failed the whole restore. The
*--empty* pre-clean stays multi-statement (*\gexec*, one DROP per
statement) because a single DROP transaction exhausts
*max_locks_per_transaction* on large schemas (e.g. gitlab).
- Refactor: the *--empty* pre-clean SQL moves out of the inline Python
string into *src/baudolo/restore/db/empty_preclean.sql* (loaded via
*dirname(__file__)*, declared as package-data so it ships in the wheel).
- Tests: a unit test guards the single-transaction / multi-statement split
(replay carries *--single-transaction*, pre-clean does not); a new e2e
reproduces the live-writer race and asserts the restore survives it.
## [3.1.2] - 2026-07-18
- Restore: the postgres *--empty* pre-clean also drops user-owned text
search configurations and dictionaries (*pg_ts_config*, *pg_ts_dict*),
so a schema shipping a custom dictionary (e.g. taiga's
*english_stem_nostop*) no longer aborts the replay with "duplicate key
value violates unique constraint pg_ts_dict_dictname_index" under
ON_ERROR_STOP.
- Tests: the string-assertion unit test for the pre-clean SQL is replaced
by real scenario data in the e2e: the seeded schema contains an
overloaded *f()/f(int)* pair and the nostop dictionary plus
configuration, and the restored database is queried to prove each
survives the backup, pre-clean and replay cycle exactly once.
## [3.1.1] - 2026-07-17
- Restore: the postgres *--empty* pre-clean drops functions and procedures
by their identity signature (*pg_get_function_identity_arguments*), so a
schema that overloads a function name (e.g. discourse) no longer aborts
the replay with "function name is not unique" under ON_ERROR_STOP.
Identifier quoting moves from the outer DROP format into each object
branch, since the *name(args)* compound must not be quoted as a whole; a
unit test pins the per-branch *%I* quoting so future branches cannot
regress unquoted.
## [3.1.0] - 2026-07-15
- Restore: the postgres *--empty* pre-clean emits one DROP per object and
runs them via *\gexec* instead of a single DO-block, so large schemas
(e.g. gitlab) no longer exhaust *max_locks_per_transaction* in one
transaction. It also drops user-owned non-public schemas, so dumps that
CREATE SCHEMA (e.g. discourse's *discourse_functions*) no longer abort
on the already-existing schema under ON_ERROR_STOP.
- Backup: *--database-containers* and *--images-no-stop-required* are now
optional and default to an empty list, so a pure file backup needs no
dummy arguments; an empty stop whitelist keeps the conservative
stop-all behavior.
- Tests: new e2e test restores *--empty* against a fully populated
database containing a non-public schema and every dropped object class.
*make test* runs the three suites concurrently after a single
clean+build; *E2E_TEST_PATTERN* runs an e2e subset.
## [3.0.0] - 2026-07-12
- Backup: *--images-no-stop-required* and *--images-no-backup-required* now
match a container's exact *.Config.Image* (full *repo:tag*, registry
prefix included) instead of a substring, so a near-miss image name no
longer flips the stop/skip decision. Callers must pass exact image
references. **Breaking.**
- Backup: renamed *--hard-compose-restart* to *--hard-restart-projects*
(its value stays a list of compose project dir names). **Breaking:** the
old flag name is removed.
## [2.0.0] - 2026-07-12
- Backup: renamed *--docker-compose-hard-restart-required* to

View File

@@ -1,5 +1,6 @@
.PHONY: install build \
test-e2e test test-unit test-integration
.PHONY: install build clean \
test test-unit test-integration test-e2e \
test-unit-run test-integration-run test-e2e-run
# Default python if no venv is active
PY_DEFAULT ?= python3
@@ -33,25 +34,32 @@ build:
clean:
git clean -fdX .
# ------------------------------------------------------------
# Run E2E tests inside the container (Docker socket required)
# ------------------------------------------------------------
# E2E via isolated Docker-in-Docker (DinD)
# - depends on local image build
# - starts a DinD daemon container on a dedicated network
# - loads the freshly built image into DinD
# - runs the unittest suite inside a container that talks to DinD via DOCKER_HOST
test-e2e: clean build
@bash scripts/test-e2e.sh
# clean + build run once and in order, then the three suites run concurrently
# via -j3; the *-run targets carry no clean/build prereq so the sub-make cannot
# race a second clean against build.
test:
@$(MAKE) clean
@$(MAKE) build
@$(MAKE) -j3 test-unit-run test-integration-run test-e2e-run
test: test-unit test-integration test-e2e
test-unit: clean build test-unit-run
test-unit: clean build
test-integration: clean build test-integration-run
test-e2e: clean build test-e2e-run
test-unit-run:
@echo ">> Running unit tests"
@docker run --rm -t $(IMAGE) \
bash -lc 'python -m unittest discover -t . -s tests/unit -p "test_*.py" -v'
test-integration: clean build
test-integration-run:
@echo ">> Running integration tests"
@docker run --rm -t $(IMAGE) \
bash -lc 'python -m unittest discover -t . -s tests/integration -p "test_*.py" -v'
# E2E via isolated Docker-in-Docker (DinD): starts a DinD daemon on a dedicated
# network, loads the freshly built image into it, and runs tests/e2e inside a
# container that talks to DinD via DOCKER_HOST.
test-e2e-run:
@bash scripts/test-e2e.sh

View File

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "backup-docker-to-local"
version = "2.0.0"
version = "3.1.4"
description = "Backup Docker volumes to local with rsync and optional DB dumps."
readme = "README.md"
requires-python = ">=3.9"
@@ -27,3 +27,6 @@ package-dir = { "" = "src" }
[tool.setuptools.packages.find]
where = ["src"]
exclude = ["tests*"]
[tool.setuptools.package-data]
"baudolo.restore.db" = ["*.sql"]

View File

@@ -37,6 +37,9 @@ KEEP_ON_FAIL="${E2E_KEEP_ON_FAIL:-0}"
KEEP_VOLUMES="${E2E_KEEP_VOLUMES:-0}"
DEBUG_SHELL="${E2E_DEBUG_SHELL:-0}"
# Override to run a subset, e.g. E2E_TEST_PATTERN=test_e2e_postgres_empty_drop_hard.py
TEST_PATTERN="${E2E_TEST_PATTERN:-test_*.py}"
FAILED=0
TS="$(date +%Y%m%d%H%M%S)"
@@ -194,6 +197,7 @@ else
docker run --rm \
--network "${NET}" \
-e DOCKER_HOST="${DIND_HOST_IN_NET}" \
-e E2E_TEST_PATTERN="${TEST_PATTERN}" \
-v "${DIND_VOL}:/var/lib/docker" \
-v "${E2E_TMP_VOL}:/tmp" \
"${IMG}" \
@@ -209,7 +213,7 @@ else
cat /proc/sys/kernel/random/uuid > /etc/machine-id
fi
python -m unittest discover -t . -s tests/e2e -p "test_*.py" -v -f
python -m unittest discover -t . -s tests/e2e -p "${E2E_TEST_PATTERN}" -v -f
'
rc=$?
fi

View File

@@ -52,7 +52,7 @@ def is_image_ignored(container: str, images_no_backup_required: list[str]) -> bo
if not images_no_backup_required:
return False
img = get_image_info(container)
return any(pat in img for pat in images_no_backup_required)
return img in images_no_backup_required
def volume_is_fully_ignored(
@@ -68,15 +68,15 @@ def volume_is_fully_ignored(
def requires_stop(containers: list[str], images_no_stop_required: list[str]) -> bool:
"""
Stop is required if ANY stoppable container image is NOT in the
whitelist patterns. Swarm task containers never count: baudolo must
Stop is required if ANY stoppable container image is NOT in the exact
image whitelist. Swarm task containers never count: baudolo must
not cycle them (see docker.is_swarm_task).
"""
for c in containers:
if is_swarm_task(c):
continue
img = get_image_info(c)
if not any(pat in img for pat in images_no_stop_required):
if img not in images_no_stop_required:
return True
return False
@@ -226,19 +226,19 @@ def main() -> int:
if args.everything:
# "everything": always do pre-rsync, then stop + rsync again
stoppable = filter_stoppable(containers)
backup_volume(versions_dir, volume_name, vol_dir)
backup_volume(versions_dir, volume_name, vol_dir, authoritative=False)
change_containers_status(stoppable, "stop")
backup_volume(versions_dir, volume_name, vol_dir)
backup_volume(versions_dir, volume_name, vol_dir, authoritative=True)
if not args.shutdown:
change_containers_status(stoppable, "start")
continue
# default: rsync, and if needed stop + rsync
backup_volume(versions_dir, volume_name, vol_dir)
backup_volume(versions_dir, volume_name, vol_dir, authoritative=False)
if requires_stop(containers, args.images_no_stop_required):
stoppable = filter_stoppable(containers)
change_containers_status(stoppable, "stop")
backup_volume(versions_dir, volume_name, vol_dir)
backup_volume(versions_dir, volume_name, vol_dir, authoritative=True)
if not args.shutdown:
change_containers_status(stoppable, "start")
@@ -247,6 +247,6 @@ def main() -> int:
print("Finished volume backups.", flush=True)
print("Handling Docker Compose services...", flush=True)
handle_docker_compose_services(args.compose_dir, args.hard_compose_restart)
handle_docker_compose_services(args.compose_dir, args.hard_restart_projects)
return 0

View File

@@ -17,7 +17,7 @@ def parse_args() -> argparse.Namespace:
help="Path to the parent directory containing docker-compose setups",
)
p.add_argument(
"--hard-compose-restart",
"--hard-restart-projects",
nargs="*",
default=[],
help="Compose dir names that require 'docker-compose down && up -d' (default: none; pass e.g. 'mailu' under compose where the DB cannot be backed up hot)",
@@ -42,20 +42,20 @@ def parse_args() -> argparse.Namespace:
p.add_argument(
"--database-containers",
nargs="+",
required=True,
default=[],
help="Container names treated as special instances for database backups",
)
p.add_argument(
"--images-no-stop-required",
nargs="+",
required=True,
help="Image name patterns for which containers should not be stopped during file backup",
default=[],
help="Exact image references (repo:tag, incl. any registry prefix) whose containers must not be stopped during file backup",
)
p.add_argument(
"--images-no-backup-required",
nargs="+",
default=[],
help="Image name patterns for which no backup should be performed",
help="Exact image references (repo:tag, incl. any registry prefix) for which no backup should be performed",
)
p.add_argument(

View File

@@ -24,16 +24,26 @@ def get_last_backup_dir(
return None
def backup_volume(versions_dir: str, volume_name: str, volume_dir: str) -> None:
"""Perform incremental file backup of a Docker volume."""
def backup_volume(
versions_dir: str, volume_name: str, volume_dir: str, *, authoritative: bool
) -> None:
"""Perform incremental file backup of a Docker volume.
Args:
authoritative: compare source and destination by content instead of by
size and whole-second mtime. Required on a pass whose destination was
already written from a live source, where a file can differ while
both attributes still agree.
"""
dest = os.path.join(volume_dir, "files") + "/"
pathlib.Path(dest).mkdir(parents=True, exist_ok=True)
last = get_last_backup_dir(versions_dir, volume_name, dest)
link_dest = f"--link-dest='{last}'" if last else ""
source = get_storage_path(volume_name)
verify = "--checksum " if authoritative else ""
cmd = f"rsync -abP --delete --delete-excluded {link_dest} {source} {dest}"
cmd = f"rsync -abP --delete --delete-excluded {verify}{link_dest} {source} {dest}"
try:
execute_shell_command(cmd)

View File

@@ -0,0 +1,64 @@
-- Owner-filtered pre-clean for `restore --empty`. Emitted as one DROP per row and
-- run via \gexec so each executes as its own top-level statement: a single DO-block
-- would run every DROP in one transaction and exhaust max_locks_per_transaction on
-- large schemas (e.g. gitlab). Also drops user-owned non-public schemas so a dump
-- that CREATE SCHEMAs (e.g. discourse's discourse_functions) does not fail on an
-- already-existing schema. Extension members (pg_trgm's set_limit) are
-- superuser-owned; IF EXISTS absorbs the CASCADE fallout.
SELECT format('DROP %s IF EXISTS public.%s CASCADE', obj.type, obj.name)
FROM (
SELECT format('%I', c.relname) AS name,
CASE c.relkind
WHEN 'v' THEN 'VIEW'
WHEN 'm' THEN 'MATERIALIZED VIEW'
WHEN 'f' THEN 'FOREIGN TABLE'
ELSE 'TABLE'
END AS type
FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'public' AND c.relkind IN ('r', 'p', 'v', 'm', 'f')
AND pg_get_userbyid(c.relowner) = current_user
UNION ALL
-- Overloaded functions share a proname; DROP needs the identity
-- signature or psql aborts with "function name is not unique".
SELECT format('%I(%s)', p.proname, pg_get_function_identity_arguments(p.oid)) AS name,
CASE p.prokind WHEN 'p' THEN 'PROCEDURE' ELSE 'FUNCTION' END AS type
FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace
WHERE n.nspname = 'public' AND p.prokind IN ('f', 'p', 'w')
AND pg_get_userbyid(p.proowner) = current_user
UNION ALL
SELECT format('%I', c.relname) AS name, 'SEQUENCE' AS type
FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'public' AND c.relkind = 'S'
AND pg_get_userbyid(c.relowner) = current_user
UNION ALL
SELECT format('%I', t.typname) AS name, 'TYPE' AS type
FROM pg_type t JOIN pg_namespace n ON n.oid = t.typnamespace
WHERE n.nspname = 'public'
AND pg_get_userbyid(t.typowner) = current_user
AND (t.typtype IN ('e', 'd')
OR (t.typtype = 'c' AND EXISTS (
SELECT 1 FROM pg_class c2
WHERE c2.oid = t.typrelid AND c2.relkind = 'c')))
UNION ALL
SELECT format('%I', col.collname) AS name, 'COLLATION' AS type
FROM pg_collation col JOIN pg_namespace n ON n.oid = col.collnamespace
WHERE n.nspname = 'public'
AND pg_get_userbyid(col.collowner) = current_user
UNION ALL
SELECT format('%I', ts.cfgname) AS name, 'TEXT SEARCH CONFIGURATION' AS type
FROM pg_ts_config ts JOIN pg_namespace n ON n.oid = ts.cfgnamespace
WHERE n.nspname = 'public'
AND pg_get_userbyid(ts.cfgowner) = current_user
UNION ALL
SELECT format('%I', d.dictname) AS name, 'TEXT SEARCH DICTIONARY' AS type
FROM pg_ts_dict d JOIN pg_namespace n ON n.oid = d.dictnamespace
WHERE n.nspname = 'public'
AND pg_get_userbyid(d.dictowner) = current_user
) obj
UNION ALL
SELECT format('DROP SCHEMA IF EXISTS %I CASCADE', n.nspname)
FROM pg_namespace n
WHERE NOT starts_with(n.nspname, 'pg_')
AND n.nspname NOT IN ('public', 'information_schema')
AND pg_get_userbyid(n.nspowner) = current_user
\gexec

View File

@@ -7,6 +7,7 @@ from collections.abc import Iterable, Iterator
from ..run import docker_exec
_SUPERUSER_ONLY_PREFIXES = (b"COMMENT ON EXTENSION", b"ALTER DEFAULT PRIVILEGES")
_EMPTY_PRECLEAN_SQL = os.path.join(os.path.dirname(__file__), "empty_preclean.sql")
def filter_superuser_only_lines(lines: Iterable[bytes]) -> Iterator[bytes]:
@@ -53,51 +54,8 @@ def restore_postgres_sql(
docker_env = {"PGPASSWORD": password}
if empty:
# Owner-filtered: extension members (pg_trgm's set_limit) are superuser-owned; IF EXISTS absorbs CASCADE fallout.
drop_sql = r"""
DO $$ DECLARE r RECORD;
BEGIN
FOR r IN (
SELECT c.relname AS name,
CASE c.relkind
WHEN 'v' THEN 'VIEW'
WHEN 'm' THEN 'MATERIALIZED VIEW'
WHEN 'f' THEN 'FOREIGN TABLE'
ELSE 'TABLE'
END AS type
FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'public' AND c.relkind IN ('r', 'p', 'v', 'm', 'f')
AND pg_get_userbyid(c.relowner) = current_user
UNION ALL
SELECT p.proname AS name,
CASE p.prokind WHEN 'p' THEN 'PROCEDURE' ELSE 'FUNCTION' END AS type
FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace
WHERE n.nspname = 'public' AND p.prokind IN ('f', 'p', 'w')
AND pg_get_userbyid(p.proowner) = current_user
UNION ALL
SELECT c.relname AS name, 'SEQUENCE' AS type
FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'public' AND c.relkind = 'S'
AND pg_get_userbyid(c.relowner) = current_user
UNION ALL
SELECT t.typname AS name, 'TYPE' AS type
FROM pg_type t JOIN pg_namespace n ON n.oid = t.typnamespace
WHERE n.nspname = 'public'
AND pg_get_userbyid(t.typowner) = current_user
AND (t.typtype IN ('e', 'd')
OR (t.typtype = 'c' AND EXISTS (
SELECT 1 FROM pg_class c2
WHERE c2.oid = t.typrelid AND c2.relkind = 'c')))
UNION ALL
SELECT col.collname AS name, 'COLLATION' AS type
FROM pg_collation col JOIN pg_namespace n ON n.oid = col.collnamespace
WHERE n.nspname = 'public'
AND pg_get_userbyid(col.collowner) = current_user
) LOOP
EXECUTE format('DROP %s IF EXISTS public.%I CASCADE', r.type, r.name);
END LOOP;
END $$;
"""
with open(_EMPTY_PRECLEAN_SQL, encoding="utf-8") as preclean:
drop_sql = preclean.read()
docker_exec(
container,
["psql", "-v", "ON_ERROR_STOP=1", "-U", user, "-d", db_name],
@@ -114,7 +72,16 @@ END $$;
filtered.seek(0)
docker_exec(
container,
["psql", "-v", "ON_ERROR_STOP=1", "-U", user, "-d", db_name],
[
"psql",
"--single-transaction",
"-v",
"ON_ERROR_STOP=1",
"-U",
user,
"-d",
db_name,
],
stdin=filtered,
docker_env=docker_env,
)

View File

@@ -180,7 +180,7 @@ def backup_run(
"baudolo",
"--compose-dir",
compose_dir,
"--hard-compose-restart",
"--hard-restart-projects",
"mailu",
"--repo-name",
repo_name,

View File

@@ -81,7 +81,7 @@ class TestE2EDumpOnlyFallbackToFiles(unittest.TestCase):
"baudolo",
"--compose-dir",
cls.compose_dir,
"--hard-compose-restart",
"--hard-restart-projects",
"mailu",
"--repo-name",
cls.repo_name,
@@ -92,10 +92,7 @@ class TestE2EDumpOnlyFallbackToFiles(unittest.TestCase):
"--database-containers",
cls.pg_container,
"--images-no-stop-required",
"postgres",
"mariadb",
"mysql",
"alpine",
POSTGRES_IMAGE,
"--dump-only-sql",
]
cp = run(cmd, capture=True, check=True)

View File

@@ -124,10 +124,7 @@ class TestE2EDumpOnlySqlMixedRun(unittest.TestCase):
"--database-containers",
cls.pg_container,
"--images-no-stop-required",
"alpine",
"postgres",
"mariadb",
"mysql",
POSTGRES_IMAGE,
"--dump-only-sql",
"--backups-dir",
cls.backups_dir,

View File

@@ -57,7 +57,7 @@ class TestE2EFilesFull(unittest.TestCase):
compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv,
database_containers=["dummy-db"],
images_no_stop_required=["alpine", "postgres", "mariadb", "mysql"],
images_no_stop_required=["alpine:3.20"],
)
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)

View File

@@ -55,7 +55,7 @@ class TestE2EFilesNoCopy(unittest.TestCase):
compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv,
database_containers=["dummy-db"],
images_no_stop_required=["alpine", "postgres", "mariadb", "mysql"],
images_no_stop_required=["alpine:3.20"],
dump_only_sql=True,
)

View File

@@ -76,7 +76,7 @@ class TestE2EImagesNoBackupRequiredEarlySkip(unittest.TestCase):
"baudolo",
"--compose-dir",
cls.compose_dir,
"--hard-compose-restart",
"--hard-restart-projects",
"mailu",
"--repo-name",
cls.repo_name,
@@ -87,13 +87,9 @@ class TestE2EImagesNoBackupRequiredEarlySkip(unittest.TestCase):
"--database-containers",
"dummy-db",
"--images-no-stop-required",
"alpine",
"redis",
"postgres",
"mariadb",
"mysql",
"redis:alpine",
"--images-no-backup-required",
"redis",
"redis:alpine",
]
cp = run(cmd, capture=True, check=True)
cls.stdout = cp.stdout or ""

View File

@@ -99,7 +99,7 @@ class TestE2EMariaDBFull(unittest.TestCase):
compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv,
database_containers=[cls.db_container],
images_no_stop_required=["mariadb", "mysql", "alpine", "postgres"],
images_no_stop_required=[MARIADB_IMAGE],
)
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)

View File

@@ -96,7 +96,7 @@ class TestE2EMariaDBNoCopy(unittest.TestCase):
compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv,
database_containers=[cls.db_container],
images_no_stop_required=["mariadb", "mysql", "alpine", "postgres"],
images_no_stop_required=[MARIADB_IMAGE],
dump_only_sql=True,
)

View File

@@ -0,0 +1,186 @@
# tests/e2e/test_e2e_postgres_empty_drop_hard.py
import unittest
from .helpers import (
POSTGRES_IMAGE,
POSTGRES_DATA_DIR,
backup_run,
cleanup_docker,
create_minimal_compose_dir,
ensure_empty_dir,
latest_version_dir,
require_docker,
run,
unique,
wait_for_postgres,
write_databases_csv,
)
# The scenario the --empty pre-clean must survive: a non-public user schema
# plus one object of every class the discovery SELECT enumerates. Restore
# --empty runs against the still-populated DB (no wipe), so the pre-clean must
# drop discourse_functions too or the dump's CREATE SCHEMA aborts the replay
# under ON_ERROR_STOP; the old public-only DROP left it and broke discourse.
# The f()/f(int) pair reproduces discourse's overload abort ("function name
# is not unique") and english_stem_nostop reproduces taiga's text search
# dictionary abort (duplicate pg_ts_dict_dictname_index).
SCENARIO_SQL = (
"CREATE SCHEMA discourse_functions;"
"CREATE TABLE discourse_functions.helper (id int);"
"INSERT INTO discourse_functions.helper VALUES (1);"
"CREATE TABLE public.t (id int primary key, v text);"
"INSERT INTO public.t VALUES (1, 'ok');"
"CREATE VIEW public.t_view AS SELECT * FROM public.t;"
"CREATE SEQUENCE public.s;"
"CREATE TYPE public.mood AS ENUM ('ok', 'bad');"
"CREATE FUNCTION public.f() RETURNS int LANGUAGE sql AS 'SELECT 1';"
"CREATE FUNCTION public.f(i int) RETURNS int LANGUAGE sql AS 'SELECT i';"
"CREATE COLLATION public.c (locale = 'C');"
"CREATE TEXT SEARCH DICTIONARY public.english_stem_nostop"
" (Template = snowball, Language = english);"
"CREATE TEXT SEARCH CONFIGURATION public.english_nostop (COPY = english);"
"ALTER TEXT SEARCH CONFIGURATION public.english_nostop"
" ALTER MAPPING FOR asciiword WITH public.english_stem_nostop;"
)
class TestE2EPostgresEmptyDropHard(unittest.TestCase):
@classmethod
def setUpClass(cls) -> None:
require_docker()
cls.prefix = unique("baudolo-e2e-postgres-empty-drop-hard")
cls.backups_dir = f"/tmp/{cls.prefix}/Backups"
ensure_empty_dir(cls.backups_dir)
cls.compose_dir = create_minimal_compose_dir(f"/tmp/{cls.prefix}")
cls.repo_name = cls.prefix
cls.pg_container = f"{cls.prefix}-pg"
cls.pg_volume = f"{cls.prefix}-pg-vol"
cls.containers = [cls.pg_container]
cls.volumes = [cls.pg_volume]
run(["docker", "volume", "create", cls.pg_volume])
run(
[
"docker",
"run",
"-d",
"--name",
cls.pg_container,
"-e",
"POSTGRES_PASSWORD=pgpw",
"-e",
"POSTGRES_DB=appdb",
"-e",
"POSTGRES_USER=postgres",
"-v",
f"{cls.pg_volume}:{POSTGRES_DATA_DIR}",
POSTGRES_IMAGE,
]
)
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)
run(
[
"docker",
"exec",
cls.pg_container,
"sh",
"-lc",
f'psql -U postgres -d appdb -v ON_ERROR_STOP=1 -c "{SCENARIO_SQL}"',
]
)
cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv"
write_databases_csv(
cls.databases_csv, [(cls.pg_container, "appdb", "postgres", "pgpw")]
)
backup_run(
backups_dir=cls.backups_dir,
repo_name=cls.repo_name,
compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv,
database_containers=[cls.pg_container],
images_no_stop_required=[POSTGRES_IMAGE],
)
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)
# No wipe: restore --empty must pre-clean the fully-populated DB
# (incl. the non-public schema) before replaying the dump.
run(
[
"baudolo-restore",
"postgres",
cls.pg_volume,
cls.hash,
cls.version,
"--backups-dir",
cls.backups_dir,
"--repo-name",
cls.repo_name,
"--container",
cls.pg_container,
"--db-name",
"appdb",
"--db-user",
"postgres",
"--db-password",
"pgpw",
"--empty",
]
)
@classmethod
def tearDownClass(cls) -> None:
cleanup_docker(containers=cls.containers, volumes=cls.volumes)
def _scalar(self, sql: str) -> str:
p = run(
[
"docker",
"exec",
self.pg_container,
"sh",
"-lc",
f'psql -U postgres -d appdb -t -A -c "{sql}"',
]
)
return (p.stdout or "").strip()
def test_public_data_restored(self) -> None:
self.assertEqual(self._scalar("SELECT v FROM public.t WHERE id=1;"), "ok")
def test_view_restored(self) -> None:
self.assertEqual(self._scalar("SELECT count(*) FROM public.t_view;"), "1")
def test_non_public_schema_restored(self) -> None:
self.assertEqual(
self._scalar(
"SELECT count(*) FROM pg_namespace WHERE nspname='discourse_functions';"
),
"1",
)
def test_overloaded_functions_restored_once_each(self) -> None:
self.assertEqual(
self._scalar("SELECT count(*) FROM pg_proc WHERE proname='f';"), "2"
)
self.assertEqual(self._scalar("SELECT public.f(41) + public.f();"), "42")
def test_text_search_dictionary_restored_once(self) -> None:
self.assertEqual(
self._scalar(
"SELECT count(*) FROM pg_ts_dict WHERE dictname='english_stem_nostop';"
),
"1",
)
self.assertEqual(
self._scalar(
"SELECT count(*) FROM pg_ts_config WHERE cfgname='english_nostop';"
),
"1",
)
if __name__ == "__main__":
unittest.main()

View File

@@ -78,7 +78,7 @@ class TestE2EPostgresFull(unittest.TestCase):
compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv,
database_containers=[cls.pg_container],
images_no_stop_required=["postgres", "mariadb", "mysql", "alpine"],
images_no_stop_required=[POSTGRES_IMAGE],
)
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)

View File

@@ -76,7 +76,7 @@ class TestE2EPostgresNoCopy(unittest.TestCase):
compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv,
database_containers=[cls.pg_container],
images_no_stop_required=["postgres", "mariadb", "mysql", "alpine"],
images_no_stop_required=[POSTGRES_IMAGE],
dump_only_sql=True,
)

View File

@@ -0,0 +1,185 @@
# tests/e2e/test_e2e_postgres_single_transaction_live_writer.py
import unittest
from .helpers import (
POSTGRES_IMAGE,
POSTGRES_DATA_DIR,
backup_run,
cleanup_docker,
create_minimal_compose_dir,
ensure_empty_dir,
latest_version_dir,
require_docker,
run,
unique,
wait_for_postgres,
write_databases_csv,
)
# The discourse restore-drill race: `restore --empty` replays the dump into a
# LIVE database while a background writer keeps touching a primary-key row
# (discourse's mini_scheduler upserts scheduler_stats(id=1)). Without a
# single-transaction replay, the pre-clean drops the table, the replay recreates
# it and auto-commits, the writer wins the gap and inserts id=1, and the dump's
# COPY of the same id then aborts with a duplicate-key violation under
# ON_ERROR_STOP -> the whole restore fails. The --single-transaction replay keeps
# the recreated table invisible until commit, so the writer can never insert the
# racing row and the restore completes. A wide filler table makes the COPY slow
# enough that the non-transactional variant loses the race deterministically.
SEED_SQL = (
"CREATE TABLE public.scheduler_stats (id int primary key, v text);"
"INSERT INTO public.scheduler_stats VALUES (1, 'from-dump');"
"CREATE TABLE public.filler (id serial primary key, blob text);"
"INSERT INTO public.filler (blob)"
" SELECT repeat('x', 512) FROM generate_series(1, 100000);"
)
WRITER_LOOP = (
"while true; do "
"psql -h 127.0.0.1 -U postgres -d appdb "
"-c \"INSERT INTO public.scheduler_stats(id, v) VALUES (1, 'live') "
'ON CONFLICT (id) DO NOTHING;" >/dev/null 2>&1; '
"done"
)
class TestE2EPostgresSingleTransactionLiveWriter(unittest.TestCase):
@classmethod
def setUpClass(cls) -> None:
require_docker()
cls.prefix = unique("baudolo-e2e-pg-single-txn")
cls.backups_dir = f"/tmp/{cls.prefix}/Backups"
ensure_empty_dir(cls.backups_dir)
cls.compose_dir = create_minimal_compose_dir(f"/tmp/{cls.prefix}")
cls.repo_name = cls.prefix
cls.pg_container = f"{cls.prefix}-pg"
cls.pg_volume = f"{cls.prefix}-pg-vol"
cls.writer = f"{cls.prefix}-writer"
cls.containers = [cls.pg_container, cls.writer]
cls.volumes = [cls.pg_volume]
run(["docker", "volume", "create", cls.pg_volume])
run(
[
"docker",
"run",
"-d",
"--name",
cls.pg_container,
"-e",
"POSTGRES_PASSWORD=pgpw",
"-e",
"POSTGRES_DB=appdb",
"-e",
"POSTGRES_USER=postgres",
"-v",
f"{cls.pg_volume}:{POSTGRES_DATA_DIR}",
POSTGRES_IMAGE,
]
)
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)
run(
[
"docker",
"exec",
cls.pg_container,
"sh",
"-lc",
f'psql -U postgres -d appdb -v ON_ERROR_STOP=1 -c "{SEED_SQL}"',
]
)
cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv"
write_databases_csv(
cls.databases_csv, [(cls.pg_container, "appdb", "postgres", "pgpw")]
)
backup_run(
backups_dir=cls.backups_dir,
repo_name=cls.repo_name,
compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv,
database_containers=[cls.pg_container],
images_no_stop_required=[POSTGRES_IMAGE],
)
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)
run(
[
"docker",
"run",
"-d",
"--name",
cls.writer,
"--network",
f"container:{cls.pg_container}",
"-e",
"PGPASSWORD=pgpw",
POSTGRES_IMAGE,
"sh",
"-lc",
WRITER_LOOP,
]
)
cls.restore = run(
[
"baudolo-restore",
"postgres",
cls.pg_volume,
cls.hash,
cls.version,
"--backups-dir",
cls.backups_dir,
"--repo-name",
cls.repo_name,
"--container",
cls.pg_container,
"--db-name",
"appdb",
"--db-user",
"postgres",
"--db-password",
"pgpw",
"--empty",
],
capture=True,
check=False,
)
run(["docker", "rm", "-f", cls.writer], capture=True, check=False)
@classmethod
def tearDownClass(cls) -> None:
cleanup_docker(containers=cls.containers, volumes=cls.volumes)
def _scalar(self, sql: str) -> str:
p = run(
[
"docker",
"exec",
self.pg_container,
"sh",
"-lc",
f'psql -U postgres -d appdb -t -A -c "{sql}"',
]
)
return (p.stdout or "").strip()
def test_restore_survived_the_live_writer(self) -> None:
self.assertEqual(
self.restore.returncode,
0,
f"restore aborted (duplicate-key race not contained):\n{self.restore.stderr}",
)
def test_primary_key_row_restored(self) -> None:
self.assertEqual(
self._scalar("SELECT count(*) FROM public.scheduler_stats WHERE id=1;"),
"1",
)
if __name__ == "__main__":
unittest.main()

View File

@@ -167,10 +167,7 @@ class TestE2ESeedStarAndDbEntriesBackupPostgres(unittest.TestCase):
"--database-containers",
cls.pg_container,
"--images-no-stop-required",
"alpine",
"postgres",
"mariadb",
"mysql",
POSTGRES_IMAGE,
"--dump-only-sql",
"--backups-dir",
cls.backups_dir,

View File

@@ -279,15 +279,15 @@ class HardRestartArgTests(unittest.TestCase):
def test_default_is_empty(self) -> None:
args = self._parse([])
self.assertEqual(args.hard_compose_restart, [])
self.assertEqual(args.hard_restart_projects, [])
def test_empty_flag_stays_empty(self) -> None:
args = self._parse(["--hard-compose-restart"])
self.assertEqual(args.hard_compose_restart, [])
args = self._parse(["--hard-restart-projects"])
self.assertEqual(args.hard_restart_projects, [])
def test_explicit_names_preserved(self) -> None:
args = self._parse(["--hard-compose-restart", "mailu", "foo"])
self.assertEqual(args.hard_compose_restart, ["mailu", "foo"])
args = self._parse(["--hard-restart-projects", "mailu", "foo"])
self.assertEqual(args.hard_restart_projects, ["mailu", "foo"])
def test_backups_dir_is_required(self) -> None:
import sys

View File

@@ -1,43 +0,0 @@
"""The --empty pre-clean drop must cover every object class a dump can
re-CREATE ahead of tables; a missed class aborts the replay under
ON_ERROR_STOP (OpenProject's ICU collation public.versions_name did)."""
import tempfile
import unittest
from pathlib import Path
from unittest import mock
from baudolo.restore.db import postgres
class TestPostgresEmptyDrop(unittest.TestCase):
def _drop_sql(self) -> str:
with tempfile.NamedTemporaryFile(suffix=".sql") as fh:
Path(fh.name).write_bytes(b"SELECT 1;\n")
with mock.patch.object(postgres, "docker_exec") as run:
postgres.restore_postgres_sql(
container="c",
db_name="db",
user="u",
password="p",
sql_path=fh.name,
empty=True,
)
first_call = run.call_args_list[0]
return first_call.kwargs["stdin"].decode()
def test_drop_covers_collations(self) -> None:
sql = self._drop_sql()
self.assertIn("pg_collation", sql)
self.assertIn("'COLLATION' AS type", sql)
self.assertIn("pg_get_userbyid(col.collowner) = current_user", sql)
def test_drop_still_covers_the_other_classes(self) -> None:
sql = self._drop_sql()
for marker in ("pg_class", "pg_proc", "'SEQUENCE' AS type", "'TYPE' AS type"):
self.assertIn(marker, sql)
self.assertIn("DROP %s IF EXISTS public.%I CASCADE", sql)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,44 @@
import tempfile
import unittest
from unittest.mock import MagicMock, patch
from baudolo.restore.db import postgres as pg_mod
class TestPostgresSingleTransaction(unittest.TestCase):
def test_replay_is_single_transaction_but_preclean_is_not(self) -> None:
calls = []
def _capture(container, argv, **kwargs):
calls.append(argv)
return MagicMock()
with tempfile.NamedTemporaryFile(suffix=".sql") as sql:
sql.write(b"CREATE TABLE t (id int);\nINSERT INTO t VALUES (1);\n")
sql.flush()
with patch.object(pg_mod, "docker_exec", side_effect=_capture):
pg_mod.restore_postgres_sql(
container="db",
db_name="discourse",
user="discourse",
password="pw",
sql_path=sql.name,
empty=True,
)
self.assertEqual(len(calls), 2, f"expected pre-clean + replay: {calls}")
preclean, replay = calls[0], calls[1]
self.assertNotIn(
"--single-transaction",
preclean,
"pre-clean must stay multi-statement or it exhausts max_locks on large schemas",
)
self.assertIn(
"--single-transaction",
replay,
"dump replay must be atomic so a live concurrent writer cannot trip a duplicate-key abort",
)
if __name__ == "__main__":
unittest.main()

View File

@@ -10,13 +10,12 @@ class TestRequiresStop(unittest.TestCase):
def test_requires_stop_false_when_all_images_are_whitelisted(
self, mock_get_image_info, _mock_is_swarm_task
):
# All containers use images containing allowed substrings
mock_get_image_info.side_effect = [
"repo/mastodon:v4",
"repo/wordpress:latest",
]
containers = ["c1", "c2"]
whitelist = ["mastodon", "wordpress"]
whitelist = ["repo/mastodon:v4", "repo/wordpress:latest"]
self.assertFalse(requires_stop(containers, whitelist))
@patch("baudolo.backup.app.get_image_info")
@@ -28,9 +27,17 @@ class TestRequiresStop(unittest.TestCase):
"repo/nginx:latest",
]
containers = ["c1", "c2"]
whitelist = ["mastodon", "wordpress"]
whitelist = ["repo/mastodon:v4", "repo/wordpress:latest"]
self.assertTrue(requires_stop(containers, whitelist))
@patch("baudolo.backup.app.get_image_info")
def test_requires_stop_true_on_substring_only_match(
self, mock_get_image_info, _mock_is_swarm_task
):
mock_get_image_info.return_value = "reg:5000/repo/mastodon:v4"
self.assertTrue(requires_stop(["c1"], ["mastodon"]))
self.assertTrue(requires_stop(["c1"], ["repo/mastodon:v4"]))
@patch("baudolo.backup.app.get_image_info")
def test_requires_stop_true_when_whitelist_empty(
self, mock_get_image_info, _mock_is_swarm_task