14 Commits

Author SHA1 Message Date
bf5f6db7c3 Release version 3.0.0 2026-07-12 18:29:49 +02:00
b4d7e7f396 Merge branch 'feat/exact-image-match'
# Conflicts:
#	tests/e2e/helpers.py
#	tests/e2e/test_e2e_dump_only_fallback_to_files.py
#	tests/e2e/test_e2e_images_no_backup_required_early_skip.py
2026-07-12 18:27:05 +02:00
f9776ac47a feat(backup)!: exact --images-* matching and --hard-restart-projects
Match --images-no-stop-required and --images-no-backup-required against
the container's exact .Config.Image instead of a substring, so callers
pass full repo:tag references (registry prefix included) and near-miss
image names no longer flip the stop/skip decision. Rename the opt-in
--hard-compose-restart flag to --hard-restart-projects.

The e2e suite pins a SPOT for the DB images and in-container data dirs
(postgres:alpine at /var/lib/postgresql, mariadb:latest at
/var/lib/mysql) and passes exact image refs to the --images-* flags.

BREAKING CHANGE: --images-* now require exact image references, not
substrings; --hard-compose-restart is renamed to --hard-restart-projects.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 18:19:20 +02:00
8c1a6cc465 refactor(restore)!: restore volume files via direct host rsync
Drop the alpine-rsync helper image: resolve the volume mountpoint with
docker volume inspect and rsync the backup straight into it. Removes the
--rsync-image flag and the E2E_RSYNC_IMAGE pre-pull; the e2e test
container now mounts /var/lib/docker rw so the direct restore can write.

BREAKING CHANGE: the restore 'files' subcommand no longer accepts
--rsync-image; rsync must be available on the host running baudolo.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 18:18:57 +02:00
d6d4773fd9 Release version 2.0.0 2026-07-12 14:56:02 +02:00
82913291b6 test(e2e): SPOT db images, PG18 mount layout, drop removed flags
Add a single source of truth in tests/e2e/helpers.py for the database
images and their in-container data dirs (POSTGRES_IMAGE=postgres:alpine,
POSTGRES_DATA_DIR=/var/lib/postgresql, MARIADB_IMAGE=mariadb:latest,
MARIADB_DATA_DIR=/var/lib/mysql) and route every test through it.
postgres:alpine now tracks 18+, which refuses a mount at
/var/lib/postgresql/data and stores data under /var/lib/postgresql, so
the mounts and the marker path move there. Drop the --rsync-image and the
renamed hard-restart flag from the invocations.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 14:54:16 +02:00
e5da813a9f refactor(restore)!: rsync volumes directly on the host, drop alpine-rsync
Restore now resolves the target volume's mountpoint via docker volume
inspect and rsyncs into it directly, mirroring how the backup path
already reads the mountpoint; the alpine-rsync container and the
--rsync-image flag are gone. The e2e harness mounts /var/lib/docker
read-write in the test container so the direct restore can write, the
same way baudolo runs as root on a real host.

BREAKING CHANGE: the restore 'files' subcommand no longer accepts
--rsync-image; the caller must have write access to the docker volume
root (root on the host), which is baudolo's normal privilege.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 14:53:55 +02:00
331931d617 feat(backup)!: opt-in hard restart and mandatory backups-dir
Rename --docker-compose-hard-restart-required to --hard-compose-restart
and change its default from ["mailu"] to [] (nargs="*"): the compose
down/up is now opt-in, so compose hosts pass "mailu" while swarm hosts,
where the dir is a stack whose overlay network collides with compose up,
pass nothing. Make --backups-dir mandatory (no /var/lib/backup/ default)
so a run can never silently target the wrong backup root.

BREAKING CHANGE: the old flag name is removed, the implicit mailu default
is gone, and --backups-dir must be passed explicitly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 14:53:13 +02:00
45d3b0ad7c Release version 1.8.1 2026-07-12 02:59:17 +02:00
fe5bed8254 Merge branch 'main' of github.com:kevinveenbirkenbach/backup-docker-to-local 2026-07-12 02:58:36 +02:00
7a7ec57b54 fix(restore): drop user-owned collations in the --empty pre-clean
The drop_sql loop covered relations, routines, sequences and types but
not pg_collation, so a dump's CREATE COLLATION (OpenProject's ICU
public.versions_name) aborted the ON_ERROR_STOP replay with 'collation
already exists'. Add the fifth UNION ALL branch; DROP COLLATION IF
EXISTS public.<name> CASCADE rides the existing loop, and the loop
already drops every user table, so CASCADE fallout is absorbed by the
IF EXISTS no-ops.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 02:55:29 +02:00
2bbe7d180a Merge pull request #2 from kevinveenbirkenbach/dependabot/github_actions/actions-cad83fc9bf
Bump the actions group with 4 updates
2026-07-11 14:16:28 +02:00
dependabot[bot]
286ef179da Bump the actions group with 4 updates
Bumps the actions group with 4 updates: [actions/checkout](https://github.com/actions/checkout), [actions/upload-artifact](https://github.com/actions/upload-artifact), [docker/login-action](https://github.com/docker/login-action) and [dependabot/fetch-metadata](https://github.com/dependabot/fetch-metadata).


Updates `actions/checkout` from 4 to 7
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v7)

Updates `actions/upload-artifact` from 4 to 7
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v4...v7)

Updates `docker/login-action` from 3 to 4
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v3...v4)

Updates `dependabot/fetch-metadata` from 2 to 3
- [Release notes](https://github.com/dependabot/fetch-metadata/releases)
- [Commits](https://github.com/dependabot/fetch-metadata/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: actions/upload-artifact
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: docker/login-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: dependabot/fetch-metadata
  dependency-version: '3'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-11 07:44:12 +00:00
dependabot[bot]
6cb0b8a548 Bump python from 3.11-slim to 3.14-slim (#1)
Bumps python from 3.11-slim to 3.14-slim.

---
updated-dependencies:
- dependency-name: python
  dependency-version: 3.14-slim
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-11 07:44:09 +00:00
26 changed files with 246 additions and 106 deletions

View File

@@ -22,7 +22,7 @@ jobs:
steps: steps:
- name: Checkout - name: Checkout
uses: actions/checkout@v4 uses: actions/checkout@v7
- name: Show docker info - name: Show docker info
run: | run: |
@@ -35,7 +35,7 @@ jobs:
- name: Upload E2E artifacts (always) - name: Upload E2E artifacts (always)
if: always() if: always()
uses: actions/upload-artifact@v4 uses: actions/upload-artifact@v7
with: with:
name: e2e-artifacts name: e2e-artifacts
path: artifacts path: artifacts
@@ -49,7 +49,7 @@ jobs:
steps: steps:
- name: Checkout (full history for tags) - name: Checkout (full history for tags)
uses: actions/checkout@v4 uses: actions/checkout@v7
with: with:
fetch-depth: 0 fetch-depth: 0
@@ -67,7 +67,7 @@ jobs:
git push -f origin stable git push -f origin stable
- name: Login to GHCR - name: Login to GHCR
uses: docker/login-action@v3 uses: docker/login-action@v4
with: with:
registry: ${{ env.REGISTRY }} registry: ${{ env.REGISTRY }}
username: ${{ github.actor }} username: ${{ github.actor }}

View File

@@ -17,7 +17,7 @@ jobs:
steps: steps:
- name: Fetch dependency metadata - name: Fetch dependency metadata
id: metadata id: metadata
uses: dependabot/fetch-metadata@v2 uses: dependabot/fetch-metadata@v3
with: with:
github-token: ${{ secrets.GITHUB_TOKEN }} github-token: ${{ secrets.GITHUB_TOKEN }}

View File

@@ -1,5 +1,43 @@
# Changelog # Changelog
## [3.0.0] - 2026-07-12
- Backup: *--images-no-stop-required* and *--images-no-backup-required* now
match a container's exact *.Config.Image* (full *repo:tag*, registry
prefix included) instead of a substring, so a near-miss image name no
longer flips the stop/skip decision. Callers must pass exact image
references. **Breaking.**
- Backup: renamed *--hard-compose-restart* to *--hard-restart-projects*
(its value stays a list of compose project dir names). **Breaking:** the
old flag name is removed.
## [2.0.0] - 2026-07-12
- Backup: renamed *--docker-compose-hard-restart-required* to
*--hard-compose-restart* and changed its default from *["mailu"]* to *[]*
(nargs="*"). The compose down/up is now opt-in: compose hosts pass
*mailu* explicitly, while swarm hosts pass nothing, since there the dir is
a stack whose overlay network collides with *compose up*. **Breaking:** the
old flag name is removed and the implicit mailu default is gone.
- Backup: *--backups-dir* is now required (no */var/lib/backup/* default) so
a run can never silently target the wrong backup root. **Breaking.**
- Restore: volume files are rsynced directly into the target volume's
mountpoint (resolved via *docker volume inspect*), mirroring the backup
path; the *alpine-rsync* helper image and the *--rsync-image* flag are
gone. The caller needs write access to the docker volume root (root on the
host, baudolo's normal privilege). **Breaking:** the restore *files*
subcommand no longer accepts *--rsync-image*.
- Tests: the e2e suite tracks *postgres:alpine* (18+, mounted at
*/var/lib/postgresql*) and *mariadb:latest* from a single source of truth.
## [1.8.1] - 2026-07-12
- Restore: the postgres empty mode also drops user-owned collations in
public; dumps containing CREATE COLLATION (e.g. OpenProject's ICU
collation versions_name) no longer abort the replay with 'collation
already exists'.
- Maintenance: base image bumped from python 3.11-slim to 3.14-slim.
## [1.8.0] - 2026-07-11 ## [1.8.0] - 2026-07-11
Swarm-aware backups and replayable restores. Swarm-aware backups and replayable restores.

View File

@@ -1,5 +1,5 @@
# syntax=docker/dockerfile:1 # syntax=docker/dockerfile:1
FROM python:3.11-slim FROM python:3.14-slim
WORKDIR /app WORKDIR /app

View File

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "backup-docker-to-local" name = "backup-docker-to-local"
version = "1.8.0" version = "3.0.0"
description = "Backup Docker volumes to local with rsync and optional DB dumps." description = "Backup Docker volumes to local with rsync and optional DB dumps."
readme = "README.md" readme = "README.md"
requires-python = ">=3.9" requires-python = ">=3.9"

View File

@@ -27,7 +27,6 @@ dind() { docker exec "${DIND}" docker "$@"; }
dind_stdin() { docker exec -i "${DIND}" docker "$@"; } dind_stdin() { docker exec -i "${DIND}" docker "$@"; }
IMG="${E2E_IMAGE:-baudolo:local}" IMG="${E2E_IMAGE:-baudolo:local}"
RSYNC_IMG="${E2E_RSYNC_IMAGE:-ghcr.io/kevinveenbirkenbach/alpine-rsync}"
READY_TIMEOUT_SECONDS="${E2E_READY_TIMEOUT_SECONDS:-120}" READY_TIMEOUT_SECONDS="${E2E_READY_TIMEOUT_SECONDS:-120}"
ARTIFACTS_DIR="${E2E_ARTIFACTS_DIR:-./artifacts}" ARTIFACTS_DIR="${E2E_ARTIFACTS_DIR:-./artifacts}"
@@ -164,10 +163,6 @@ for i in $(seq 1 "${READY_TIMEOUT_SECONDS}"); do
fi fi
done done
log "Pre-pulling helper images in DinD..."
log " - Pulling: ${RSYNC_IMG}"
dind pull "${RSYNC_IMG}"
log "Ensuring alpine exists in DinD (for debug helpers)" log "Ensuring alpine exists in DinD (for debug helpers)"
dind pull alpine:3.20 >/dev/null dind pull alpine:3.20 >/dev/null
@@ -181,8 +176,7 @@ if [ "${DEBUG_SHELL}" = "1" ]; then
docker run --rm -it \ docker run --rm -it \
--network "${NET}" \ --network "${NET}" \
-e DOCKER_HOST="${DIND_HOST_IN_NET}" \ -e DOCKER_HOST="${DIND_HOST_IN_NET}" \
-e E2E_RSYNC_IMAGE="${RSYNC_IMG}" \ -v "${DIND_VOL}:/var/lib/docker" \
-v "${DIND_VOL}:/var/lib/docker:ro" \
-v "${E2E_TMP_VOL}:/tmp" \ -v "${E2E_TMP_VOL}:/tmp" \
"${IMG}" \ "${IMG}" \
bash -lc ' bash -lc '
@@ -200,8 +194,7 @@ else
docker run --rm \ docker run --rm \
--network "${NET}" \ --network "${NET}" \
-e DOCKER_HOST="${DIND_HOST_IN_NET}" \ -e DOCKER_HOST="${DIND_HOST_IN_NET}" \
-e E2E_RSYNC_IMAGE="${RSYNC_IMG}" \ -v "${DIND_VOL}:/var/lib/docker" \
-v "${DIND_VOL}:/var/lib/docker:ro" \
-v "${E2E_TMP_VOL}:/tmp" \ -v "${E2E_TMP_VOL}:/tmp" \
"${IMG}" \ "${IMG}" \
bash -lc ' bash -lc '

View File

@@ -52,7 +52,7 @@ def is_image_ignored(container: str, images_no_backup_required: list[str]) -> bo
if not images_no_backup_required: if not images_no_backup_required:
return False return False
img = get_image_info(container) img = get_image_info(container)
return any(pat in img for pat in images_no_backup_required) return img in images_no_backup_required
def volume_is_fully_ignored( def volume_is_fully_ignored(
@@ -68,15 +68,15 @@ def volume_is_fully_ignored(
def requires_stop(containers: list[str], images_no_stop_required: list[str]) -> bool: def requires_stop(containers: list[str], images_no_stop_required: list[str]) -> bool:
""" """
Stop is required if ANY stoppable container image is NOT in the Stop is required if ANY stoppable container image is NOT in the exact
whitelist patterns. Swarm task containers never count: baudolo must image whitelist. Swarm task containers never count: baudolo must
not cycle them (see docker.is_swarm_task). not cycle them (see docker.is_swarm_task).
""" """
for c in containers: for c in containers:
if is_swarm_task(c): if is_swarm_task(c):
continue continue
img = get_image_info(c) img = get_image_info(c)
if not any(pat in img for pat in images_no_stop_required): if img not in images_no_stop_required:
return True return True
return False return False
@@ -247,8 +247,6 @@ def main() -> int:
print("Finished volume backups.", flush=True) print("Finished volume backups.", flush=True)
print("Handling Docker Compose services...", flush=True) print("Handling Docker Compose services...", flush=True)
handle_docker_compose_services( handle_docker_compose_services(args.compose_dir, args.hard_restart_projects)
args.compose_dir, args.docker_compose_hard_restart_required
)
return 0 return 0

View File

@@ -17,10 +17,10 @@ def parse_args() -> argparse.Namespace:
help="Path to the parent directory containing docker-compose setups", help="Path to the parent directory containing docker-compose setups",
) )
p.add_argument( p.add_argument(
"--docker-compose-hard-restart-required", "--hard-restart-projects",
nargs="+", nargs="*",
default=["mailu"], default=[],
help="Compose dir names that require 'docker-compose down && up -d' (default: mailu)", help="Compose dir names that require 'docker-compose down && up -d' (default: none; pass e.g. 'mailu' under compose where the DB cannot be backed up hot)",
) )
p.add_argument( p.add_argument(
@@ -35,8 +35,8 @@ def parse_args() -> argparse.Namespace:
) )
p.add_argument( p.add_argument(
"--backups-dir", "--backups-dir",
default="/var/lib/backup/", required=True,
help="Backup root directory (default: /var/lib/backup/)", help="Backup root directory (e.g. /var/lib/backup/)",
) )
p.add_argument( p.add_argument(
@@ -49,13 +49,13 @@ def parse_args() -> argparse.Namespace:
"--images-no-stop-required", "--images-no-stop-required",
nargs="+", nargs="+",
required=True, required=True,
help="Image name patterns for which containers should not be stopped during file backup", help="Exact image references (repo:tag, incl. any registry prefix) whose containers must not be stopped during file backup",
) )
p.add_argument( p.add_argument(
"--images-no-backup-required", "--images-no-backup-required",
nargs="+", nargs="+",
default=[], default=[],
help="Image name patterns for which no backup should be performed", help="Exact image references (repo:tag, incl. any registry prefix) for which no backup should be performed",
) )
p.add_argument( p.add_argument(

View File

@@ -38,10 +38,6 @@ def main(argv: list[str] | None = None) -> int:
# ------------------------------------------------------------------ # ------------------------------------------------------------------
p_files = sub.add_parser("files", help="Restore files into a docker volume") p_files = sub.add_parser("files", help="Restore files into a docker volume")
_add_common_backup_args(p_files) _add_common_backup_args(p_files)
p_files.add_argument(
"--rsync-image",
default="ghcr.io/kevinveenbirkenbach/alpine-rsync",
)
p_files.add_argument( p_files.add_argument(
"--source-volume", "--source-volume",
default=None, default=None,
@@ -95,7 +91,6 @@ def main(argv: list[str] | None = None) -> int:
return restore_volume_files( return restore_volume_files(
args.volume_name, args.volume_name,
bp_files.files_dir(), bp_files.files_dir(),
rsync_image=args.rsync_image,
) )
if args.cmd == "postgres": if args.cmd == "postgres":

View File

@@ -88,6 +88,11 @@ BEGIN
OR (t.typtype = 'c' AND EXISTS ( OR (t.typtype = 'c' AND EXISTS (
SELECT 1 FROM pg_class c2 SELECT 1 FROM pg_class c2
WHERE c2.oid = t.typrelid AND c2.relkind = 'c'))) WHERE c2.oid = t.typrelid AND c2.relkind = 'c')))
UNION ALL
SELECT col.collname AS name, 'COLLATION' AS type
FROM pg_collation col JOIN pg_namespace n ON n.oid = col.collnamespace
WHERE n.nspname = 'public'
AND pg_get_userbyid(col.collowner) = current_user
) LOOP ) LOOP
EXECUTE format('DROP %s IF EXISTS public.%I CASCADE', r.type, r.name); EXECUTE format('DROP %s IF EXISTS public.%I CASCADE', r.type, r.name);
END LOOP; END LOOP;

View File

@@ -3,12 +3,10 @@ from __future__ import annotations
import os import os
import sys import sys
from .run import run, docker_volume_exists from .run import docker_volume_exists, run
def restore_volume_files( def restore_volume_files(volume_name: str, backup_files_dir: str) -> int:
volume_name: str, backup_files_dir: str, *, rsync_image: str
) -> int:
if not os.path.isdir(backup_files_dir): if not os.path.isdir(backup_files_dir):
print(f"ERROR: backup files dir not found: {backup_files_dir}", file=sys.stderr) print(f"ERROR: backup files dir not found: {backup_files_dir}", file=sys.stderr)
return 2 return 2
@@ -19,21 +17,21 @@ def restore_volume_files(
else: else:
print(f"Volume {volume_name} already exists.") print(f"Volume {volume_name} already exists.")
# Keep behavior close to the old script: rsync -avv --delete cp = run(
run( ["docker", "volume", "inspect", "--format", "{{ .Mountpoint }}", volume_name],
[ capture=True,
"docker",
"run",
"--rm",
"-v",
f"{volume_name}:/recover/",
"-v",
f"{backup_files_dir}:/backup/",
rsync_image,
"sh",
"-lc",
"rsync -avv --delete /backup/ /recover/",
]
) )
raw = cp.stdout or b""
mountpoint = (raw.decode() if isinstance(raw, bytes) else raw).strip()
if not mountpoint:
print(
f"ERROR: could not resolve mountpoint for volume {volume_name}",
file=sys.stderr,
)
return 2
src = os.path.join(backup_files_dir, "")
dest = os.path.join(mountpoint, "")
run(["rsync", "-avv", "--delete", src, dest])
print("File restore complete.") print("File restore complete.")
return 0 return 0

View File

@@ -7,6 +7,14 @@ import time
import uuid import uuid
from pathlib import Path from pathlib import Path
# SPOT for the database images and their in-container data dirs the e2e
# suite runs against. postgres:alpine tracks latest (18+), which mounts at
# /var/lib/postgresql (not /var/lib/postgresql/data); bump here only.
POSTGRES_IMAGE = "postgres:alpine"
POSTGRES_DATA_DIR = "/var/lib/postgresql"
MARIADB_IMAGE = "mariadb:latest"
MARIADB_DATA_DIR = "/var/lib/mysql"
def run( def run(
cmd: list[str], cmd: list[str],
@@ -172,7 +180,7 @@ def backup_run(
"baudolo", "baudolo",
"--compose-dir", "--compose-dir",
compose_dir, compose_dir,
"--docker-compose-hard-restart-required", "--hard-restart-projects",
"mailu", "mailu",
"--repo-name", "--repo-name",
repo_name, repo_name,

View File

@@ -2,6 +2,8 @@
import unittest import unittest
from .helpers import ( from .helpers import (
POSTGRES_IMAGE,
POSTGRES_DATA_DIR,
backup_path, backup_path,
cleanup_docker, cleanup_docker,
create_minimal_compose_dir, create_minimal_compose_dir,
@@ -50,8 +52,8 @@ class TestE2EDumpOnlyFallbackToFiles(unittest.TestCase):
"-e", "-e",
"POSTGRES_USER=postgres", "POSTGRES_USER=postgres",
"-v", "-v",
f"{cls.pg_volume}:/var/lib/postgresql/data", f"{cls.pg_volume}:{POSTGRES_DATA_DIR}",
"postgres:16", POSTGRES_IMAGE,
] ]
) )
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90) wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)
@@ -65,7 +67,7 @@ class TestE2EDumpOnlyFallbackToFiles(unittest.TestCase):
cls.pg_container, cls.pg_container,
"sh", "sh",
"-lc", "-lc",
f"echo '{cls.marker}' > /var/lib/postgresql/data/marker.txt", f"echo '{cls.marker}' > {POSTGRES_DATA_DIR}/marker.txt",
] ]
) )
@@ -79,7 +81,7 @@ class TestE2EDumpOnlyFallbackToFiles(unittest.TestCase):
"baudolo", "baudolo",
"--compose-dir", "--compose-dir",
cls.compose_dir, cls.compose_dir,
"--docker-compose-hard-restart-required", "--hard-restart-projects",
"mailu", "mailu",
"--repo-name", "--repo-name",
cls.repo_name, cls.repo_name,
@@ -90,10 +92,7 @@ class TestE2EDumpOnlyFallbackToFiles(unittest.TestCase):
"--database-containers", "--database-containers",
cls.pg_container, cls.pg_container,
"--images-no-stop-required", "--images-no-stop-required",
"postgres", POSTGRES_IMAGE,
"mariadb",
"mysql",
"alpine",
"--dump-only-sql", "--dump-only-sql",
] ]
cp = run(cmd, capture=True, check=True) cp = run(cmd, capture=True, check=True)
@@ -116,8 +115,6 @@ class TestE2EDumpOnlyFallbackToFiles(unittest.TestCase):
cls.repo_name, cls.repo_name,
"--source-volume", "--source-volume",
cls.pg_volume, cls.pg_volume,
"--rsync-image",
"ghcr.io/kevinveenbirkenbach/alpine-rsync",
] ]
) )

View File

@@ -1,6 +1,8 @@
import unittest import unittest
from .helpers import ( from .helpers import (
POSTGRES_IMAGE,
POSTGRES_DATA_DIR,
backup_path, backup_path,
cleanup_docker, cleanup_docker,
create_minimal_compose_dir, create_minimal_compose_dir,
@@ -70,8 +72,8 @@ class TestE2EDumpOnlySqlMixedRun(unittest.TestCase):
"-e", "-e",
f"POSTGRES_PASSWORD={cls.pg_password}", f"POSTGRES_PASSWORD={cls.pg_password}",
"-v", "-v",
f"{cls.db_volume}:/var/lib/postgresql/data", f"{cls.db_volume}:{POSTGRES_DATA_DIR}",
"postgres:16-alpine", POSTGRES_IMAGE,
] ]
) )
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90) wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)
@@ -122,10 +124,7 @@ class TestE2EDumpOnlySqlMixedRun(unittest.TestCase):
"--database-containers", "--database-containers",
cls.pg_container, cls.pg_container,
"--images-no-stop-required", "--images-no-stop-required",
"alpine", POSTGRES_IMAGE,
"postgres",
"mariadb",
"mysql",
"--dump-only-sql", "--dump-only-sql",
"--backups-dir", "--backups-dir",
cls.backups_dir, cls.backups_dir,

View File

@@ -57,7 +57,7 @@ class TestE2EFilesFull(unittest.TestCase):
compose_dir=cls.compose_dir, compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv, databases_csv=cls.databases_csv,
database_containers=["dummy-db"], database_containers=["dummy-db"],
images_no_stop_required=["alpine", "postgres", "mariadb", "mysql"], images_no_stop_required=["alpine:3.20"],
) )
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name) cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)
@@ -94,8 +94,6 @@ class TestE2EFilesFull(unittest.TestCase):
self.repo_name, self.repo_name,
"--source-volume", "--source-volume",
self.volume_src, self.volume_src,
"--rsync-image",
"ghcr.io/kevinveenbirkenbach/alpine-rsync",
] ]
) )

View File

@@ -55,7 +55,7 @@ class TestE2EFilesNoCopy(unittest.TestCase):
compose_dir=cls.compose_dir, compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv, databases_csv=cls.databases_csv,
database_containers=["dummy-db"], database_containers=["dummy-db"],
images_no_stop_required=["alpine", "postgres", "mariadb", "mysql"], images_no_stop_required=["alpine:3.20"],
dump_only_sql=True, dump_only_sql=True,
) )

View File

@@ -76,7 +76,7 @@ class TestE2EImagesNoBackupRequiredEarlySkip(unittest.TestCase):
"baudolo", "baudolo",
"--compose-dir", "--compose-dir",
cls.compose_dir, cls.compose_dir,
"--docker-compose-hard-restart-required", "--hard-restart-projects",
"mailu", "mailu",
"--repo-name", "--repo-name",
cls.repo_name, cls.repo_name,
@@ -87,13 +87,9 @@ class TestE2EImagesNoBackupRequiredEarlySkip(unittest.TestCase):
"--database-containers", "--database-containers",
"dummy-db", "dummy-db",
"--images-no-stop-required", "--images-no-stop-required",
"alpine", "redis:alpine",
"redis",
"postgres",
"mariadb",
"mysql",
"--images-no-backup-required", "--images-no-backup-required",
"redis", "redis:alpine",
] ]
cp = run(cmd, capture=True, check=True) cp = run(cmd, capture=True, check=True)
cls.stdout = cp.stdout or "" cls.stdout = cp.stdout or ""

View File

@@ -30,6 +30,8 @@ import pandas
from baudolo.backup import db as db_mod from baudolo.backup import db as db_mod
from .helpers import ( from .helpers import (
MARIADB_IMAGE,
MARIADB_DATA_DIR,
cleanup_docker, cleanup_docker,
require_docker, require_docker,
run, run,
@@ -69,8 +71,8 @@ class TestE2EMariaDBAnonymousPreemption(unittest.TestCase):
"-e", "-e",
f"MARIADB_ROOT_PASSWORD={cls.root_password}", f"MARIADB_ROOT_PASSWORD={cls.root_password}",
"-v", "-v",
f"{cls.db_volume}:/var/lib/mysql", f"{cls.db_volume}:{MARIADB_DATA_DIR}",
"mariadb:12.2", MARIADB_IMAGE,
] ]
) )

View File

@@ -2,6 +2,8 @@
import unittest import unittest
from .helpers import ( from .helpers import (
MARIADB_IMAGE,
MARIADB_DATA_DIR,
backup_run, backup_run,
backup_path, backup_path,
cleanup_docker, cleanup_docker,
@@ -56,8 +58,8 @@ class TestE2EMariaDBFull(unittest.TestCase):
"-e", "-e",
f"MARIADB_PASSWORD={cls.db_password}", f"MARIADB_PASSWORD={cls.db_password}",
"-v", "-v",
f"{cls.db_volume}:/var/lib/mysql", f"{cls.db_volume}:{MARIADB_DATA_DIR}",
"mariadb:11", MARIADB_IMAGE,
] ]
) )
@@ -97,7 +99,7 @@ class TestE2EMariaDBFull(unittest.TestCase):
compose_dir=cls.compose_dir, compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv, databases_csv=cls.databases_csv,
database_containers=[cls.db_container], database_containers=[cls.db_container],
images_no_stop_required=["mariadb", "mysql", "alpine", "postgres"], images_no_stop_required=[MARIADB_IMAGE],
) )
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name) cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)

View File

@@ -2,6 +2,8 @@
import unittest import unittest
from .helpers import ( from .helpers import (
MARIADB_IMAGE,
MARIADB_DATA_DIR,
backup_run, backup_run,
backup_path, backup_path,
cleanup_docker, cleanup_docker,
@@ -55,8 +57,8 @@ class TestE2EMariaDBNoCopy(unittest.TestCase):
"-e", "-e",
f"MARIADB_PASSWORD={cls.db_password}", f"MARIADB_PASSWORD={cls.db_password}",
"-v", "-v",
f"{cls.db_volume}:/var/lib/mysql", f"{cls.db_volume}:{MARIADB_DATA_DIR}",
"mariadb:11", MARIADB_IMAGE,
] ]
) )
@@ -94,7 +96,7 @@ class TestE2EMariaDBNoCopy(unittest.TestCase):
compose_dir=cls.compose_dir, compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv, databases_csv=cls.databases_csv,
database_containers=[cls.db_container], database_containers=[cls.db_container],
images_no_stop_required=["mariadb", "mysql", "alpine", "postgres"], images_no_stop_required=[MARIADB_IMAGE],
dump_only_sql=True, dump_only_sql=True,
) )

View File

@@ -2,6 +2,8 @@
import unittest import unittest
from .helpers import ( from .helpers import (
POSTGRES_IMAGE,
POSTGRES_DATA_DIR,
backup_run, backup_run,
backup_path, backup_path,
cleanup_docker, cleanup_docker,
@@ -47,8 +49,8 @@ class TestE2EPostgresFull(unittest.TestCase):
"-e", "-e",
"POSTGRES_USER=postgres", "POSTGRES_USER=postgres",
"-v", "-v",
f"{cls.pg_volume}:/var/lib/postgresql/data", f"{cls.pg_volume}:{POSTGRES_DATA_DIR}",
"postgres:16", POSTGRES_IMAGE,
] ]
) )
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90) wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)
@@ -76,7 +78,7 @@ class TestE2EPostgresFull(unittest.TestCase):
compose_dir=cls.compose_dir, compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv, databases_csv=cls.databases_csv,
database_containers=[cls.pg_container], database_containers=[cls.pg_container],
images_no_stop_required=["postgres", "mariadb", "mysql", "alpine"], images_no_stop_required=[POSTGRES_IMAGE],
) )
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name) cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)

View File

@@ -2,6 +2,8 @@
import unittest import unittest
from .helpers import ( from .helpers import (
POSTGRES_IMAGE,
POSTGRES_DATA_DIR,
backup_run, backup_run,
backup_path, backup_path,
cleanup_docker, cleanup_docker,
@@ -46,8 +48,8 @@ class TestE2EPostgresNoCopy(unittest.TestCase):
"-e", "-e",
"POSTGRES_USER=postgres", "POSTGRES_USER=postgres",
"-v", "-v",
f"{cls.pg_volume}:/var/lib/postgresql/data", f"{cls.pg_volume}:{POSTGRES_DATA_DIR}",
"postgres:16", POSTGRES_IMAGE,
] ]
) )
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90) wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)
@@ -74,7 +76,7 @@ class TestE2EPostgresNoCopy(unittest.TestCase):
compose_dir=cls.compose_dir, compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv, databases_csv=cls.databases_csv,
database_containers=[cls.pg_container], database_containers=[cls.pg_container],
images_no_stop_required=["postgres", "mariadb", "mysql", "alpine"], images_no_stop_required=[POSTGRES_IMAGE],
dump_only_sql=True, dump_only_sql=True,
) )

View File

@@ -1,6 +1,8 @@
import unittest import unittest
from .helpers import ( from .helpers import (
POSTGRES_IMAGE,
POSTGRES_DATA_DIR,
backup_path, backup_path,
cleanup_docker, cleanup_docker,
create_minimal_compose_dir, create_minimal_compose_dir,
@@ -66,8 +68,8 @@ class TestE2ESeedStarAndDbEntriesBackupPostgres(unittest.TestCase):
"-e", "-e",
f"POSTGRES_PASSWORD={cls.pg_password}", f"POSTGRES_PASSWORD={cls.pg_password}",
"-v", "-v",
f"{cls.db_volume}:/var/lib/postgresql/data", f"{cls.db_volume}:{POSTGRES_DATA_DIR}",
"postgres:16-alpine", POSTGRES_IMAGE,
] ]
) )
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90) wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)
@@ -165,10 +167,7 @@ class TestE2ESeedStarAndDbEntriesBackupPostgres(unittest.TestCase):
"--database-containers", "--database-containers",
cls.pg_container, cls.pg_container,
"--images-no-stop-required", "--images-no-stop-required",
"alpine", POSTGRES_IMAGE,
"postgres",
"mariadb",
"mysql",
"--dump-only-sql", "--dump-only-sql",
"--backups-dir", "--backups-dir",
cls.backups_dir, cls.backups_dir,

View File

@@ -251,5 +251,61 @@ class TestCompose(unittest.TestCase):
) )
class HardRestartArgTests(unittest.TestCase):
"""The hard-restart list defaults to empty (no compose down/up); callers
opt in per dir, e.g. compose hosts pass 'mailu' while swarm hosts, where
the dir is a stack whose overlay network collides with compose up, pass
nothing."""
def _parse(self, extra: List[str]):
import sys
from baudolo.backup import cli
argv = [
"baudolo",
"--compose-dir",
"/tmp",
"--backups-dir",
"/tmp/backup",
"--database-containers",
"postgres",
"--images-no-stop-required",
"redis",
*extra,
]
with patch.object(sys, "argv", argv):
return cli.parse_args()
def test_default_is_empty(self) -> None:
args = self._parse([])
self.assertEqual(args.hard_restart_projects, [])
def test_empty_flag_stays_empty(self) -> None:
args = self._parse(["--hard-restart-projects"])
self.assertEqual(args.hard_restart_projects, [])
def test_explicit_names_preserved(self) -> None:
args = self._parse(["--hard-restart-projects", "mailu", "foo"])
self.assertEqual(args.hard_restart_projects, ["mailu", "foo"])
def test_backups_dir_is_required(self) -> None:
import sys
from baudolo.backup import cli
argv = [
"baudolo",
"--compose-dir",
"/tmp",
"--database-containers",
"postgres",
"--images-no-stop-required",
"redis",
]
with patch.object(sys, "argv", argv), self.assertRaises(SystemExit):
cli.parse_args()
if __name__ == "__main__": if __name__ == "__main__":
unittest.main(verbosity=2) unittest.main(verbosity=2)

View File

@@ -0,0 +1,43 @@
"""The --empty pre-clean drop must cover every object class a dump can
re-CREATE ahead of tables; a missed class aborts the replay under
ON_ERROR_STOP (OpenProject's ICU collation public.versions_name did)."""
import tempfile
import unittest
from pathlib import Path
from unittest import mock
from baudolo.restore.db import postgres
class TestPostgresEmptyDrop(unittest.TestCase):
def _drop_sql(self) -> str:
with tempfile.NamedTemporaryFile(suffix=".sql") as fh:
Path(fh.name).write_bytes(b"SELECT 1;\n")
with mock.patch.object(postgres, "docker_exec") as run:
postgres.restore_postgres_sql(
container="c",
db_name="db",
user="u",
password="p",
sql_path=fh.name,
empty=True,
)
first_call = run.call_args_list[0]
return first_call.kwargs["stdin"].decode()
def test_drop_covers_collations(self) -> None:
sql = self._drop_sql()
self.assertIn("pg_collation", sql)
self.assertIn("'COLLATION' AS type", sql)
self.assertIn("pg_get_userbyid(col.collowner) = current_user", sql)
def test_drop_still_covers_the_other_classes(self) -> None:
sql = self._drop_sql()
for marker in ("pg_class", "pg_proc", "'SEQUENCE' AS type", "'TYPE' AS type"):
self.assertIn(marker, sql)
self.assertIn("DROP %s IF EXISTS public.%I CASCADE", sql)
if __name__ == "__main__":
unittest.main()

View File

@@ -10,13 +10,12 @@ class TestRequiresStop(unittest.TestCase):
def test_requires_stop_false_when_all_images_are_whitelisted( def test_requires_stop_false_when_all_images_are_whitelisted(
self, mock_get_image_info, _mock_is_swarm_task self, mock_get_image_info, _mock_is_swarm_task
): ):
# All containers use images containing allowed substrings
mock_get_image_info.side_effect = [ mock_get_image_info.side_effect = [
"repo/mastodon:v4", "repo/mastodon:v4",
"repo/wordpress:latest", "repo/wordpress:latest",
] ]
containers = ["c1", "c2"] containers = ["c1", "c2"]
whitelist = ["mastodon", "wordpress"] whitelist = ["repo/mastodon:v4", "repo/wordpress:latest"]
self.assertFalse(requires_stop(containers, whitelist)) self.assertFalse(requires_stop(containers, whitelist))
@patch("baudolo.backup.app.get_image_info") @patch("baudolo.backup.app.get_image_info")
@@ -28,9 +27,17 @@ class TestRequiresStop(unittest.TestCase):
"repo/nginx:latest", "repo/nginx:latest",
] ]
containers = ["c1", "c2"] containers = ["c1", "c2"]
whitelist = ["mastodon", "wordpress"] whitelist = ["repo/mastodon:v4", "repo/wordpress:latest"]
self.assertTrue(requires_stop(containers, whitelist)) self.assertTrue(requires_stop(containers, whitelist))
@patch("baudolo.backup.app.get_image_info")
def test_requires_stop_true_on_substring_only_match(
self, mock_get_image_info, _mock_is_swarm_task
):
mock_get_image_info.return_value = "reg:5000/repo/mastodon:v4"
self.assertTrue(requires_stop(["c1"], ["mastodon"]))
self.assertTrue(requires_stop(["c1"], ["repo/mastodon:v4"]))
@patch("baudolo.backup.app.get_image_info") @patch("baudolo.backup.app.get_image_info")
def test_requires_stop_true_when_whitelist_empty( def test_requires_stop_true_when_whitelist_empty(
self, mock_get_image_info, _mock_is_swarm_task self, mock_get_image_info, _mock_is_swarm_task