4 Commits

Author SHA1 Message Date
d6d4773fd9 Release version 2.0.0 2026-07-12 14:56:02 +02:00
82913291b6 test(e2e): SPOT db images, PG18 mount layout, drop removed flags
Add a single source of truth in tests/e2e/helpers.py for the database
images and their in-container data dirs (POSTGRES_IMAGE=postgres:alpine,
POSTGRES_DATA_DIR=/var/lib/postgresql, MARIADB_IMAGE=mariadb:latest,
MARIADB_DATA_DIR=/var/lib/mysql) and route every test through it.
postgres:alpine now tracks 18+, which refuses a mount at
/var/lib/postgresql/data and stores data under /var/lib/postgresql, so
the mounts and the marker path move there. Drop the --rsync-image and the
renamed hard-restart flag from the invocations.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 14:54:16 +02:00
e5da813a9f refactor(restore)!: rsync volumes directly on the host, drop alpine-rsync
Restore now resolves the target volume's mountpoint via docker volume
inspect and rsyncs into it directly, mirroring how the backup path
already reads the mountpoint; the alpine-rsync container and the
--rsync-image flag are gone. The e2e harness mounts /var/lib/docker
read-write in the test container so the direct restore can write, the
same way baudolo runs as root on a real host.

BREAKING CHANGE: the restore 'files' subcommand no longer accepts
--rsync-image; the caller must have write access to the docker volume
root (root on the host), which is baudolo's normal privilege.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 14:53:55 +02:00
331931d617 feat(backup)!: opt-in hard restart and mandatory backups-dir
Rename --docker-compose-hard-restart-required to --hard-compose-restart
and change its default from ["mailu"] to [] (nargs="*"): the compose
down/up is now opt-in, so compose hosts pass "mailu" while swarm hosts,
where the dir is a stack whose overlay network collides with compose up,
pass nothing. Make --backups-dir mandatory (no /var/lib/backup/ default)
so a run can never silently target the wrong backup root.

BREAKING CHANGE: the old flag name is removed, the implicit mailu default
is gone, and --backups-dir must be passed explicitly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 14:53:13 +02:00
19 changed files with 146 additions and 67 deletions

View File

@@ -1,5 +1,24 @@
# Changelog # Changelog
## [2.0.0] - 2026-07-12
- Backup: renamed *--docker-compose-hard-restart-required* to
*--hard-compose-restart* and changed its default from *["mailu"]* to *[]*
(nargs="*"). The compose down/up is now opt-in: compose hosts pass
*mailu* explicitly, while swarm hosts pass nothing, since there the dir is
a stack whose overlay network collides with *compose up*. **Breaking:** the
old flag name is removed and the implicit mailu default is gone.
- Backup: *--backups-dir* is now required (no */var/lib/backup/* default) so
a run can never silently target the wrong backup root. **Breaking.**
- Restore: volume files are rsynced directly into the target volume's
mountpoint (resolved via *docker volume inspect*), mirroring the backup
path; the *alpine-rsync* helper image and the *--rsync-image* flag are
gone. The caller needs write access to the docker volume root (root on the
host, baudolo's normal privilege). **Breaking:** the restore *files*
subcommand no longer accepts *--rsync-image*.
- Tests: the e2e suite tracks *postgres:alpine* (18+, mounted at
*/var/lib/postgresql*) and *mariadb:latest* from a single source of truth.
## [1.8.1] - 2026-07-12 ## [1.8.1] - 2026-07-12
- Restore: the postgres empty mode also drops user-owned collations in - Restore: the postgres empty mode also drops user-owned collations in

View File

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "backup-docker-to-local" name = "backup-docker-to-local"
version = "1.8.1" version = "2.0.0"
description = "Backup Docker volumes to local with rsync and optional DB dumps." description = "Backup Docker volumes to local with rsync and optional DB dumps."
readme = "README.md" readme = "README.md"
requires-python = ">=3.9" requires-python = ">=3.9"

View File

@@ -27,7 +27,6 @@ dind() { docker exec "${DIND}" docker "$@"; }
dind_stdin() { docker exec -i "${DIND}" docker "$@"; } dind_stdin() { docker exec -i "${DIND}" docker "$@"; }
IMG="${E2E_IMAGE:-baudolo:local}" IMG="${E2E_IMAGE:-baudolo:local}"
RSYNC_IMG="${E2E_RSYNC_IMAGE:-ghcr.io/kevinveenbirkenbach/alpine-rsync}"
READY_TIMEOUT_SECONDS="${E2E_READY_TIMEOUT_SECONDS:-120}" READY_TIMEOUT_SECONDS="${E2E_READY_TIMEOUT_SECONDS:-120}"
ARTIFACTS_DIR="${E2E_ARTIFACTS_DIR:-./artifacts}" ARTIFACTS_DIR="${E2E_ARTIFACTS_DIR:-./artifacts}"
@@ -164,10 +163,6 @@ for i in $(seq 1 "${READY_TIMEOUT_SECONDS}"); do
fi fi
done done
log "Pre-pulling helper images in DinD..."
log " - Pulling: ${RSYNC_IMG}"
dind pull "${RSYNC_IMG}"
log "Ensuring alpine exists in DinD (for debug helpers)" log "Ensuring alpine exists in DinD (for debug helpers)"
dind pull alpine:3.20 >/dev/null dind pull alpine:3.20 >/dev/null
@@ -181,8 +176,7 @@ if [ "${DEBUG_SHELL}" = "1" ]; then
docker run --rm -it \ docker run --rm -it \
--network "${NET}" \ --network "${NET}" \
-e DOCKER_HOST="${DIND_HOST_IN_NET}" \ -e DOCKER_HOST="${DIND_HOST_IN_NET}" \
-e E2E_RSYNC_IMAGE="${RSYNC_IMG}" \ -v "${DIND_VOL}:/var/lib/docker" \
-v "${DIND_VOL}:/var/lib/docker:ro" \
-v "${E2E_TMP_VOL}:/tmp" \ -v "${E2E_TMP_VOL}:/tmp" \
"${IMG}" \ "${IMG}" \
bash -lc ' bash -lc '
@@ -200,8 +194,7 @@ else
docker run --rm \ docker run --rm \
--network "${NET}" \ --network "${NET}" \
-e DOCKER_HOST="${DIND_HOST_IN_NET}" \ -e DOCKER_HOST="${DIND_HOST_IN_NET}" \
-e E2E_RSYNC_IMAGE="${RSYNC_IMG}" \ -v "${DIND_VOL}:/var/lib/docker" \
-v "${DIND_VOL}:/var/lib/docker:ro" \
-v "${E2E_TMP_VOL}:/tmp" \ -v "${E2E_TMP_VOL}:/tmp" \
"${IMG}" \ "${IMG}" \
bash -lc ' bash -lc '

View File

@@ -247,8 +247,6 @@ def main() -> int:
print("Finished volume backups.", flush=True) print("Finished volume backups.", flush=True)
print("Handling Docker Compose services...", flush=True) print("Handling Docker Compose services...", flush=True)
handle_docker_compose_services( handle_docker_compose_services(args.compose_dir, args.hard_compose_restart)
args.compose_dir, args.docker_compose_hard_restart_required
)
return 0 return 0

View File

@@ -17,10 +17,10 @@ def parse_args() -> argparse.Namespace:
help="Path to the parent directory containing docker-compose setups", help="Path to the parent directory containing docker-compose setups",
) )
p.add_argument( p.add_argument(
"--docker-compose-hard-restart-required", "--hard-compose-restart",
nargs="+", nargs="*",
default=["mailu"], default=[],
help="Compose dir names that require 'docker-compose down && up -d' (default: mailu)", help="Compose dir names that require 'docker-compose down && up -d' (default: none; pass e.g. 'mailu' under compose where the DB cannot be backed up hot)",
) )
p.add_argument( p.add_argument(
@@ -35,8 +35,8 @@ def parse_args() -> argparse.Namespace:
) )
p.add_argument( p.add_argument(
"--backups-dir", "--backups-dir",
default="/var/lib/backup/", required=True,
help="Backup root directory (default: /var/lib/backup/)", help="Backup root directory (e.g. /var/lib/backup/)",
) )
p.add_argument( p.add_argument(

View File

@@ -38,10 +38,6 @@ def main(argv: list[str] | None = None) -> int:
# ------------------------------------------------------------------ # ------------------------------------------------------------------
p_files = sub.add_parser("files", help="Restore files into a docker volume") p_files = sub.add_parser("files", help="Restore files into a docker volume")
_add_common_backup_args(p_files) _add_common_backup_args(p_files)
p_files.add_argument(
"--rsync-image",
default="ghcr.io/kevinveenbirkenbach/alpine-rsync",
)
p_files.add_argument( p_files.add_argument(
"--source-volume", "--source-volume",
default=None, default=None,
@@ -95,7 +91,6 @@ def main(argv: list[str] | None = None) -> int:
return restore_volume_files( return restore_volume_files(
args.volume_name, args.volume_name,
bp_files.files_dir(), bp_files.files_dir(),
rsync_image=args.rsync_image,
) )
if args.cmd == "postgres": if args.cmd == "postgres":

View File

@@ -3,12 +3,10 @@ from __future__ import annotations
import os import os
import sys import sys
from .run import run, docker_volume_exists from .run import docker_volume_exists, run
def restore_volume_files( def restore_volume_files(volume_name: str, backup_files_dir: str) -> int:
volume_name: str, backup_files_dir: str, *, rsync_image: str
) -> int:
if not os.path.isdir(backup_files_dir): if not os.path.isdir(backup_files_dir):
print(f"ERROR: backup files dir not found: {backup_files_dir}", file=sys.stderr) print(f"ERROR: backup files dir not found: {backup_files_dir}", file=sys.stderr)
return 2 return 2
@@ -19,21 +17,21 @@ def restore_volume_files(
else: else:
print(f"Volume {volume_name} already exists.") print(f"Volume {volume_name} already exists.")
# Keep behavior close to the old script: rsync -avv --delete cp = run(
run( ["docker", "volume", "inspect", "--format", "{{ .Mountpoint }}", volume_name],
[ capture=True,
"docker",
"run",
"--rm",
"-v",
f"{volume_name}:/recover/",
"-v",
f"{backup_files_dir}:/backup/",
rsync_image,
"sh",
"-lc",
"rsync -avv --delete /backup/ /recover/",
]
) )
raw = cp.stdout or b""
mountpoint = (raw.decode() if isinstance(raw, bytes) else raw).strip()
if not mountpoint:
print(
f"ERROR: could not resolve mountpoint for volume {volume_name}",
file=sys.stderr,
)
return 2
src = os.path.join(backup_files_dir, "")
dest = os.path.join(mountpoint, "")
run(["rsync", "-avv", "--delete", src, dest])
print("File restore complete.") print("File restore complete.")
return 0 return 0

View File

@@ -7,6 +7,14 @@ import time
import uuid import uuid
from pathlib import Path from pathlib import Path
# SPOT for the database images and their in-container data dirs the e2e
# suite runs against. postgres:alpine tracks latest (18+), which mounts at
# /var/lib/postgresql (not /var/lib/postgresql/data); bump here only.
POSTGRES_IMAGE = "postgres:alpine"
POSTGRES_DATA_DIR = "/var/lib/postgresql"
MARIADB_IMAGE = "mariadb:latest"
MARIADB_DATA_DIR = "/var/lib/mysql"
def run( def run(
cmd: list[str], cmd: list[str],
@@ -172,7 +180,7 @@ def backup_run(
"baudolo", "baudolo",
"--compose-dir", "--compose-dir",
compose_dir, compose_dir,
"--docker-compose-hard-restart-required", "--hard-compose-restart",
"mailu", "mailu",
"--repo-name", "--repo-name",
repo_name, repo_name,

View File

@@ -2,6 +2,8 @@
import unittest import unittest
from .helpers import ( from .helpers import (
POSTGRES_IMAGE,
POSTGRES_DATA_DIR,
backup_path, backup_path,
cleanup_docker, cleanup_docker,
create_minimal_compose_dir, create_minimal_compose_dir,
@@ -50,8 +52,8 @@ class TestE2EDumpOnlyFallbackToFiles(unittest.TestCase):
"-e", "-e",
"POSTGRES_USER=postgres", "POSTGRES_USER=postgres",
"-v", "-v",
f"{cls.pg_volume}:/var/lib/postgresql/data", f"{cls.pg_volume}:{POSTGRES_DATA_DIR}",
"postgres:16", POSTGRES_IMAGE,
] ]
) )
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90) wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)
@@ -65,7 +67,7 @@ class TestE2EDumpOnlyFallbackToFiles(unittest.TestCase):
cls.pg_container, cls.pg_container,
"sh", "sh",
"-lc", "-lc",
f"echo '{cls.marker}' > /var/lib/postgresql/data/marker.txt", f"echo '{cls.marker}' > {POSTGRES_DATA_DIR}/marker.txt",
] ]
) )
@@ -79,7 +81,7 @@ class TestE2EDumpOnlyFallbackToFiles(unittest.TestCase):
"baudolo", "baudolo",
"--compose-dir", "--compose-dir",
cls.compose_dir, cls.compose_dir,
"--docker-compose-hard-restart-required", "--hard-compose-restart",
"mailu", "mailu",
"--repo-name", "--repo-name",
cls.repo_name, cls.repo_name,
@@ -116,8 +118,6 @@ class TestE2EDumpOnlyFallbackToFiles(unittest.TestCase):
cls.repo_name, cls.repo_name,
"--source-volume", "--source-volume",
cls.pg_volume, cls.pg_volume,
"--rsync-image",
"ghcr.io/kevinveenbirkenbach/alpine-rsync",
] ]
) )

View File

@@ -1,6 +1,8 @@
import unittest import unittest
from .helpers import ( from .helpers import (
POSTGRES_IMAGE,
POSTGRES_DATA_DIR,
backup_path, backup_path,
cleanup_docker, cleanup_docker,
create_minimal_compose_dir, create_minimal_compose_dir,
@@ -70,8 +72,8 @@ class TestE2EDumpOnlySqlMixedRun(unittest.TestCase):
"-e", "-e",
f"POSTGRES_PASSWORD={cls.pg_password}", f"POSTGRES_PASSWORD={cls.pg_password}",
"-v", "-v",
f"{cls.db_volume}:/var/lib/postgresql/data", f"{cls.db_volume}:{POSTGRES_DATA_DIR}",
"postgres:16-alpine", POSTGRES_IMAGE,
] ]
) )
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90) wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)

View File

@@ -94,8 +94,6 @@ class TestE2EFilesFull(unittest.TestCase):
self.repo_name, self.repo_name,
"--source-volume", "--source-volume",
self.volume_src, self.volume_src,
"--rsync-image",
"ghcr.io/kevinveenbirkenbach/alpine-rsync",
] ]
) )

View File

@@ -76,7 +76,7 @@ class TestE2EImagesNoBackupRequiredEarlySkip(unittest.TestCase):
"baudolo", "baudolo",
"--compose-dir", "--compose-dir",
cls.compose_dir, cls.compose_dir,
"--docker-compose-hard-restart-required", "--hard-compose-restart",
"mailu", "mailu",
"--repo-name", "--repo-name",
cls.repo_name, cls.repo_name,

View File

@@ -30,6 +30,8 @@ import pandas
from baudolo.backup import db as db_mod from baudolo.backup import db as db_mod
from .helpers import ( from .helpers import (
MARIADB_IMAGE,
MARIADB_DATA_DIR,
cleanup_docker, cleanup_docker,
require_docker, require_docker,
run, run,
@@ -69,8 +71,8 @@ class TestE2EMariaDBAnonymousPreemption(unittest.TestCase):
"-e", "-e",
f"MARIADB_ROOT_PASSWORD={cls.root_password}", f"MARIADB_ROOT_PASSWORD={cls.root_password}",
"-v", "-v",
f"{cls.db_volume}:/var/lib/mysql", f"{cls.db_volume}:{MARIADB_DATA_DIR}",
"mariadb:12.2", MARIADB_IMAGE,
] ]
) )

View File

@@ -2,6 +2,8 @@
import unittest import unittest
from .helpers import ( from .helpers import (
MARIADB_IMAGE,
MARIADB_DATA_DIR,
backup_run, backup_run,
backup_path, backup_path,
cleanup_docker, cleanup_docker,
@@ -56,8 +58,8 @@ class TestE2EMariaDBFull(unittest.TestCase):
"-e", "-e",
f"MARIADB_PASSWORD={cls.db_password}", f"MARIADB_PASSWORD={cls.db_password}",
"-v", "-v",
f"{cls.db_volume}:/var/lib/mysql", f"{cls.db_volume}:{MARIADB_DATA_DIR}",
"mariadb:11", MARIADB_IMAGE,
] ]
) )

View File

@@ -2,6 +2,8 @@
import unittest import unittest
from .helpers import ( from .helpers import (
MARIADB_IMAGE,
MARIADB_DATA_DIR,
backup_run, backup_run,
backup_path, backup_path,
cleanup_docker, cleanup_docker,
@@ -55,8 +57,8 @@ class TestE2EMariaDBNoCopy(unittest.TestCase):
"-e", "-e",
f"MARIADB_PASSWORD={cls.db_password}", f"MARIADB_PASSWORD={cls.db_password}",
"-v", "-v",
f"{cls.db_volume}:/var/lib/mysql", f"{cls.db_volume}:{MARIADB_DATA_DIR}",
"mariadb:11", MARIADB_IMAGE,
] ]
) )

View File

@@ -2,6 +2,8 @@
import unittest import unittest
from .helpers import ( from .helpers import (
POSTGRES_IMAGE,
POSTGRES_DATA_DIR,
backup_run, backup_run,
backup_path, backup_path,
cleanup_docker, cleanup_docker,
@@ -47,8 +49,8 @@ class TestE2EPostgresFull(unittest.TestCase):
"-e", "-e",
"POSTGRES_USER=postgres", "POSTGRES_USER=postgres",
"-v", "-v",
f"{cls.pg_volume}:/var/lib/postgresql/data", f"{cls.pg_volume}:{POSTGRES_DATA_DIR}",
"postgres:16", POSTGRES_IMAGE,
] ]
) )
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90) wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)

View File

@@ -2,6 +2,8 @@
import unittest import unittest
from .helpers import ( from .helpers import (
POSTGRES_IMAGE,
POSTGRES_DATA_DIR,
backup_run, backup_run,
backup_path, backup_path,
cleanup_docker, cleanup_docker,
@@ -46,8 +48,8 @@ class TestE2EPostgresNoCopy(unittest.TestCase):
"-e", "-e",
"POSTGRES_USER=postgres", "POSTGRES_USER=postgres",
"-v", "-v",
f"{cls.pg_volume}:/var/lib/postgresql/data", f"{cls.pg_volume}:{POSTGRES_DATA_DIR}",
"postgres:16", POSTGRES_IMAGE,
] ]
) )
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90) wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)

View File

@@ -1,6 +1,8 @@
import unittest import unittest
from .helpers import ( from .helpers import (
POSTGRES_IMAGE,
POSTGRES_DATA_DIR,
backup_path, backup_path,
cleanup_docker, cleanup_docker,
create_minimal_compose_dir, create_minimal_compose_dir,
@@ -66,8 +68,8 @@ class TestE2ESeedStarAndDbEntriesBackupPostgres(unittest.TestCase):
"-e", "-e",
f"POSTGRES_PASSWORD={cls.pg_password}", f"POSTGRES_PASSWORD={cls.pg_password}",
"-v", "-v",
f"{cls.db_volume}:/var/lib/postgresql/data", f"{cls.db_volume}:{POSTGRES_DATA_DIR}",
"postgres:16-alpine", POSTGRES_IMAGE,
] ]
) )
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90) wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)

View File

@@ -251,5 +251,61 @@ class TestCompose(unittest.TestCase):
) )
class HardRestartArgTests(unittest.TestCase):
"""The hard-restart list defaults to empty (no compose down/up); callers
opt in per dir, e.g. compose hosts pass 'mailu' while swarm hosts, where
the dir is a stack whose overlay network collides with compose up, pass
nothing."""
def _parse(self, extra: List[str]):
import sys
from baudolo.backup import cli
argv = [
"baudolo",
"--compose-dir",
"/tmp",
"--backups-dir",
"/tmp/backup",
"--database-containers",
"postgres",
"--images-no-stop-required",
"redis",
*extra,
]
with patch.object(sys, "argv", argv):
return cli.parse_args()
def test_default_is_empty(self) -> None:
args = self._parse([])
self.assertEqual(args.hard_compose_restart, [])
def test_empty_flag_stays_empty(self) -> None:
args = self._parse(["--hard-compose-restart"])
self.assertEqual(args.hard_compose_restart, [])
def test_explicit_names_preserved(self) -> None:
args = self._parse(["--hard-compose-restart", "mailu", "foo"])
self.assertEqual(args.hard_compose_restart, ["mailu", "foo"])
def test_backups_dir_is_required(self) -> None:
import sys
from baudolo.backup import cli
argv = [
"baudolo",
"--compose-dir",
"/tmp",
"--database-containers",
"postgres",
"--images-no-stop-required",
"redis",
]
with patch.object(sys, "argv", argv), self.assertRaises(SystemExit):
cli.parse_args()
if __name__ == "__main__": if __name__ == "__main__":
unittest.main(verbosity=2) unittest.main(verbosity=2)