Compare commits

...

12 Commits

Author SHA1 Message Date
30fd68bdcf Release version 3.4.2 2026-08-15 15:28:44 +02:00
36b2336742 fix(backup): match the engine on the image name
has_image tested the pattern against the raw .Config.Image, so anything in the reference could decide which dump tool runs -- including the registry host and the tag. A swarm node that hosts the local registry prefixes every pull with its own name, and that node is named after the app under test, so a Postgres container reads as svc-db-mariadb-swarm-mgr-01:5000/postgres_custom:17-3.5. dumps.py tries mariadb before postgres, matched on the hostname, and dumped Postgres with mariadb-dump: exit 127, the image does not ship it. The BackupException took the backup unit down with it.

image_name strips digest, tag and registry host and matches on the repository path, so the decision rests on the image alone. Same intent as the exact --images-* matching from f9776ac, applied to the one place that commit did not reach. Tags stop deciding too: xwiki_custom:lts-postgres-tomcat no longer reads as Postgres.

The e2e reproduces the shape without a registry -- a docker tag is enough for .Config.Image to carry the reference verbatim -- and asserts a real pg_dump lands. Under the old code mariadb-dump aborts and no dump file exists.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-15 15:25:38 +02:00
756e236d10 Release version 3.4.1 2026-08-05 13:18:57 +02:00
1dfeb17ab4 chore(claude): require confirmation before a commit
infinito-nexus-core gates `git commit` behind an explicit per-invocation
confirmation. This repository did not, so an agent working here could commit
without being asked. Mirror the rule.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 13:00:05 +02:00
cd21f1fa67 fix(backup): keep no twin of what the cold pass replaces
Each volume is copied twice into the same destination: once hot with the
container running, once cold after it is stopped (backup/app.py:127-131).
rsync ran with -b, so --delete did not remove a file the source had dropped
between the passes - it renamed it. Stopping a container is exactly what
makes the source drop files: a graceful shutdown flushes, and the format
rolls its commit point.

For an opaque payload the twin is stale bytes nobody reads. For a format
that enumerates its own directory it is corruption. Lucene resolves the
current commit by parsing every file starting with segments as a radix-36
generation, so a restored segments_3~ raises NumberFormatException, the
shard store cannot be read, and the primary is left NO_VALID_SHARD_COPY.
With .security-7 unallocatable the reserved elastic user has no password
hash, every probe gets HTTP 401, and the container never turns healthy.

Seen in infinito-nexus-core CI run 30963648828: the restore drill waited
1200s on elasticsearch while all 29 other containers came back healthy;
the generation held segments_3~ next to segments_4 in all three index
directories. A run two days earlier passed the same drill because that
stack was idle and nothing rolled between the passes - which is why this
reads as flaky rather than broken.

Reproduced with real rsync in all three shapes: two passes into the same
destination with -b leave segments_3~ beside segments_4, without -b only
segments_4 survives, and a single pass keeps segments_3. Measured on the
same fixtures, dropping -b leaves the predecessor generation byte-identical
and keeps the --link-dest hardlinks intact, so the incremental scheme is
unaffected; generations get smaller, never larger.

--link-dest already provides the cheap incrementals. -b contributed nothing
on top of it but the twins, and no caller reads them: the restore path is an
unfiltered rsync -avv --delete into the live volume (restore/files.py:35).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 12:43:08 +02:00
8a93a61ca9 Release version 3.4.0 2026-08-02 12:07:34 +02:00
988d92534c fix(backup): keep kernel objects out of a generation
-a implies -D, so a generation was written with --devices --specials and rsync recreated every unix socket and fifo it found in a volume. On the swarm manager that generation lives on an nfs-ganesha export, and ganesha accepts the socket on write but cannot serve it back: the remote pull's sender then fails with readdir/readlink_stat 'Invalid argument (22)' and exits 23, deterministically, for all twelve retries - 58 minutes per run.

Postfix's queue directory is the case that surfaced it, where public/ and private/ hold roughly fifty AF_UNIX sockets and nothing else. The class is wider: a discourse /shared with its in-container postgres socket, a checkmk OMD site with tmp/run/nagios.cmd, a container whose /tmp is a persisted volume. --no-D is type-based and closes all of them without anyone having to know which image binds a socket where.

Nothing restorable is lost. A socket inode is meaningless after a restore; postfix's master, checkmk's omd start and discourse's supervisor recreate theirs. The whole postfix queue survives - incoming, active, deferred, hold, maildrop - so accepted-but-undelivered mail stays in the backup, which excluding the volume outright would have dropped. Device nodes go too, and the only volume that could hold them is a nested docker data root, already carrying backup: false.

This is the writer side, whose source is a local docker volume. On the reader the same flag provably does nothing: rsync still stats the entry before -D decides, and getdents64 on the containing directory is outside its reach entirely.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 12:06:42 +02:00
934e693810 Release version 3.3.0 2026-08-02 09:11:54 +02:00
2e0e67ca87 Autolint 2026-08-02 09:10:55 +02:00
95c34d4db0 feat(backup): exclude a volume by name, not only by image
volume_is_fully_ignored can only skip a volume when every container using it is ignored, so a container holding a derived tree next to state that must be kept cannot express the exclusion at all. The matrix docker-in-docker runner is exactly that: matrix_mdad_docker, matrix_mdad_matrix and matrix_mdad_state all hang off one container, and the derived one is an inner overlay2 store that no rsync in the chain can restore faithfully (none carries -X, so trusted.overlay.* is stripped in both directions).

--volumes-no-backup-required names volumes directly. The check runs before containers_using_volume, so an excluded volume costs no docker call and the decision no longer depends on which containers happen to exist at backup time.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 09:10:20 +02:00
c2f1cb8e8c Release version 3.2.2 2026-07-31 19:20:49 +02:00
eeaa838d02 fix(backup): carve the btrfs snapshot inside its subject, not beside it
The kernel refuses a snapshot whose destination sits on another
filesystem. Placing it at <parent>/.baudolo-<tag> hit that on the most
sensible layout there is: a dedicated disk mounted straight onto
/var/lib/docker, where the parent directory belongs to a different
filesystem and every run failed with EXDEV.

Placing it at <subject>/.baudolo-<tag> makes source and destination the
same filesystem by construction, so the failure cannot occur on any
layout. It also aligns the two backends: the zfs path already resolves
its snapshot inside the subject, at <subject>/.zfs/snapshot/<tag>.

btrfs does not include nested subvolumes in a snapshot, so a leftover
from an interrupted run appears in the next snapshot as an empty
directory rather than recursing, and the copy only ever reads the volume
tree below it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-31 19:19:42 +02:00
22 changed files with 578 additions and 35 deletions

View File

@@ -1,6 +1,7 @@
{
"permissions": {
"ask": [
"Bash(git commit*)",
"Edit(CHANGELOG.md)",
"Write(CHANGELOG.md)",
"Edit(pyproject.toml)",

View File

@@ -1,5 +1,82 @@
# Changelog
## [3.4.2] - 2026-08-15
- Backup: *has_image* matched the raw *.Config.Image*, so the registry host and
the tag decided the dump tool. On a swarm node named after the app under test,
*svc-db-mariadb-swarm-mgr-01:5000/postgres_custom* read as MariaDB and
*mariadb-dump* ran inside a Postgres container: exit 127, and the
*BackupException* took the backup unit with it.
- Backup: *image_name* strips digest, tag and registry host, so the engine rests
on the repository path alone — the exact-matching intent of 3.0.0 applied to
the one place that change did not reach.
- Tests: both false-positive directions on *has_image*, plus an e2e that
reproduces the shape with a *docker tag* and asserts a real *pg_dump* lands.
## [3.4.1] - 2026-08-05
- Backup: each volume is copied twice into the same destination — once hot,
once cold after the container is stopped — and rsync ran with *-b*, so
*--delete* renamed rather than removed a file the source had dropped between
the passes. Stopping a container is what makes the source drop files: a
graceful shutdown flushes and the format rolls its commit point. The
superseded file survived as *name~* beside the real one and was restored into
live data.
- Backup: for an opaque payload that is stale bytes nobody reads; for a format
that enumerates its own directory it is corruption. Lucene resolves the
current commit by parsing every file starting with *segments* as a radix-36
generation, so a restored *segments_3~* leaves the shard store unreadable and
the primary at *NO_VALID_SHARD_COPY*. With *.security-7* unallocatable the
reserved *elastic* user has no password hash, every probe answers 401 and the
container never turns healthy.
- Backup: *--link-dest* already provides the incrementals and nothing reads the
twins — the restore path is an unfiltered *rsync -avv --delete* into the live
volume. Dropping *-b* leaves the predecessor generation byte-identical, keeps
the hardlinks intact and makes generations smaller, never larger.
- Tests: the absence of *--backup* is asserted on the rsync invocation.
## [3.4.0] - 2026-08-02
- Backup: *-a* implies *-D*, so a generation was written with
*--devices --specials* and rsync recreated every unix socket and fifo found in
a volume. Where the backup root is an nfs-ganesha export, ganesha accepts the
socket on write but cannot serve it back, and the remote pull's sender then
fails with *readdir* / *readlink_stat* "Invalid argument (22)" and exits 23 —
deterministically, for every retry. *--no-D* keeps them out of the generation.
- Backup: nothing restorable is lost. Sockets and fifos are recreated by the
daemons that own them, and the postfix queue itself — *incoming*, *active*,
*deferred*, *hold*, *maildrop* — is unaffected, so accepted-but-undelivered
mail stays in the backup. Device nodes go too; the only volume that could hold
them is a nested docker data root, which does not belong in a backup anyway.
- Tests: the flag is asserted on the rsync invocation.
## [3.3.0] - 2026-08-02
- Backup: *--volumes-no-backup-required* excludes a volume by name.
*--images-no-backup-required* resolves through *volume_is_fully_ignored*,
which skips a volume only when every container using it is ignored — a
container holding a derived tree beside state that must be kept cannot
express the exclusion at all. A docker-in-docker data root is exactly that
shape, and excluding by image would drop all three of its volumes.
- Backup: the name check runs before *containers_using_volume*, so an excluded
volume costs no docker inspection and the decision does not depend on which
containers exist when the run starts.
- Tests: two volumes off one container, asserting the sibling survives — the
property the image lever cannot provide — as unit and end-to-end.
## [3.2.2] - 2026-07-31
- Backup: the btrfs snapshot is carved inside its subject, as
*<data root>/.baudolo-<tag>*, not beside it. The kernel refuses a snapshot
whose destination is on another filesystem, which is exactly what the parent
directory is when the data root is a mountpoint of its own — a dedicated disk
mounted onto */var/lib/docker* failed every run with EXDEV. Placing it inside
makes source and destination the same filesystem by construction, and aligns
btrfs with the zfs path, which already resolves its snapshot inside the
subject at *<subject>/.zfs/snapshot/<tag>*. A leftover from an interrupted run
appears in the next snapshot as an empty directory rather than recursing,
since btrfs does not include nested subvolumes.
## [3.2.1] - 2026-07-31
- Backup: the snapshot resolver keeps the trailing separator *get_storage_path*

View File

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "backup-docker-to-local"
version = "3.2.1"
version = "3.4.2"
description = "Backup Docker volumes to local with rsync and optional DB dumps."
readme = "README.md"
requires-python = ">=3.9"

View File

@@ -54,6 +54,14 @@ def main() -> int:
for volume_name in docker_volume_names():
print(f"Start backup routine for volume: {volume_name}", flush=True)
if volume_name in args.volumes_no_backup_required:
print(
f"Skipping volume '{volume_name}' entirely (declared no-backup).",
flush=True,
)
continue
containers = containers_using_volume(volume_name)
if volume_is_fully_ignored(containers, args.images_no_backup_required):

View File

@@ -68,6 +68,13 @@ def parse_args() -> argparse.Namespace:
help="Exact image references (repo:tag, incl. any registry prefix) for which no backup should be performed",
)
p.add_argument(
"--volumes-no-backup-required",
nargs="+",
default=[],
help="Exact volume names that are never backed up, whatever containers use them. For derived trees a restore cannot reproduce, above all a nested docker data root",
)
p.add_argument(
"--everything",
action="store_true",
@@ -93,7 +100,9 @@ def parse_args() -> argparse.Namespace:
if bool(args.snapshot) != bool(args.snapshot_subject):
p.error("--snapshot and --snapshot-subject must be given together")
if args.snapshot and args.shutdown:
p.error("--shutdown is meaningless with --snapshot: containers are never stopped")
p.error(
"--shutdown is meaningless with --snapshot: containers are never stopped"
)
if args.snapshot and args.hard_restart_projects:
p.error(
"--hard-restart-projects is meaningless with --snapshot: the flag exists "

View File

@@ -9,9 +9,29 @@ def get_image_info(container: str) -> str:
)[0]
def image_name(container: str) -> str:
"""The image's repository path, without registry host, tag or digest.
A swarm node that hosts the local registry puts its own hostname in front
of every pull, so the raw reference of a Postgres container can read
`svc-db-mariadb-swarm-mgr-01:5000/postgres_custom:17-3.5`. Matching the
whole reference finds "mariadb" there and dumps the database with
mariadb-dump, which the Postgres image does not ship (exit 127). Tags bite
the same way: `xwiki_custom:lts-postgres-tomcat`.
"""
reference = get_image_info(container).strip().split("@", 1)[0]
head, _, tail = reference.rpartition("/")
tail = tail.split(":", 1)[0]
if head:
registry = head.split("/", 1)[0]
if "." in registry or ":" in registry or registry == "localhost":
head = head.partition("/")[2]
return f"{head}/{tail}" if head else tail
def has_image(container: str, pattern: str) -> bool:
"""Return True if container's image contains the pattern."""
return pattern in get_image_info(container)
"""Return True if the container's image name contains the pattern."""
return pattern in image_name(container)
def docker_volume_names() -> list[str]:

View File

@@ -41,7 +41,10 @@ def _resolver(subject: str, root: str) -> Callable[[str], str]:
def _btrfs(subject: str, name: str, run: Callable[[str], list[str]]) -> tuple[str, str]:
target = os.path.join(os.path.dirname(os.path.abspath(subject)), f".{name}")
# The snapshot goes inside the subject, never beside it: the kernel rejects
# a snapshot whose destination is on another filesystem, which is exactly
# what the parent directory is when the subject is a mountpoint of its own.
target = os.path.join(os.path.abspath(subject), f".{name}")
run(f"btrfs subvolume snapshot -r {subject} {target}")
return target, f"btrfs subvolume delete {target}"

View File

@@ -49,7 +49,10 @@ def backup_volume(
link_dest = f"--link-dest='{last}'" if last else ""
verify = "--checksum " if authoritative else ""
cmd = f"rsync -abP --delete --delete-excluded {verify}{link_dest} {source} {dest}"
cmd = (
f"rsync -aP --no-D --delete --delete-excluded "
f"{verify}{link_dest} {source} {dest}"
)
try:
execute_shell_command(cmd)

View File

@@ -1,4 +1,5 @@
"""Fixtures and paths the e2e suite builds its scenarios from."""
from __future__ import annotations
import shutil

View File

@@ -1,4 +1,5 @@
"""Process, docker and readiness helpers for the e2e suite."""
from __future__ import annotations
import subprocess

View File

@@ -25,7 +25,9 @@ GENERATION = f"{VERSIONS}/20260731"
def shell(command: str) -> list[str]:
proc = subprocess.run(command, shell=True, capture_output=True, text=True)
if proc.returncode != 0:
raise SnapshotError(f"{command} exited {proc.returncode}: {proc.stderr.strip()}")
raise SnapshotError(
f"{command} exited {proc.returncode}: {proc.stderr.strip()}"
)
return proc.stdout.splitlines()

View File

@@ -22,7 +22,9 @@ EXPECT = sys.argv[3]
def shell(command: str) -> list[str]:
proc = subprocess.run(command, shell=True, capture_output=True, text=True)
if proc.returncode != 0:
raise SnapshotError(f"{command} exited {proc.returncode}: {proc.stderr.strip()}")
raise SnapshotError(
f"{command} exited {proc.returncode}: {proc.stderr.strip()}"
)
return proc.stdout.splitlines()
@@ -55,5 +57,8 @@ with volume_snapshot(KIND, SUBJECT, "e2e", run=shell) as resolve:
root = Path(resolve(SUBJECT))
check("the snapshot is removed afterwards", not root.exists() or not (root / "volumes").exists())
check(
"the snapshot is removed afterwards",
not root.exists() or not (root / "volumes").exists(),
)
print("ALL OK", flush=True)

View File

@@ -0,0 +1,105 @@
import unittest
from .helpers import (
POSTGRES_IMAGE,
POSTGRES_DATA_DIR,
backup_run,
backup_path,
cleanup_docker,
create_minimal_compose_dir,
ensure_empty_dir,
latest_version_dir,
require_docker,
unique,
write_databases_csv,
run,
wait_for_postgres,
)
REGISTRY_HOST = "svc-db-mariadb-swarm-mgr-01:5000"
class TestE2EEngineDetectionRegistryPrefix(unittest.TestCase):
@classmethod
def setUpClass(cls) -> None:
require_docker()
cls.prefix = unique("baudolo-e2e-registry-prefix")
cls.backups_dir = f"/tmp/{cls.prefix}/Backups"
ensure_empty_dir(cls.backups_dir)
cls.compose_dir = create_minimal_compose_dir(f"/tmp/{cls.prefix}")
cls.repo_name = cls.prefix
cls.image = f"{REGISTRY_HOST}/postgres_custom:17-3.5"
cls.pg_container = f"{cls.prefix}-pg"
cls.pg_volume = f"{cls.prefix}-pg-vol"
cls.containers = [cls.pg_container]
cls.volumes = [cls.pg_volume]
run(["docker", "pull", POSTGRES_IMAGE])
run(["docker", "tag", POSTGRES_IMAGE, cls.image])
run(["docker", "volume", "create", cls.pg_volume])
run(
[
"docker",
"run",
"-d",
"--name",
cls.pg_container,
"-e",
"POSTGRES_PASSWORD=pgpw",
"-e",
"POSTGRES_DB=appdb",
"-e",
"POSTGRES_USER=postgres",
"-v",
f"{cls.pg_volume}:{POSTGRES_DATA_DIR}",
cls.image,
]
)
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)
run(
[
"docker",
"exec",
cls.pg_container,
"sh",
"-lc",
"psql -U postgres -d appdb -c \"CREATE TABLE t (id int primary key, v text); INSERT INTO t VALUES (1,'ok');\"",
]
)
cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv"
write_databases_csv(
cls.databases_csv, [(cls.pg_container, "appdb", "postgres", "pgpw")]
)
backup_run(
backups_dir=cls.backups_dir,
repo_name=cls.repo_name,
compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv,
database_containers=[cls.pg_container],
images_no_stop_required=[cls.image],
)
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)
@classmethod
def tearDownClass(cls) -> None:
cleanup_docker(containers=cls.containers, volumes=cls.volumes)
run(["docker", "rmi", cls.image], check=False)
def test_the_registry_host_does_not_pick_the_engine(self) -> None:
p = (
backup_path(self.backups_dir, self.repo_name, self.version, self.pg_volume)
/ "sql"
/ "appdb.backup.sql"
)
self.assertTrue(p.is_file(), f"Expected a pg_dump at: {p}")
self.assertIn("Dumped by pg_dump", p.read_text(encoding="utf-8"))
if __name__ == "__main__":
unittest.main()

View File

@@ -29,7 +29,7 @@ LOOP_FS = {
"ext4": "mkdir -p /subject/docker",
}
# A container carries no /lib/modules, so modprobe fails even on a loaded module.
ZFS_READY = '{ [ -c /dev/zfs ] || modprobe zfs 2>/dev/null; }; [ -c /dev/zfs ]'
ZFS_READY = "{ [ -c /dev/zfs ] || modprobe zfs 2>/dev/null; }; [ -c /dev/zfs ]"
def mount_script(fstype: str) -> str:
@@ -51,7 +51,13 @@ def zfs_usable() -> bool:
"""Whether this host's kernel can serve zfs to a privileged container."""
proc = run(
[
"docker", "run", "--rm", "--privileged", IMAGE, "sh", "-lc",
"docker",
"run",
"--rm",
"--privileged",
IMAGE,
"sh",
"-lc",
f"apk add -q zfs >/dev/null 2>&1 && {ZFS_READY}",
],
capture=True,
@@ -87,11 +93,20 @@ def drive(fstype: str, kind: str, expect: str) -> str:
try:
proc = run(
[
"docker", "run", "--rm", "--privileged",
"--name", staged.name,
"-v", f"{staged / 'src'}:/src:ro",
"-v", f"{staged / 'driver.py'}:/driver.py:ro",
IMAGE, "sh", "-lc", script,
"docker",
"run",
"--rm",
"--privileged",
"--name",
staged.name,
"-v",
f"{staged / 'src'}:/src:ro",
"-v",
f"{staged / 'driver.py'}:/driver.py:ro",
IMAGE,
"sh",
"-lc",
script,
],
capture=True,
check=False,
@@ -99,7 +114,9 @@ def drive(fstype: str, kind: str, expect: str) -> str:
finally:
shutil.rmtree(staged, ignore_errors=True)
if proc.returncode != 0:
raise AssertionError(f"{fstype}/{kind} driver failed:\n{proc.stdout}\n{proc.stderr}")
raise AssertionError(
f"{fstype}/{kind} driver failed:\n{proc.stdout}\n{proc.stderr}"
)
return proc.stdout
@@ -125,7 +142,9 @@ class TestE2ESnapshot(unittest.TestCase):
"E2E_REQUIRE_FILESYSTEMS demands zfs, but this kernel provides no "
"zfs module; load it before running the suite"
)
self.skipTest("this kernel provides no zfs module, so no pool can be created")
self.skipTest(
"this kernel provides no zfs module, so no pool can be created"
)
self.assert_freezes("zfs")
def test_ext4_has_no_snapshot_and_says_so(self) -> None:

View File

@@ -68,11 +68,20 @@ class TestE2ESnapshotDatabase(unittest.TestCase):
try:
proc = run(
[
"docker", "run", "--rm", "--privileged",
"--name", staged.name,
"-v", f"{staged / 'src'}:/src:ro",
"-v", f"{staged / 'driver.py'}:/driver.py:ro",
IMAGE, "sh", "-lc", SCRIPT,
"docker",
"run",
"--rm",
"--privileged",
"--name",
staged.name,
"-v",
f"{staged / 'src'}:/src:ro",
"-v",
f"{staged / 'driver.py'}:/driver.py:ro",
IMAGE,
"sh",
"-lc",
SCRIPT,
],
capture=True,
check=False,

View File

@@ -0,0 +1,125 @@
import unittest
from .helpers import (
backup_path,
cleanup_docker,
create_minimal_compose_dir,
ensure_empty_dir,
latest_version_dir,
require_docker,
run,
unique,
write_databases_csv,
)
class TestE2EVolumesNoBackupRequiredEarlySkip(unittest.TestCase):
"""Both volumes hang off the same container, so an image-level exclusion
could only drop both. Only the named one may disappear."""
@classmethod
def setUpClass(cls) -> None:
require_docker()
cls.prefix = unique("baudolo-e2e-early-skip-no-backup-volume")
cls.backups_dir = f"/tmp/{cls.prefix}/Backups"
ensure_empty_dir(cls.backups_dir)
cls.compose_dir = create_minimal_compose_dir(f"/tmp/{cls.prefix}")
cls.repo_name = cls.prefix
cls.container = f"{cls.prefix}-app"
cls.excluded_volume = f"{cls.prefix}-derived-vol"
cls.kept_volume = f"{cls.prefix}-state-vol"
cls.containers = [cls.container]
cls.volumes = [cls.excluded_volume, cls.kept_volume]
run(["docker", "volume", "create", cls.excluded_volume])
run(["docker", "volume", "create", cls.kept_volume])
run(
[
"docker",
"run",
"--rm",
"-v",
f"{cls.excluded_volume}:/derived",
"-v",
f"{cls.kept_volume}:/state",
"alpine:3.20",
"sh",
"-lc",
"echo derived > /derived/derived.txt && echo state > /state/state.txt",
]
)
run(
[
"docker",
"run",
"-d",
"--name",
cls.container,
"-v",
f"{cls.excluded_volume}:/derived",
"-v",
f"{cls.kept_volume}:/state",
"alpine:3.20",
"sleep",
"600",
]
)
cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv"
write_databases_csv(cls.databases_csv, [])
cmd = [
"baudolo",
"--compose-dir",
cls.compose_dir,
"--repo-name",
cls.repo_name,
"--databases-csv",
cls.databases_csv,
"--backups-dir",
cls.backups_dir,
"--images-no-stop-required",
"alpine:3.20",
"--volumes-no-backup-required",
cls.excluded_volume,
]
cp = run(cmd, capture=True, check=True)
cls.stdout = cp.stdout or ""
cls.stderr = cp.stderr or ""
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)
@classmethod
def tearDownClass(cls) -> None:
cleanup_docker(containers=cls.containers, volumes=cls.volumes)
def test_excluded_volume_has_no_backup_directory_at_all(self) -> None:
p = backup_path(
self.backups_dir,
self.repo_name,
self.version,
self.excluded_volume,
)
self.assertFalse(
p.exists(),
f"Expected NO backup directory for the excluded volume, but found: {p}",
)
def test_sibling_volume_of_the_same_container_is_still_backed_up(self) -> None:
p = (
backup_path(
self.backups_dir,
self.repo_name,
self.version,
self.kept_volume,
)
/ "files"
/ "state.txt"
)
self.assertTrue(p.is_file(), f"Expected backed up file at: {p}")

View File

@@ -12,6 +12,7 @@ from baudolo.backup.snapshot import volume_snapshot
def stubbed_snapshot(kind: str, subject: str, tag: str):
return volume_snapshot(kind, subject, tag, run=lambda command: [])
ARGV = [
"baudolo",
"--compose-dir",
@@ -64,7 +65,7 @@ class TestSnapshotBranch(unittest.TestCase):
def test_it_reads_from_the_snapshot_and_not_from_the_live_tree(self) -> None:
source = drive(present=True)[0]["source"]
self.assertTrue(source.startswith("/var/lib/.baudolo-"), source)
self.assertTrue(source.startswith("/var/lib/docker/.baudolo-"), source)
def test_it_compares_by_content_against_the_previous_generation(self) -> None:
self.assertTrue(drive(present=True)[0]["authoritative"])

View File

@@ -0,0 +1,82 @@
"""Contract of --volumes-no-backup-required: exclusion is per volume name,
independent of which containers use it."""
from __future__ import annotations
import unittest
from unittest import mock
from baudolo.backup import app
ARGV = [
"baudolo",
"--compose-dir",
"/compose",
"--backups-dir",
"/backups",
"--volumes-no-backup-required",
"derived",
]
def drive() -> tuple[list[str], list[str], list[str]]:
backed_up: list[str] = []
created: list[str] = []
inspected: list[str] = []
def record_backup(versions_dir, volume_name, volume_dir, *, authoritative, source):
backed_up.append(volume_name)
with (
mock.patch("sys.argv", ARGV),
mock.patch.object(app, "get_machine_id", return_value="machine"),
mock.patch.object(app, "create_version_directory", return_value="/gen"),
mock.patch.object(
app,
"create_volume_directory",
side_effect=lambda _version_dir, name: created.append(name) or "/gen/vol",
),
mock.patch.object(app, "load_databases_df", return_value=None),
mock.patch.object(
app, "docker_volume_names", return_value=["derived", "state"]
),
mock.patch.object(
app,
"containers_using_volume",
side_effect=lambda name: inspected.append(name) or ["app"],
),
mock.patch.object(app, "volume_is_fully_ignored", return_value=False),
mock.patch.object(app, "backup_dumps_for_volume", return_value=(False, False)),
mock.patch.object(app, "get_storage_path", return_value="/data/"),
mock.patch.object(app, "stamp_directory"),
mock.patch.object(app, "handle_docker_compose_services"),
mock.patch.object(app.os.path, "isdir", return_value=True),
mock.patch.object(app, "backup_volume", side_effect=record_backup),
mock.patch.object(app, "filter_stoppable", return_value=[]),
mock.patch.object(app, "requires_stop", return_value=False),
mock.patch.object(app, "change_containers_status"),
):
app.main()
return backed_up, created, inspected
class TestVolumesNoBackupRequired(unittest.TestCase):
def test_the_named_volume_is_never_backed_up(self) -> None:
backed_up, _created, _inspected = drive()
self.assertNotIn("derived", backed_up)
def test_a_sibling_volume_of_the_same_container_survives(self) -> None:
backed_up, _created, _inspected = drive()
self.assertEqual(backed_up, ["state"])
def test_no_generation_directory_is_created_for_it(self) -> None:
_backed_up, created, _inspected = drive()
self.assertEqual(created, ["state"])
def test_the_skip_precedes_the_container_inspection(self) -> None:
_backed_up, _created, inspected = drive()
self.assertEqual(inspected, ["state"])
if __name__ == "__main__":
unittest.main()

View File

@@ -39,7 +39,9 @@ class TestSnapshotFlags(unittest.TestCase):
parse("--snapshot", "ext4", "--snapshot-subject", "/var/lib/docker")
def test_zfs_is_accepted(self) -> None:
self.assertEqual(parse("--snapshot", "zfs", "--snapshot-subject", "/d").snapshot, "zfs")
self.assertEqual(
parse("--snapshot", "zfs", "--snapshot-subject", "/d").snapshot, "zfs"
)
def test_shutdown_is_rejected_because_nothing_is_stopped(self) -> None:
with self.assertRaises(SystemExit):

View File

@@ -0,0 +1,50 @@
import unittest
from unittest.mock import patch
from baudolo.backup import docker as docker_mod
def _with_image(reference: str):
return patch.object(docker_mod, "execute_shell_command", return_value=[reference])
class TestImageName(unittest.TestCase):
def test_plain_reference(self) -> None:
with _with_image("postgres:16"):
self.assertEqual(docker_mod.image_name("c1"), "postgres")
def test_registry_host_is_dropped(self) -> None:
with _with_image("svc-db-mariadb-swarm-mgr-01:5000/postgres_custom:17-3.5"):
self.assertEqual(docker_mod.image_name("c1"), "postgres_custom")
def test_pull_through_path_is_kept(self) -> None:
with _with_image(
"svc-db-mariadb-swarm-mgr-01:5000/ghcr.io/x/mirror/docker.io/postgres:16"
):
self.assertEqual(
docker_mod.image_name("c1"), "ghcr.io/x/mirror/docker.io/postgres"
)
def test_digest_is_dropped(self) -> None:
with _with_image("registry:5000/postgres@sha256:" + "0" * 64):
self.assertEqual(docker_mod.image_name("c1"), "postgres")
class TestHasImage(unittest.TestCase):
def test_registry_hostname_does_not_decide_the_engine(self) -> None:
with _with_image("svc-db-mariadb-swarm-mgr-01:5000/postgres_custom:17-3.5"):
self.assertFalse(docker_mod.has_image("c1", "mariadb"))
with _with_image("svc-db-mariadb-swarm-mgr-01:5000/postgres_custom:17-3.5"):
self.assertTrue(docker_mod.has_image("c1", "postgres"))
def test_tag_does_not_decide_the_engine(self) -> None:
with _with_image("registry:5000/xwiki_custom:lts-postgres-tomcat"):
self.assertFalse(docker_mod.has_image("c1", "postgres"))
def test_mirrored_mariadb_still_matches(self) -> None:
with _with_image("registry:5000/ghcr.io/x/mirror/docker.io/mariadb:11"):
self.assertTrue(docker_mod.has_image("c1", "mariadb"))
if __name__ == "__main__":
unittest.main()

View File

@@ -22,35 +22,41 @@ class Runner:
class TestBtrfs(unittest.TestCase):
def test_it_creates_a_read_only_snapshot_beside_the_subject(self) -> None:
def test_it_creates_a_read_only_snapshot_inside_the_subject(self) -> None:
run = Runner()
with volume_snapshot("btrfs", "/var/lib/docker", "20260731", run=run):
pass
self.assertEqual(
run.calls[0],
"btrfs subvolume snapshot -r /var/lib/docker /var/lib/.baudolo-20260731",
"btrfs subvolume snapshot -r /var/lib/docker /var/lib/docker/.baudolo-20260731",
)
def test_it_removes_the_snapshot_afterwards(self) -> None:
run = Runner()
with volume_snapshot("btrfs", "/var/lib/docker", "20260731", run=run):
pass
self.assertEqual(run.calls[-1], "btrfs subvolume delete /var/lib/.baudolo-20260731")
self.assertEqual(
run.calls[-1], "btrfs subvolume delete /var/lib/docker/.baudolo-20260731"
)
def test_it_maps_a_volume_path_into_the_snapshot(self) -> None:
run = Runner()
with volume_snapshot("btrfs", "/var/lib/docker", "20260731", run=run) as resolve:
with volume_snapshot(
"btrfs", "/var/lib/docker", "20260731", run=run
) as resolve:
self.assertEqual(
resolve("/var/lib/docker/volumes/postgres_data/_data"),
"/var/lib/.baudolo-20260731/volumes/postgres_data/_data",
"/var/lib/docker/.baudolo-20260731/volumes/postgres_data/_data",
)
def test_it_keeps_the_trailing_slash_rsync_reads_as_contents(self) -> None:
run = Runner()
with volume_snapshot("btrfs", "/var/lib/docker", "20260731", run=run) as resolve:
with volume_snapshot(
"btrfs", "/var/lib/docker", "20260731", run=run
) as resolve:
self.assertEqual(
resolve("/var/lib/docker/volumes/postgres_data/_data/"),
"/var/lib/.baudolo-20260731/volumes/postgres_data/_data/",
"/var/lib/docker/.baudolo-20260731/volumes/postgres_data/_data/",
)
def test_it_removes_the_snapshot_even_when_the_body_raises(self) -> None:
@@ -102,14 +108,20 @@ class TestRejections(unittest.TestCase):
def test_a_path_outside_the_subject_is_rejected(self) -> None:
run = Runner()
with volume_snapshot("btrfs", "/var/lib/docker", "20260731", run=run) as resolve:
with volume_snapshot(
"btrfs", "/var/lib/docker", "20260731", run=run
) as resolve:
with self.assertRaises(SnapshotError):
resolve("/etc/passwd")
def test_the_subject_itself_resolves_to_the_snapshot_root(self) -> None:
run = Runner()
with volume_snapshot("btrfs", "/var/lib/docker", "20260731", run=run) as resolve:
self.assertEqual(resolve("/var/lib/docker"), "/var/lib/.baudolo-20260731")
with volume_snapshot(
"btrfs", "/var/lib/docker", "20260731", run=run
) as resolve:
self.assertEqual(
resolve("/var/lib/docker"), "/var/lib/docker/.baudolo-20260731"
)
class Busy(Runner):

View File

@@ -43,6 +43,14 @@ class TestBackupVolume(unittest.TestCase):
def test_it_always_deletes_what_the_source_no_longer_has(self) -> None:
self.assertIn("--delete", self.copy())
def test_it_carries_no_kernel_objects_into_a_generation(self) -> None:
self.assertIn("--no-D", self.copy())
def test_it_keeps_no_twin_of_what_the_second_pass_replaces(self) -> None:
command = self.copy(authoritative=True)
self.assertIn("rsync -aP ", command)
self.assertNotIn("--backup", command)
def test_it_creates_the_destination(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
dest = Path(tmp) / "gen" / "demo"