Compare commits

...
6 Commits
Author SHA1 Message Date
kevinveenbirkenbach b43177f0e7 Release version 7.1.2 2026-10-18 00:43:44 +02:00
kevinveenbirkenbachandClaude Opus 5.5 0735e5fa7b fix(backup): mount the zfs snapshot itself so a container can read it
The zfs snapshot was read through <subject>/.zfs/snapshot/<name>. The
kernel automounts that directory through a usermode helper that runs in
the host's mount namespace, so a containerised baudolo never sees the
mount and finds no snapshot. The snapshot is now mounted explicitly at
<subject>/.baudolo-<tag>, beside where btrfs carves its own, and removed
in order: umount, rmdir, zfs destroy. A failed mount destroys the
snapshot at once instead of leaking one per run.

The e2e suite could not reach this code at all: zpool create on the /img
file failed with "no such pool or dataset", because the kernel opens
vdevs from a taskq thread that resolves paths in the host's root, where
the container's /img does not exist. The pool now sits on a loop device,
whose /dev/loopN node the kernel creates on the host as well, and it is
destroyed on exit once this script created it, since a pool is
kernel-wide.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-18 00:22:53 +02:00
kevinveenbirkenbach 389b418f42 Release version 7.1.1 2026-10-17 23:31:28 +02:00
kevinveenbirkenbachandClaude Opus 5.5 948b8347f4 fix(test): wait for MariaDB with the root password, not the init window
The entrypoint installs root@localhost without unix_socket auth and sets
its password during init, so the passwordless socket probe only passed
when it landed in the window before SET PASSWORD. Missing that window
timed the e2e MariaDB suites out at random.

The probe now authenticates with the root password it already received
and requires @@skip_networking = 0, so the entrypoint's temporary server
no longer counts as ready. The anonymous-preemption bootstrap logs in
with the root password as well.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-17 23:30:00 +02:00
kevinveenbirkenbach f824fabea1 Release version 7.1.0 2026-10-17 22:35:15 +02:00
kevinveenbirkenbachandClaude Opus 5 07d819cb41 fix(backup): carry extended attributes, ACLs and hardlinks into a generation
rsync -a expands to -rlptgoD, so the backup stored neither extended
attributes nor ACLs nor hardlinks while the restore side asks for
-aHAX. A generation therefore never held what a restore wants back, and
rsync faithfully mirrored a snapshot that has none.

Reproduced end to end against a source carrying user.oc.* attributes and
a hardlink: with the old argv, getfattr -d -R over the generation returns
nothing and the hardlink splits into two inodes; with -aHAX both sides
come back identical to the source. That is the shape in which an
OpenCloud volume survived a disaster-recovery drill with its files intact
and its storage metadata gone, so its posix driver skipped every space it
could no longer read.

restore/files.py had the same gap on this repository's own restore path.

Two consequences worth knowing. The first backup after this re-copies
every attribute-carrying volume once, because rsync declines to hardlink
against a generation whose xattrs do not match. And a backup destination
on a filesystem without xattr support now fails the run instead of
producing a generation that cannot be restored.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-17 22:28:49 +02:00
10 changed files with 151 additions and 27 deletions

No files matched your search

+39
View File
@@ -1,5 +1,44 @@
# Changelog # Changelog
## [7.1.2] - 2026-10-18
Fixed:
- Backup: *--snapshot zfs* found no snapshot inside a container; now mounts it
- Tests: e2e zfs pool now sits on a loop device; a file vdev never worked
Upgrade notes:
- Backup: zfs snapshots mount at *.baudolo-TAG* in the subject, not *.zfs*
- Image: first ghcr image since 3.1.2; the 4.0.0 to 7.1.1 notes apply
## [7.1.1] - 2026-10-17
Fixed:
- Tests: e2e MariaDB probe ran root passwordless and timed out at random
- Tests: probe now logs in with the root password, skips the temp server
- Tests: anonymous-preemption bootstrap now uses the root password too
Upgrade notes:
- 7.1.0 never reached stable: its CI failed, no image was published
- 7.1.1 is the first published -aHAX build; 7.1.0 upgrade notes apply
## [7.1.0] - 2026-10-17
Fixed:
- Backup: rsync -a stored no xattrs, ACLs or hardlinks; now -aHAX
- Restore: restore_volume_files had the same gap; now -aHAX too
- An OpenCloud volume lost its storage metadata through a DR drill
Upgrade notes:
- First backup re-copies every attribute-carrying volume once in full
- A destination without xattr support now fails the backup run outright
- exFAT, xattr-less NFS and object-storage FUSE mounts must move first
## [7.0.2] - 2026-09-23 ## [7.0.2] - 2026-09-23
* Backup: failed live pre-copy no longer aborts when a stopped copy follows * Backup: failed live pre-copy no longer aborts when a stopped copy follows
+1 -1
View File
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "backup-docker-to-local" name = "backup-docker-to-local"
version = "7.0.2" version = "7.1.2"
description = "Backup Docker volumes to local with rsync and optional DB dumps." description = "Backup Docker volumes to local with rsync and optional DB dumps."
readme = "README.md" readme = "README.md"
requires-python = ">=3.9" requires-python = ">=3.9"
+30 -11
View File
@@ -58,27 +58,50 @@ def _resolver(subject: str, root: str) -> Callable[[str], str]:
return resolve return resolve
def _remove(
root: str, commands: list[list[str]], run: Callable[[list[str]], list[str]]
) -> None:
try:
for command in commands:
run(command)
except BackupError as error:
# Raising here would also mask whatever the body raised.
print(f"WARNING: {root} could not be removed: {error}", flush=True)
def _btrfs( def _btrfs(
subject: str, name: str, run: Callable[[list[str]], list[str]] subject: str, name: str, run: Callable[[list[str]], list[str]]
) -> tuple[str, list[str]]: ) -> tuple[str, list[list[str]]]:
# The snapshot goes inside the subject, never beside it: the kernel rejects # The snapshot goes inside the subject, never beside it: the kernel rejects
# a snapshot whose destination is on another filesystem, which is exactly # a snapshot whose destination is on another filesystem, which is exactly
# what the parent directory is when the subject is a mountpoint of its own. # what the parent directory is when the subject is a mountpoint of its own.
target = str(Path(os.path.abspath(subject)) / f".{name}") # noqa: PTH100 - see _resolver target = str(Path(os.path.abspath(subject)) / f".{name}") # noqa: PTH100 - see _resolver
run(["btrfs", "subvolume", "snapshot", "-r", subject, target]) run(["btrfs", "subvolume", "snapshot", "-r", subject, target])
return target, ["btrfs", "subvolume", "delete", target] return target, [["btrfs", "subvolume", "delete", target]]
def _zfs( def _zfs(
subject: str, name: str, run: Callable[[list[str]], list[str]] subject: str, name: str, run: Callable[[list[str]], list[str]]
) -> tuple[str, list[str]]: ) -> tuple[str, list[list[str]]]:
output = run(["zfs", "list", "-H", "-o", "name", subject]) output = run(["zfs", "list", "-H", "-o", "name", subject])
dataset = (output[0] if output else "").strip() dataset = (output[0] if output else "").strip()
if not dataset: if not dataset:
raise SnapshotError(f"no zfs dataset is mounted at {subject}") raise SnapshotError(f"no zfs dataset is mounted at {subject}")
run(["zfs", "snapshot", f"{dataset}@{name}"]) snapshot = f"{dataset}@{name}"
root = str(Path(subject) / ".zfs" / "snapshot" / name) # Exception: mounted here rather than read through .zfs/snapshot, which the
return root, ["zfs", "destroy", f"{dataset}@{name}"] # kernel automounts via a usermode helper in the host's mount namespace,
# so a containerised run never sees it.
root = str(Path(os.path.abspath(subject)) / f".{name}") # noqa: PTH100 - see _resolver
run(["zfs", "snapshot", snapshot])
remove = [["zfs", "destroy", snapshot]]
try:
run(["mkdir", root])
remove.insert(0, ["rmdir", root])
run(["mount", "-t", "zfs", snapshot, root])
except BaseException:
_remove(root, remove, run)
raise
return root, [["umount", root], *remove]
_CREATE = {"btrfs": _btrfs, "zfs": _zfs} _CREATE = {"btrfs": _btrfs, "zfs": _zfs}
@@ -169,8 +192,4 @@ def volume_snapshot(
try: try:
yield _resolver(subject, root) yield _resolver(subject, root)
finally: finally:
try: _remove(root, remove, run)
run(remove)
except BackupError as error:
# Raising here would also mask whatever the body raised.
print(f"WARNING: {root} could not be removed: {error}", flush=True)
+1 -1
View File
@@ -75,7 +75,7 @@ def backup_volume(
pathlib.Path(dest).mkdir(parents=True, exist_ok=True) pathlib.Path(dest).mkdir(parents=True, exist_ok=True)
last = get_last_backup_dir(versions_dir, volume_name, dest) last = get_last_backup_dir(versions_dir, volume_name, dest)
cmd = ["rsync", "-aP", "--no-D", "--delete", "--delete-excluded"] cmd = ["rsync", "-aHAX", "-P", "--no-D", "--delete", "--delete-excluded"]
if authoritative: if authoritative:
cmd.append("--checksum") cmd.append("--checksum")
if last: if last:
+1 -1
View File
@@ -59,6 +59,6 @@ def restore_volume_files(volume_name: str, backup_files_dir: str) -> int:
# rsync reads "dir/" as its contents and "dir" as the directory itself. # rsync reads "dir/" as its contents and "dir" as the directory itself.
src = f"{Path(backup_files_dir)}{os.sep}" src = f"{Path(backup_files_dir)}{os.sep}"
dest = f"{Path(mountpoint)}{os.sep}" dest = f"{Path(mountpoint)}{os.sep}"
run(["rsync", "-avv", "--delete", src, dest]) run(["rsync", "-aHAX", "-vv", "--delete", src, dest])
print("File restore complete.") print("File restore complete.")
return 0 return 0
+11 -7
View File
@@ -96,12 +96,13 @@ def wait_for_mariadb(
container: str, *, root_password: str, timeout_s: int = 90 container: str, *, root_password: str, timeout_s: int = 90
) -> None: ) -> None:
""" """
Liveness probe for MariaDB. Readiness probe for MariaDB, as root over the socket.
IMPORTANT (MariaDB 11): The entrypoint drops root@127.0.0.1, so root is socket-only, and it
Root TCP auth is often restricted (unix_socket auth), so a TCP ping like installs root@localhost without unix_socket auth: a probe without the
`mariadb-admin -uroot -p... -h localhost ping` can fail even though the server is up. password only passes in the init window before the password is set.
We therefore check readiness via a socket-based query. `@@skip_networking` must be 0 so the entrypoint's temporary server,
which shuts down right after init, does not count as ready.
""" """
deadline = time.time() + timeout_s deadline = time.time() + timeout_s
while time.time() < deadline: while time.time() < deadline:
@@ -112,12 +113,15 @@ def wait_for_mariadb(
container, container,
"sh", "sh",
"-lc", "-lc",
'mariadb -uroot --protocol=socket -e "SELECT 1;"', (
f"mariadb -uroot -p{root_password} --protocol=socket "
'-N -e "SELECT @@skip_networking;"'
),
], ],
capture=True, capture=True,
check=False, check=False,
) )
if p.returncode == 0: if p.returncode == 0 and p.stdout.strip() == "0":
return return
time.sleep(1) time.sleep(1)
raise TimeoutError( raise TimeoutError(
@@ -100,7 +100,10 @@ class TestE2EMariaDBAnonymousPreemption(unittest.TestCase):
cls.db_container, cls.db_container,
"sh", "sh",
"-lc", "-lc",
f'mariadb -uroot --protocol=socket -e "{bootstrap_sql}"', (
f"mariadb -uroot -p{cls.root_password} --protocol=socket "
f'-e "{bootstrap_sql}"'
),
] ]
) )
+7 -2
View File
@@ -36,9 +36,14 @@ ZFS_READY = "{ [ -c /dev/zfs ] || modprobe zfs 2>/dev/null; }; [ -c /dev/zfs ]"
def mount_script(fstype: str) -> str: def mount_script(fstype: str) -> str:
"""Build a filesystem on a loop device and carve out the snapshot subject.""" """Build a filesystem on a loop device and carve out the snapshot subject."""
if fstype == "zfs": if fstype == "zfs":
# Exception: a loop device, not the /img file, because the kernel opens
# vdevs from a taskq thread that resolves paths in the host's root.
# The pool is kernel-wide, so it is destroyed on exit, but only once
# this script created it.
return ( return (
f"{PACKAGES} && {ZFS_READY} && truncate -s 400M /img " f"{PACKAGES} && {ZFS_READY} && truncate -s 400M /img && {ATTACH} "
"&& zpool create -m none baudolo /img " '&& zpool create -m none baudolo "$LOOP" '
"&& trap 'zpool destroy -f baudolo; losetup -d \"$LOOP\"' EXIT "
"&& zfs create -o mountpoint=/subject/docker baudolo/docker" "&& zfs create -o mountpoint=/subject/docker baudolo/docker"
) )
return ( return (
+53 -2
View File
@@ -21,6 +21,14 @@ class Runner:
return [] return []
class Unmountable(Runner):
def __call__(self, command: list[str]) -> list[str]:
result = super().__call__(command)
if command[0] == "mount":
raise BackupError("permission denied")
return result
class TestBtrfs(unittest.TestCase): class TestBtrfs(unittest.TestCase):
def test_it_creates_a_read_only_snapshot_inside_the_subject(self) -> None: def test_it_creates_a_read_only_snapshot_inside_the_subject(self) -> None:
run = Runner() run = Runner()
@@ -95,12 +103,55 @@ class TestZfs(unittest.TestCase):
run.calls[-1], ["zfs", "destroy", "tank/docker@baudolo-20260731"] run.calls[-1], ["zfs", "destroy", "tank/docker@baudolo-20260731"]
) )
def test_it_maps_a_volume_path_through_the_dot_zfs_directory(self) -> None: def test_it_mounts_the_snapshot_inside_the_subject(self) -> None:
run = self._run()
with volume_snapshot("zfs", "/var/lib/docker", "20260731", run=run):
pass
self.assertIn(
[
"mount",
"-t",
"zfs",
"tank/docker@baudolo-20260731",
"/var/lib/docker/.baudolo-20260731",
],
run.calls,
)
def test_it_maps_a_volume_path_into_the_mounted_snapshot(self) -> None:
run = self._run() run = self._run()
with volume_snapshot("zfs", "/var/lib/docker", "20260731", run=run) as resolve: with volume_snapshot("zfs", "/var/lib/docker", "20260731", run=run) as resolve:
self.assertEqual( self.assertEqual(
resolve("/var/lib/docker/volumes/postgres_data/_data"), resolve("/var/lib/docker/volumes/postgres_data/_data"),
"/var/lib/docker/.zfs/snapshot/baudolo-20260731/volumes/postgres_data/_data", "/var/lib/docker/.baudolo-20260731/volumes/postgres_data/_data",
)
def test_it_unmounts_before_it_destroys(self) -> None:
run = self._run()
with volume_snapshot("zfs", "/var/lib/docker", "20260731", run=run):
pass
self.assertEqual(
run.calls[-3:],
[
["umount", "/var/lib/docker/.baudolo-20260731"],
["rmdir", "/var/lib/docker/.baudolo-20260731"],
["zfs", "destroy", "tank/docker@baudolo-20260731"],
],
)
def test_a_failed_mount_leaves_no_snapshot_behind(self) -> None:
run = Unmountable({"zfs list": ["tank/docker"]})
with (
self.assertRaises(BackupError),
volume_snapshot("zfs", "/var/lib/docker", "20260731", run=run),
):
pass
self.assertEqual(
run.calls[-2:],
[
["rmdir", "/var/lib/docker/.baudolo-20260731"],
["zfs", "destroy", "tank/docker@baudolo-20260731"],
],
) )
def test_an_unmounted_dataset_is_an_error(self) -> None: def test_an_unmounted_dataset_is_an_error(self) -> None:
+4 -1
View File
@@ -46,9 +46,12 @@ class TestBackupVolume(unittest.TestCase):
def test_it_carries_no_kernel_objects_into_a_generation(self) -> None: def test_it_carries_no_kernel_objects_into_a_generation(self) -> None:
self.assertIn("--no-D", self.copy()) self.assertIn("--no-D", self.copy())
def test_it_stores_the_metadata_a_restore_has_to_give_back(self) -> None:
self.assertIn("-aHAX", self.copy())
def test_it_keeps_no_twin_of_what_the_second_pass_replaces(self) -> None: def test_it_keeps_no_twin_of_what_the_second_pass_replaces(self) -> None:
command = self.copy(authoritative=True) command = self.copy(authoritative=True)
self.assertEqual(command[:2], ["rsync", "-aP"]) self.assertEqual(command[:3], ["rsync", "-aHAX", "-P"])
self.assertNotIn("--backup", command) self.assertNotIn("--backup", command)
def test_it_creates_the_destination(self) -> None: def test_it_creates_the_destination(self) -> None: