Compare commits

..

5 Commits

Author SHA1 Message Date
756e236d10 Release version 3.4.1 2026-08-05 13:18:57 +02:00
1dfeb17ab4 chore(claude): require confirmation before a commit
infinito-nexus-core gates `git commit` behind an explicit per-invocation
confirmation. This repository did not, so an agent working here could commit
without being asked. Mirror the rule.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 13:00:05 +02:00
cd21f1fa67 fix(backup): keep no twin of what the cold pass replaces
Each volume is copied twice into the same destination: once hot with the
container running, once cold after it is stopped (backup/app.py:127-131).
rsync ran with -b, so --delete did not remove a file the source had dropped
between the passes - it renamed it. Stopping a container is exactly what
makes the source drop files: a graceful shutdown flushes, and the format
rolls its commit point.

For an opaque payload the twin is stale bytes nobody reads. For a format
that enumerates its own directory it is corruption. Lucene resolves the
current commit by parsing every file starting with segments as a radix-36
generation, so a restored segments_3~ raises NumberFormatException, the
shard store cannot be read, and the primary is left NO_VALID_SHARD_COPY.
With .security-7 unallocatable the reserved elastic user has no password
hash, every probe gets HTTP 401, and the container never turns healthy.

Seen in infinito-nexus-core CI run 30963648828: the restore drill waited
1200s on elasticsearch while all 29 other containers came back healthy;
the generation held segments_3~ next to segments_4 in all three index
directories. A run two days earlier passed the same drill because that
stack was idle and nothing rolled between the passes - which is why this
reads as flaky rather than broken.

Reproduced with real rsync in all three shapes: two passes into the same
destination with -b leave segments_3~ beside segments_4, without -b only
segments_4 survives, and a single pass keeps segments_3. Measured on the
same fixtures, dropping -b leaves the predecessor generation byte-identical
and keeps the --link-dest hardlinks intact, so the incremental scheme is
unaffected; generations get smaller, never larger.

--link-dest already provides the cheap incrementals. -b contributed nothing
on top of it but the twins, and no caller reads them: the restore path is an
unfiltered rsync -avv --delete into the live volume (restore/files.py:35).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-05 12:43:08 +02:00
8a93a61ca9 Release version 3.4.0 2026-08-02 12:07:34 +02:00
988d92534c fix(backup): keep kernel objects out of a generation
-a implies -D, so a generation was written with --devices --specials and rsync recreated every unix socket and fifo it found in a volume. On the swarm manager that generation lives on an nfs-ganesha export, and ganesha accepts the socket on write but cannot serve it back: the remote pull's sender then fails with readdir/readlink_stat 'Invalid argument (22)' and exits 23, deterministically, for all twelve retries - 58 minutes per run.

Postfix's queue directory is the case that surfaced it, where public/ and private/ hold roughly fifty AF_UNIX sockets and nothing else. The class is wider: a discourse /shared with its in-container postgres socket, a checkmk OMD site with tmp/run/nagios.cmd, a container whose /tmp is a persisted volume. --no-D is type-based and closes all of them without anyone having to know which image binds a socket where.

Nothing restorable is lost. A socket inode is meaningless after a restore; postfix's master, checkmk's omd start and discourse's supervisor recreate theirs. The whole postfix queue survives - incoming, active, deferred, hold, maildrop - so accepted-but-undelivered mail stays in the backup, which excluding the volume outright would have dropped. Device nodes go too, and the only volume that could hold them is a nested docker data root, already carrying backup: false.

This is the writer side, whose source is a local docker volume. On the reader the same flag provably does nothing: rsync still stats the entry before -D decides, and getdents64 on the containing directory is outside its reach entirely.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 12:06:42 +02:00
5 changed files with 51 additions and 2 deletions

View File

@@ -1,6 +1,7 @@
{
"permissions": {
"ask": [
"Bash(git commit*)",
"Edit(CHANGELOG.md)",
"Write(CHANGELOG.md)",
"Edit(pyproject.toml)",

View File

@@ -1,5 +1,42 @@
# Changelog
## [3.4.1] - 2026-08-05
- Backup: each volume is copied twice into the same destination — once hot,
once cold after the container is stopped — and rsync ran with *-b*, so
*--delete* renamed rather than removed a file the source had dropped between
the passes. Stopping a container is what makes the source drop files: a
graceful shutdown flushes and the format rolls its commit point. The
superseded file survived as *name~* beside the real one and was restored into
live data.
- Backup: for an opaque payload that is stale bytes nobody reads; for a format
that enumerates its own directory it is corruption. Lucene resolves the
current commit by parsing every file starting with *segments* as a radix-36
generation, so a restored *segments_3~* leaves the shard store unreadable and
the primary at *NO_VALID_SHARD_COPY*. With *.security-7* unallocatable the
reserved *elastic* user has no password hash, every probe answers 401 and the
container never turns healthy.
- Backup: *--link-dest* already provides the incrementals and nothing reads the
twins — the restore path is an unfiltered *rsync -avv --delete* into the live
volume. Dropping *-b* leaves the predecessor generation byte-identical, keeps
the hardlinks intact and makes generations smaller, never larger.
- Tests: the absence of *--backup* is asserted on the rsync invocation.
## [3.4.0] - 2026-08-02
- Backup: *-a* implies *-D*, so a generation was written with
*--devices --specials* and rsync recreated every unix socket and fifo found in
a volume. Where the backup root is an nfs-ganesha export, ganesha accepts the
socket on write but cannot serve it back, and the remote pull's sender then
fails with *readdir* / *readlink_stat* "Invalid argument (22)" and exits 23 —
deterministically, for every retry. *--no-D* keeps them out of the generation.
- Backup: nothing restorable is lost. Sockets and fifos are recreated by the
daemons that own them, and the postfix queue itself — *incoming*, *active*,
*deferred*, *hold*, *maildrop* — is unaffected, so accepted-but-undelivered
mail stays in the backup. Device nodes go too; the only volume that could hold
them is a nested docker data root, which does not belong in a backup anyway.
- Tests: the flag is asserted on the rsync invocation.
## [3.3.0] - 2026-08-02
- Backup: *--volumes-no-backup-required* excludes a volume by name.

View File

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "backup-docker-to-local"
version = "3.3.0"
version = "3.4.1"
description = "Backup Docker volumes to local with rsync and optional DB dumps."
readme = "README.md"
requires-python = ">=3.9"

View File

@@ -49,7 +49,10 @@ def backup_volume(
link_dest = f"--link-dest='{last}'" if last else ""
verify = "--checksum " if authoritative else ""
cmd = f"rsync -abP --delete --delete-excluded {verify}{link_dest} {source} {dest}"
cmd = (
f"rsync -aP --no-D --delete --delete-excluded "
f"{verify}{link_dest} {source} {dest}"
)
try:
execute_shell_command(cmd)

View File

@@ -43,6 +43,14 @@ class TestBackupVolume(unittest.TestCase):
def test_it_always_deletes_what_the_source_no_longer_has(self) -> None:
self.assertIn("--delete", self.copy())
def test_it_carries_no_kernel_objects_into_a_generation(self) -> None:
self.assertIn("--no-D", self.copy())
def test_it_keeps_no_twin_of_what_the_second_pass_replaces(self) -> None:
command = self.copy(authoritative=True)
self.assertIn("rsync -aP ", command)
self.assertNotIn("--backup", command)
def test_it_creates_the_destination(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
dest = Path(tmp) / "gen" / "demo"