Each volume is copied twice into the same destination: once hot with the
container running, once cold after it is stopped (backup/app.py:127-131).
rsync ran with -b, so --delete did not remove a file the source had dropped
between the passes - it renamed it. Stopping a container is exactly what
makes the source drop files: a graceful shutdown flushes, and the format
rolls its commit point.
For an opaque payload the twin is stale bytes nobody reads. For a format
that enumerates its own directory it is corruption. Lucene resolves the
current commit by parsing every file starting with segments as a radix-36
generation, so a restored segments_3~ raises NumberFormatException, the
shard store cannot be read, and the primary is left NO_VALID_SHARD_COPY.
With .security-7 unallocatable the reserved elastic user has no password
hash, every probe gets HTTP 401, and the container never turns healthy.
Seen in infinito-nexus-core CI run 30963648828: the restore drill waited
1200s on elasticsearch while all 29 other containers came back healthy;
the generation held segments_3~ next to segments_4 in all three index
directories. A run two days earlier passed the same drill because that
stack was idle and nothing rolled between the passes - which is why this
reads as flaky rather than broken.
Reproduced with real rsync in all three shapes: two passes into the same
destination with -b leave segments_3~ beside segments_4, without -b only
segments_4 survives, and a single pass keeps segments_3. Measured on the
same fixtures, dropping -b leaves the predecessor generation byte-identical
and keeps the --link-dest hardlinks intact, so the incremental scheme is
unaffected; generations get smaller, never larger.
--link-dest already provides the cheap incrementals. -b contributed nothing
on top of it but the twins, and no caller reads them: the restore path is an
unfiltered rsync -avv --delete into the live volume (restore/files.py:35).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
-a implies -D, so a generation was written with --devices --specials and rsync recreated every unix socket and fifo it found in a volume. On the swarm manager that generation lives on an nfs-ganesha export, and ganesha accepts the socket on write but cannot serve it back: the remote pull's sender then fails with readdir/readlink_stat 'Invalid argument (22)' and exits 23, deterministically, for all twelve retries - 58 minutes per run.
Postfix's queue directory is the case that surfaced it, where public/ and private/ hold roughly fifty AF_UNIX sockets and nothing else. The class is wider: a discourse /shared with its in-container postgres socket, a checkmk OMD site with tmp/run/nagios.cmd, a container whose /tmp is a persisted volume. --no-D is type-based and closes all of them without anyone having to know which image binds a socket where.
Nothing restorable is lost. A socket inode is meaningless after a restore; postfix's master, checkmk's omd start and discourse's supervisor recreate theirs. The whole postfix queue survives - incoming, active, deferred, hold, maildrop - so accepted-but-undelivered mail stays in the backup, which excluding the volume outright would have dropped. Device nodes go too, and the only volume that could hold them is a nested docker data root, already carrying backup: false.
This is the writer side, whose source is a local docker volume. On the reader the same flag provably does nothing: rsync still stats the entry before -D decides, and getdents64 on the containing directory is outside its reach entirely.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Backing up a live volume with rsync copies a moving target: a database
written to mid-copy lands on disk in a state no engine ever committed.
Stopping the container avoids that at the cost of downtime.
A snapshot removes both. `--snapshot {btrfs,zfs}` with `--snapshot-subject`
freezes the docker root once per run, and every volume copy is then read
from that frozen tree while the containers keep serving. A restore of such
a copy is an ordinary crash recovery, which every supported engine performs
on its own at startup.
An unsupported filesystem or an unknown snapshot kind fails loudly rather
than degrading to a live copy, since a silent fallback would return exactly
the torn backup the mode exists to prevent. `--shutdown` is rejected
alongside `--snapshot` instead of being ignored: under a snapshot no
container is ever stopped, so accepting the flag would promise downtime
semantics the run does not deliver.
Copies out of a snapshot skip rsync's --checksum verification. The source
is immutable for the lifetime of the copy, so size-and-mtime cannot race,
and dropping the second full read roughly halves the I/O per volume.
backup/app.py grew past what one module could carry and is split into
layout, policy and dumps along the lines it already had internally.
Tests: unit coverage for the new snapshot, layout, policy, volume and cli
units; e2e cases drive real btrfs, zfs and ext4 filesystems on loop devices
in a privileged container, including a MariaDB that is written to across
the snapshot and must recover from the restored copy without losing a
committed row. CI installs zfs and sets E2E_REQUIRE_FILESYSTEMS so a
missing kernel module fails the build instead of silently skipping a
filesystem.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each volume is copied twice into the same destination: once hot with the
container running, once cold after it is stopped. The cold pass is meant to
correct the hot one, but it uses rsync's quick check, which compares size and
whole-second mtime.
A pre-allocated 16 MiB WAL segment never changes size. When its last pre-stop
write and postgres' shutdown checkpoint fall in the same whole second, the cold
pass skips it while still replacing global/pg_control, whose previous write was
seconds earlier. The generation then holds a post-shutdown pg_control pointing at
a checkpoint LSN whose WAL bytes are still zero. Restoring it gives
invalid record length at 0/3CB7A20: expected at least 24, got 0
invalid checkpoint record
PANIC: could not locate a valid checkpoint record at 0/3CB7A20
and postgres crash-loops until the restore test gives up after 1200s. Observed in
infinito-nexus-core CI run 30586213932, debian leg; the centos leg of the same
run restored cleanly, and the two generations differ by exactly one 16 MiB WAL
segment.
authoritative is a required keyword rather than an optional flag, so each of the
four call sites states which pass it is. Set, it adds --checksum, so the cold
pass compares content. The hot passes keep the quick check.
Reproduced with real rsync in both directions: without --checksum the WAL stays
zero while pg_control is post-shutdown; with it the pair is consistent. Hardlink
dedup against --link-dest is unaffected, measured at 20/20 with generation size
unchanged.
Rejected alternatives, both measured rather than argued: --ignore-times destroys
dedup (20/20 to 0/20 hardlinks, generation 36 to 72 MiB), and --modify-window=-1
makes correctness depend on the destination filesystem preserving sub-second
mtimes, which silently degrades on NFS or ext3.
COST. The quick check scales with file count; --checksum scales with bytes, read
on both sides, and it runs while the container is stopped. Measured warm-cache on
215 MiB: 0.10s to 0.39s, a factor of 3.9. Derived for disk-bound runs, the extra
stop time stays under a minute up to roughly 4 GB on spinning disk, 15 GB on a
SATA SSD, 60 GB on NVMe. At 1 TB it is 17 minutes on NVMe and over three hours on
spinning disk; at 10 TB it is hours to days. No size threshold is built in - a
guessed one would drop the guarantee exactly where an unrestorable backup costs
most. Volumes at that scale need filesystem snapshots or pg_basebackup rather
than two rsync passes over a live tree, which is a limit of this approach and not
of this change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
