Four defects in the snapshot mode 3.2.0 introduced.
The resolver dropped the trailing separator get_storage_path puts on a
volume path, because os.path.abspath strips it. rsync reads "dir" as
"copy the directory" where "dir/" means "copy its contents", so every
snapshot generation landed at <volume>/files/_data/... while the live
path lands at <volume>/files/... . Restores read the live layout, and
--link-dest found nothing to match against the previous generation. The
e2e never caught it because its driver appended the separator by hand.
Snapshot teardown was fatal and masking. A busy `btrfs subvolume delete`
raised out of the finally, which skipped the generation stamp and the
compose handling on a run whose data was already complete, and replaced
whatever the body had raised. The leftover is reported instead; removing
it is a cleanup problem, not a reason to discard a good generation.
A volume created after the snapshot was taken aborted the whole run: the
volume list is enumerated inside the snapshot context, and nothing is
stopped in snapshot mode, so the host keeps creating volumes for the
duration of the copy. Such a volume is now copied live with a warning,
which is exactly what the pre-snapshot code did for it.
The snapshot pass compares by content again. 3.2.0 dropped --checksum
because a snapshot source cannot move, which is true, but the comparison
that matters is against --link-dest: a file that changed while keeping
its size and whole-second mtime was hard-linked stale out of the previous
generation, and the single snapshot pass had no authoritative pass to
repair it the way the live path does. It is still one pass against two.
--hard-restart-projects is refused alongside --snapshot, the same way
--shutdown already is: the flag exists for stacks whose database cannot
be backed up hot, which is what a snapshot removes.
Tests: the trailing separator, both teardown behaviours, the new refusal,
and app.main driving the snapshot branch - the caller that runs in
production, which no test had exercised and where the layout defect
therefore stayed invisible. The e2e driver now feeds the resolver the
string shape get_storage_path really produces.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Backing up a live volume with rsync copies a moving target: a database
written to mid-copy lands on disk in a state no engine ever committed.
Stopping the container avoids that at the cost of downtime.
A snapshot removes both. `--snapshot {btrfs,zfs}` with `--snapshot-subject`
freezes the docker root once per run, and every volume copy is then read
from that frozen tree while the containers keep serving. A restore of such
a copy is an ordinary crash recovery, which every supported engine performs
on its own at startup.
An unsupported filesystem or an unknown snapshot kind fails loudly rather
than degrading to a live copy, since a silent fallback would return exactly
the torn backup the mode exists to prevent. `--shutdown` is rejected
alongside `--snapshot` instead of being ignored: under a snapshot no
container is ever stopped, so accepting the flag would promise downtime
semantics the run does not deliver.
Copies out of a snapshot skip rsync's --checksum verification. The source
is immutable for the lifetime of the copy, so size-and-mtime cannot race,
and dropping the second full read roughly halves the I/O per volume.
backup/app.py grew past what one module could carry and is split into
layout, policy and dumps along the lines it already had internally.
Tests: unit coverage for the new snapshot, layout, policy, volume and cli
units; e2e cases drive real btrfs, zfs and ext4 filesystems on loop devices
in a privileged container, including a MariaDB that is written to across
the snapshot and must recover from the restored copy without losing a
committed row. CI installs zfs and sets E2E_REQUIRE_FILESYSTEMS so a
missing kernel module fails the build instead of silently skipping a
filesystem.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Match --images-no-stop-required and --images-no-backup-required against
the container's exact .Config.Image instead of a substring, so callers
pass full repo:tag references (registry prefix included) and near-miss
image names no longer flip the stop/skip decision. Rename the opt-in
--hard-compose-restart flag to --hard-restart-projects.
The e2e suite pins a SPOT for the DB images and in-container data dirs
(postgres:alpine at /var/lib/postgresql, mariadb:latest at
/var/lib/mysql) and passes exact image refs to the --images-* flags.
BREAKING CHANGE: --images-* now require exact image references, not
substrings; --hard-compose-restart is renamed to --hard-restart-projects.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rename --docker-compose-hard-restart-required to --hard-compose-restart
and change its default from ["mailu"] to [] (nargs="*"): the compose
down/up is now opt-in, so compose hosts pass "mailu" while swarm hosts,
where the dir is a stack whose overlay network collides with compose up,
pass nothing. Make --backups-dir mandatory (no /var/lib/backup/ default)
so a run can never silently target the wrong backup root.
BREAKING CHANGE: the old flag name is removed, the implicit mailu default
is gone, and --backups-dir must be passed explicitly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Treating every failed swarm-task inspect as skippable opened a false-green
window: a transient inspect failure on a still-running, non-whitelisted
container skipped the stop and backed the volume up hot while the run
reported success. Re-check whether the container is still listed; only a
genuinely vanished container skips, an existing one re-raises so a broken
daemon keeps failing the run loudly. Covered by unit tests for both paths.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Backup: a container that vanishes between the docker ps listing and the
swarm-task inspect (--rm one-shots, task-history GC) no longer aborts the
whole backup run; it counts as not stoppable and is skipped.
Restore: the postgres replay streams the dump through a spooled temp file
instead of buffering it three times in memory (multi-GB dumps OOMed the
restore mid-replay), and the superuser-only line filter is COPY-aware: data
rows inside COPY ... FROM stdin blocks pass through untouched, so a row
that happens to start with COMMENT ON EXTENSION or ALTER DEFAULT PRIVILEGES
is no longer silently dropped.
The e2e runner talks to the DinD daemon through docker exec instead of a
host-published tcp://127.0.0.1:2375: port publishing is unreachable from
sandboxed runners and from hosts with broken loopback publishing, and the
unencrypted root API port disappears from the host. The debug tmp dump
shrinks to tar plus docker cp against the DinD container itself.
New coverage: an e2e reproducing the swarm flake end to end (service task
on the volume, nothing whitelisted: the backup must succeed, the very same
task container must keep running, and the service must never replace a
task), unit tests for the COPY-aware filter, the swarm-task probe including
the vanished-container path, filter_stoppable ordering, and the one-session
FOREIGN_KEY_CHECKS drop assembly. Full suite: 35 unit, 9 integration,
30 e2e green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Restore fixes, both hit by the infinito svc-bkp e2e drill:
- mariadb --empty dropped tables one docker exec at a time with SET
FOREIGN_KEY_CHECKS=0 issued in a separate client session, so the
session-scoped toggle never applied and any FK-referenced parent table
(mailu.users) died with ERROR 1451. Issue the toggle and all DROPs in
one session.
- postgres --empty now drops only current_user-owned objects (extension
members like pg_trgm's set_limit are superuser-owned) with IF EXISTS
absorbing CASCADE fallout, and the replay skips superuser-only dump
lines (COMMENT ON EXTENSION, ALTER DEFAULT PRIVILEGES) that abort an
app-user psql run under ON_ERROR_STOP.
Backup fixes:
- pg_dump now runs with --no-owner --no-privileges so future dumps are
replayable by the owning app user in the first place.
- Swarm task containers are never stopped or started manually: the
orchestrator replaces a stopped task and a later docker start fails on
the detached overlay network. filter_stoppable skips them visibly and
the whitelist stop check ignores them.
Validated end to end against a live infinito compose stack: the full
svc-bkp-volume-2-local drill (verify, restore cycle, sql replay for
mailu, keycloak and one more db) passes with these patches applied.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Detect compose files case-insensitively and support:
- compose.yml / compose.yaml
- docker-compose.yml / docker-compose.yaml
Replace hard-coded docker-compose.yml checks with a shared
finder helper and extend unit tests accordingly.
https://chatgpt.com/share/69873720-d444-800f-99f7-f7799fc10c0b
- Handle missing or empty databases.csv gracefully with warnings and empty DataFrame
- Add unit tests for robust databases.csv loading behavior
- Adjust seed tests to assert warnings across multiple print calls
- Replace Debian docker.io with docker-ce-cli to avoid Docker API version mismatch
- Install required build tools (curl, gnupg) for Docker repo setup
https://chatgpt.com/share/697e6d9d-6458-800f-9d12-1e337509be4e
