44 Commits

Author SHA1 Message Date
779f297c85 Release version 3.1.0 2026-07-15 03:36:59 +02:00
35a4c355fe feat(cli): make --database-containers and --images-no-stop-required optional
Both default to an empty list so a pure file backup needs no dummy
arguments; an empty stop whitelist keeps the conservative stop-all
behavior.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 03:34:53 +02:00
b0ae1aba54 build(make): run the three test suites concurrently and allow e2e subsets
make test now runs clean and build once, then unit, integration and e2e
via a -j3 sub-make over run-only targets so a second clean cannot race
the build. scripts/test-e2e.sh accepts E2E_TEST_PATTERN to run a subset
of the e2e suite.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 03:34:37 +02:00
57d75b1e13 fix(restore): run --empty pre-clean per statement and drop non-public schemas
One DO-block ran every DROP in a single transaction and exhausted
max_locks_per_transaction on large schemas (gitlab); emit one DROP per
row and execute via \gexec instead. Also drop user-owned non-public
schemas so a dump that CREATE SCHEMAs (discourse) does not abort on the
existing schema under ON_ERROR_STOP.

Covered by a new DinD e2e test restoring --empty against a fully
populated database with a non-public schema and every object class.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 03:34:20 +02:00
bf5f6db7c3 Release version 3.0.0 2026-07-12 18:29:49 +02:00
b4d7e7f396 Merge branch 'feat/exact-image-match'
# Conflicts:
#	tests/e2e/helpers.py
#	tests/e2e/test_e2e_dump_only_fallback_to_files.py
#	tests/e2e/test_e2e_images_no_backup_required_early_skip.py
2026-07-12 18:27:05 +02:00
f9776ac47a feat(backup)!: exact --images-* matching and --hard-restart-projects
Match --images-no-stop-required and --images-no-backup-required against
the container's exact .Config.Image instead of a substring, so callers
pass full repo:tag references (registry prefix included) and near-miss
image names no longer flip the stop/skip decision. Rename the opt-in
--hard-compose-restart flag to --hard-restart-projects.

The e2e suite pins a SPOT for the DB images and in-container data dirs
(postgres:alpine at /var/lib/postgresql, mariadb:latest at
/var/lib/mysql) and passes exact image refs to the --images-* flags.

BREAKING CHANGE: --images-* now require exact image references, not
substrings; --hard-compose-restart is renamed to --hard-restart-projects.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 18:19:20 +02:00
8c1a6cc465 refactor(restore)!: restore volume files via direct host rsync
Drop the alpine-rsync helper image: resolve the volume mountpoint with
docker volume inspect and rsync the backup straight into it. Removes the
--rsync-image flag and the E2E_RSYNC_IMAGE pre-pull; the e2e test
container now mounts /var/lib/docker rw so the direct restore can write.

BREAKING CHANGE: the restore 'files' subcommand no longer accepts
--rsync-image; rsync must be available on the host running baudolo.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 18:18:57 +02:00
d6d4773fd9 Release version 2.0.0 2026-07-12 14:56:02 +02:00
82913291b6 test(e2e): SPOT db images, PG18 mount layout, drop removed flags
Add a single source of truth in tests/e2e/helpers.py for the database
images and their in-container data dirs (POSTGRES_IMAGE=postgres:alpine,
POSTGRES_DATA_DIR=/var/lib/postgresql, MARIADB_IMAGE=mariadb:latest,
MARIADB_DATA_DIR=/var/lib/mysql) and route every test through it.
postgres:alpine now tracks 18+, which refuses a mount at
/var/lib/postgresql/data and stores data under /var/lib/postgresql, so
the mounts and the marker path move there. Drop the --rsync-image and the
renamed hard-restart flag from the invocations.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 14:54:16 +02:00
e5da813a9f refactor(restore)!: rsync volumes directly on the host, drop alpine-rsync
Restore now resolves the target volume's mountpoint via docker volume
inspect and rsyncs into it directly, mirroring how the backup path
already reads the mountpoint; the alpine-rsync container and the
--rsync-image flag are gone. The e2e harness mounts /var/lib/docker
read-write in the test container so the direct restore can write, the
same way baudolo runs as root on a real host.

BREAKING CHANGE: the restore 'files' subcommand no longer accepts
--rsync-image; the caller must have write access to the docker volume
root (root on the host), which is baudolo's normal privilege.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 14:53:55 +02:00
331931d617 feat(backup)!: opt-in hard restart and mandatory backups-dir
Rename --docker-compose-hard-restart-required to --hard-compose-restart
and change its default from ["mailu"] to [] (nargs="*"): the compose
down/up is now opt-in, so compose hosts pass "mailu" while swarm hosts,
where the dir is a stack whose overlay network collides with compose up,
pass nothing. Make --backups-dir mandatory (no /var/lib/backup/ default)
so a run can never silently target the wrong backup root.

BREAKING CHANGE: the old flag name is removed, the implicit mailu default
is gone, and --backups-dir must be passed explicitly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 14:53:13 +02:00
45d3b0ad7c Release version 1.8.1 2026-07-12 02:59:17 +02:00
fe5bed8254 Merge branch 'main' of github.com:kevinveenbirkenbach/backup-docker-to-local 2026-07-12 02:58:36 +02:00
7a7ec57b54 fix(restore): drop user-owned collations in the --empty pre-clean
The drop_sql loop covered relations, routines, sequences and types but
not pg_collation, so a dump's CREATE COLLATION (OpenProject's ICU
public.versions_name) aborted the ON_ERROR_STOP replay with 'collation
already exists'. Add the fifth UNION ALL branch; DROP COLLATION IF
EXISTS public.<name> CASCADE rides the existing loop, and the loop
already drops every user table, so CASCADE fallout is absorbed by the
IF EXISTS no-ops.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-12 02:55:29 +02:00
2bbe7d180a Merge pull request #2 from kevinveenbirkenbach/dependabot/github_actions/actions-cad83fc9bf
Bump the actions group with 4 updates
2026-07-11 14:16:28 +02:00
dependabot[bot]
286ef179da Bump the actions group with 4 updates
Bumps the actions group with 4 updates: [actions/checkout](https://github.com/actions/checkout), [actions/upload-artifact](https://github.com/actions/upload-artifact), [docker/login-action](https://github.com/docker/login-action) and [dependabot/fetch-metadata](https://github.com/dependabot/fetch-metadata).


Updates `actions/checkout` from 4 to 7
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v4...v7)

Updates `actions/upload-artifact` from 4 to 7
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v4...v7)

Updates `docker/login-action` from 3 to 4
- [Release notes](https://github.com/docker/login-action/releases)
- [Commits](https://github.com/docker/login-action/compare/v3...v4)

Updates `dependabot/fetch-metadata` from 2 to 3
- [Release notes](https://github.com/dependabot/fetch-metadata/releases)
- [Commits](https://github.com/dependabot/fetch-metadata/compare/v2...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: actions/upload-artifact
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: docker/login-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: dependabot/fetch-metadata
  dependency-version: '3'
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
2026-07-11 07:44:12 +00:00
dependabot[bot]
6cb0b8a548 Bump python from 3.11-slim to 3.14-slim (#1)
Bumps python from 3.11-slim to 3.14-slim.

---
updated-dependencies:
- dependency-name: python
  dependency-version: 3.14-slim
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-11 07:44:09 +00:00
e1f1b602d3 Release version 1.8.0 2026-07-11 09:43:25 +02:00
b9a8b391f0 fix(backup): re-raise inspect failures for containers that still exist
Treating every failed swarm-task inspect as skippable opened a false-green
window: a transient inspect failure on a still-running, non-whitelisted
container skipped the stop and backed the volume up hot while the run
reported success. Re-check whether the container is still listed; only a
genuinely vanished container skips, an existing one re-raises so a broken
daemon keeps failing the run loudly. Covered by unit tests for both paths.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 09:32:53 +02:00
c949f2c5cf ci(deps): keep actions, docker base and pip deps updated automatically
Dependabot opens weekly PRs for the three update surfaces: github-actions
(grouped into one PR; the first one also clears the Node 20 deprecation
warning on checkout@v4 and upload-artifact@v4), the python base image, and
the pip dependencies. A companion workflow enables auto-merge for minor and
patch bumps so they land on their own once the required make test check is
green; major bumps stay manual.

Requires two repo settings: allow auto-merge, and a branch protection rule
on main with make test as required status check, which is the gate that
keeps auto-merge from landing untested changes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 09:19:12 +02:00
96e6b3ea93 fix(backup,restore): harden the branch fixes and prove them with tests
Backup: a container that vanishes between the docker ps listing and the
swarm-task inspect (--rm one-shots, task-history GC) no longer aborts the
whole backup run; it counts as not stoppable and is skipped.

Restore: the postgres replay streams the dump through a spooled temp file
instead of buffering it three times in memory (multi-GB dumps OOMed the
restore mid-replay), and the superuser-only line filter is COPY-aware: data
rows inside COPY ... FROM stdin blocks pass through untouched, so a row
that happens to start with COMMENT ON EXTENSION or ALTER DEFAULT PRIVILEGES
is no longer silently dropped.

The e2e runner talks to the DinD daemon through docker exec instead of a
host-published tcp://127.0.0.1:2375: port publishing is unreachable from
sandboxed runners and from hosts with broken loopback publishing, and the
unencrypted root API port disappears from the host. The debug tmp dump
shrinks to tar plus docker cp against the DinD container itself.

New coverage: an e2e reproducing the swarm flake end to end (service task
on the volume, nothing whitelisted: the backup must succeed, the very same
task container must keep running, and the service must never replace a
task), unit tests for the COPY-aware filter, the swarm-task probe including
the vanished-container path, filter_stoppable ordering, and the one-session
FOREIGN_KEY_CHECKS drop assembly. Full suite: 35 unit, 9 integration,
30 e2e green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 09:19:11 +02:00
79214e64e8 test(unit): mock is_swarm_task in the requires_stop tests
requires_stop now probes is_swarm_task per container, which runs a real
docker inspect; the unit CI container has no docker socket, so the three
whitelist tests died with BackupException. Mock the probe to False so
they assert the unchanged whitelist logic, and add a swarm case proving
a task container never triggers a stop and skips the image check
entirely.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 08:35:23 +02:00
e9030e8443 fix(backup,restore): make restore drills replayable and leave swarm tasks alone
Restore fixes, both hit by the infinito svc-bkp e2e drill:

- mariadb --empty dropped tables one docker exec at a time with SET
  FOREIGN_KEY_CHECKS=0 issued in a separate client session, so the
  session-scoped toggle never applied and any FK-referenced parent table
  (mailu.users) died with ERROR 1451. Issue the toggle and all DROPs in
  one session.
- postgres --empty now drops only current_user-owned objects (extension
  members like pg_trgm's set_limit are superuser-owned) with IF EXISTS
  absorbing CASCADE fallout, and the replay skips superuser-only dump
  lines (COMMENT ON EXTENSION, ALTER DEFAULT PRIVILEGES) that abort an
  app-user psql run under ON_ERROR_STOP.

Backup fixes:

- pg_dump now runs with --no-owner --no-privileges so future dumps are
  replayable by the owning app user in the first place.
- Swarm task containers are never stopped or started manually: the
  orchestrator replaces a stopped task and a later docker start fails on
  the detached overlay network. filter_stoppable skips them visibly and
  the whitelist stop check ignores them.

Validated end to end against a live infinito compose stack: the full
svc-bkp-volume-2-local drill (verify, restore cycle, sql replay for
mailu, keycloak and one more db) passes with these patches applied.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-11 08:23:11 +02:00
57ea4592c1 Release version 1.7.1 2026-05-26 00:52:37 +02:00
ad5d8fcda3 fix(backup): force TCP for mariadb-dump to match '<user>'@'%' grant
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 00:46:47 +02:00
bfa596ae30 fix(test-e2e): set DinD MTU to 1280 to fix ghcr.io pull timeouts on broken-PMTUD host paths
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 00:45:29 +02:00
21b4d237d3 Release version 1.7.0 2026-02-07 14:00:11 +01:00
ec051b4c2b backup: support all valid docker compose file names
Detect compose files case-insensitively and support:
- compose.yml / compose.yaml
- docker-compose.yml / docker-compose.yaml

Replace hard-coded docker-compose.yml checks with a shared
finder helper and extend unit tests accordingly.

https://chatgpt.com/share/69873720-d444-800f-99f7-f7799fc10c0b
2026-02-07 13:58:52 +01:00
ed78f69b3b Release version 1.6.0 2026-02-06 10:39:02 +01:00
a69074c302 test(e2e): replace ls with find to satisfy shellcheck SC2012 2026-02-06 10:37:31 +01:00
0b4696f649 backup(compose): drop custom compose/env detection and strictly delegate to wrapper or docker compose
https://chatgpt.com/share/6985b5da-d5fc-800f-b5e5-de22a199a0c8
2026-02-06 10:35:15 +01:00
e3f28098bd Release version 1.5.0 2026-01-31 22:06:05 +01:00
babadcb038 fix(backup,ci): make databases.csv optional and upgrade Docker CLI in image
- Handle missing or empty databases.csv gracefully with warnings and empty DataFrame
- Add unit tests for robust databases.csv loading behavior
- Adjust seed tests to assert warnings across multiple print calls
- Replace Debian docker.io with docker-ce-cli to avoid Docker API version mismatch
- Install required build tools (curl, gnupg) for Docker repo setup

https://chatgpt.com/share/697e6d9d-6458-800f-9d12-1e337509be4e
2026-01-31 22:01:12 +01:00
fbfdb8615f **Commit message:**
Fix Docker CLI install, switch test runners to bash, and stabilize unit tests for compose/seed mocks

https://chatgpt.com/share/697e68cd-d22c-800f-9b2e-47ef231b6502
2026-01-31 21:40:39 +01:00
2f5882f5c1 Release version 1.4.0 2026-01-31 18:28:29 +01:00
522391fdd3 Merge branch 'main' of github.com:kevinveenbirkenbach/backup-docker-to-local 2026-01-31 18:25:55 +01:00
b3c9cf5ce1 backup: restart compose stacks via wrapper-aware command resolution
- Prefer `compose` wrapper (if present) when restarting stacks to ensure
  identical file and env resolution as Infinito.Nexus
- Fallback to `docker compose` with explicit detection of:
  - docker-compose.yml
  - docker-compose.override.yml
  - docker-compose.ca.override.yml
  - .env / .env/env via --env-file
- Replace legacy `docker-compose` usage
- Log exact compose commands before execution
- Add unit tests covering wrapper vs fallback behavior

https://chatgpt.com/share/697e3b0c-85d4-800f-91a7-42324599a63c
2026-01-31 18:25:23 +01:00
2ed3472527 Ignored build/ 2026-01-16 10:45:09 +01:00
54737cefa7 Release version 1.3.0 2026-01-10 18:41:55 +01:00
d976640312 fix(seed): handle empty databases.csv and add unit tests
- Gracefully handle empty databases.csv by creating header columns and emitting a warning
- Add _empty_df() helper for consistent DataFrame initialization
- Add unit tests for baudolo-seed including empty-file regression case
- Apply minor formatting fixes across backup and e2e test files

https://chatgpt.com/share/69628f0b-8744-800f-b08d-2633e05167da
2026-01-10 18:40:22 +01:00
e4bc075474 Release version 1.2.0 2025-12-29 11:46:39 +01:00
f3ef86a444 feat(backup): stricter databases.csv semantics + atomic SQL dumps
- read databases.csv with stable types (dtype=str, keep_default_na=False)
- validate database field: require '*' or concrete name (no empty/NaN)
- support Postgres cluster dumps via '*' entries (pg_dumpall)
- write SQL dumps atomically to avoid partial/empty files
- early-skip fully ignored volumes before creating backup directories
- update seed CLI to enforce new contract and update by (instance,database)
- adjust tests: sql dir naming + add E2E coverage for early-skip and '*' seeding
2025-12-29 11:39:57 +01:00
c01ab55f2d test(e2e): add dump-only-sql mixed-run + CLI contract coverage
- rename dump-only flag to --dump-only-sql across docs and tests
- update backup logic: skip files/ only for DB volumes when dumps succeed; fallback to files when dumps fail
- extend e2e helpers to support dump_only_sql
- add e2e mixed-run regression test (DB dump => no files/, non-DB => files/)
- add e2e CLI/argparse contract test (--dump-only-sql present, --dump-only rejected)
- fix e2e files test to expect file backups for non-DB volumes in dump-only-sql mode and verify restore
- update changelog + README flag table

https://chatgpt.com/share/69522d9c-ce08-800f-9070-71df3bd779ae
2025-12-29 08:28:23 +01:00
48 changed files with 3025 additions and 370 deletions

21
.github/dependabot.yml vendored Normal file
View File

@@ -0,0 +1,21 @@
---
version: 2
updates:
- package-ecosystem: github-actions
directory: /
schedule:
interval: weekly
groups:
actions:
patterns:
- "*"
- package-ecosystem: docker
directory: /
schedule:
interval: weekly
- package-ecosystem: pip
directory: /
schedule:
interval: weekly

View File

@@ -22,7 +22,7 @@ jobs:
steps:
- name: Checkout
uses: actions/checkout@v4
uses: actions/checkout@v7
- name: Show docker info
run: |
@@ -35,7 +35,7 @@ jobs:
- name: Upload E2E artifacts (always)
if: always()
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v7
with:
name: e2e-artifacts
path: artifacts
@@ -49,7 +49,7 @@ jobs:
steps:
- name: Checkout (full history for tags)
uses: actions/checkout@v4
uses: actions/checkout@v7
with:
fetch-depth: 0
@@ -67,7 +67,7 @@ jobs:
git push -f origin stable
- name: Login to GHCR
uses: docker/login-action@v3
uses: docker/login-action@v4
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}

View File

@@ -0,0 +1,29 @@
---
name: Dependabot auto-merge
on: pull_request
permissions:
contents: write
pull-requests: write
jobs:
auto-merge:
# Enable auto-merge for minor/patch dependency bumps; majors stay manual.
# The merge only happens once every required status check (make test) is
# green, so the CI stays the gate.
if: github.actor == 'dependabot[bot]'
runs-on: ubuntu-latest
steps:
- name: Fetch dependency metadata
id: metadata
uses: dependabot/fetch-metadata@v3
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Enable auto-merge (minor + patch)
if: steps.metadata.outputs.update-type != 'version-update:semver-major'
run: gh pr merge --auto --squash "$PR_URL"
env:
PR_URL: ${{ github.event.pull_request.html_url }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

1
.gitignore vendored
View File

@@ -2,3 +2,4 @@ __pycache__
artifacts/
*.egg-info
dist/
build/

View File

@@ -1,6 +1,133 @@
# Changelog
## [3.1.0] - 2026-07-15
- Restore: the postgres *--empty* pre-clean emits one DROP per object and
runs them via *\gexec* instead of a single DO-block, so large schemas
(e.g. gitlab) no longer exhaust *max_locks_per_transaction* in one
transaction. It also drops user-owned non-public schemas, so dumps that
CREATE SCHEMA (e.g. discourse's *discourse_functions*) no longer abort
on the already-existing schema under ON_ERROR_STOP.
- Backup: *--database-containers* and *--images-no-stop-required* are now
optional and default to an empty list, so a pure file backup needs no
dummy arguments; an empty stop whitelist keeps the conservative
stop-all behavior.
- Tests: new e2e test restores *--empty* against a fully populated
database containing a non-public schema and every dropped object class.
*make test* runs the three suites concurrently after a single
clean+build; *E2E_TEST_PATTERN* runs an e2e subset.
## [3.0.0] - 2026-07-12
- Backup: *--images-no-stop-required* and *--images-no-backup-required* now
match a container's exact *.Config.Image* (full *repo:tag*, registry
prefix included) instead of a substring, so a near-miss image name no
longer flips the stop/skip decision. Callers must pass exact image
references. **Breaking.**
- Backup: renamed *--hard-compose-restart* to *--hard-restart-projects*
(its value stays a list of compose project dir names). **Breaking:** the
old flag name is removed.
## [2.0.0] - 2026-07-12
- Backup: renamed *--docker-compose-hard-restart-required* to
*--hard-compose-restart* and changed its default from *["mailu"]* to *[]*
(nargs="*"). The compose down/up is now opt-in: compose hosts pass
*mailu* explicitly, while swarm hosts pass nothing, since there the dir is
a stack whose overlay network collides with *compose up*. **Breaking:** the
old flag name is removed and the implicit mailu default is gone.
- Backup: *--backups-dir* is now required (no */var/lib/backup/* default) so
a run can never silently target the wrong backup root. **Breaking.**
- Restore: volume files are rsynced directly into the target volume's
mountpoint (resolved via *docker volume inspect*), mirroring the backup
path; the *alpine-rsync* helper image and the *--rsync-image* flag are
gone. The caller needs write access to the docker volume root (root on the
host, baudolo's normal privilege). **Breaking:** the restore *files*
subcommand no longer accepts *--rsync-image*.
- Tests: the e2e suite tracks *postgres:alpine* (18+, mounted at
*/var/lib/postgresql*) and *mariadb:latest* from a single source of truth.
## [1.8.1] - 2026-07-12
- Restore: the postgres empty mode also drops user-owned collations in
public; dumps containing CREATE COLLATION (e.g. OpenProject's ICU
collation versions_name) no longer abort the replay with 'collation
already exists'.
- Maintenance: base image bumped from python 3.11-slim to 3.14-slim.
## [1.8.0] - 2026-07-11
Swarm-aware backups and replayable restores.
- Backup: swarm task containers are never stopped or started manually
anymore; they are skipped visibly and backed up hot, while the sql dump
stays the consistent database backup.
- Backup: a container that vanishes between listing and inspect no longer
aborts the run; a failing inspect on a container that still exists keeps
failing loudly.
- Backup: pg_dump runs with the no-owner and no-privileges flags so dumps
are replayable by the owning app user.
- Restore: the mariadb empty mode drops all tables in one client session
with FOREIGN_KEY_CHECKS disabled; FK-linked parent tables no longer abort
the replay with ERROR 1451.
- Restore: the postgres empty mode drops only current-user-owned objects,
and the replay skips superuser-only dump lines without ever touching
COPY data blocks.
- Restore: the replay streams the dump through a temp file instead of
buffering it in memory; multi-GB dumps no longer OOM the restore.
- Tooling: the e2e runner reaches the DinD daemon via docker exec instead
of a host-published unencrypted API port.
- Tooling: new end-to-end test reproducing the swarm stop flake, plus unit
tests for the restore filters and the swarm probes; the suite is 36 unit,
9 integration and 30 e2e tests.
- Tooling: Dependabot with auto-merge for minor and patch updates.
## [1.7.1] - 2026-05-26
* 🔌 MariaDB SQL backups now connect over TCP loopback so the dump always matches the same wildcard-host grant the application uses — no more surprise `ERROR 1045 Access denied` when a localhost-bound auth row preempts.
* 🧪 New regression and bug-repro tests pin the TCP behaviour and prove it under the exact preemption setup that caused the production failure on MariaDB 12.
* 🩺 E2E test infrastructure: DinD bridge and inner daemon now default to MTU 1280 so registry pulls survive host paths with broken PMTUD (override via `E2E_DIND_MTU`).
## [1.7.0] - 2026-02-07
* 🚀 Backup jobs now support all valid Docker Compose file names case-insensitive and hassle-free.
## [1.6.0] - 2026-02-06
* Compose handling is now fully delegated to the Infinito.Nexus compose wrapper or plain docker compose, removing all custom env and file detection to ensure a single, consistent source of truth.
## [1.5.0] - 2026-01-31
* * Make `databases.csv` optional: missing or empty files now emit warnings and no longer break backups
* Fix Docker CLI compatibility by switching to `docker-ce-cli` and required build tools
## [1.4.0] - 2026-01-31
* Baudolo now restarts Docker Compose stacks in a wrapper-aware way (with a `docker compose` fallback), ensuring that all Compose overrides and env files are applied identically to the Infinito.Nexus workflow.
## [1.3.0] - 2026-01-10
* Empty databases.csv no longer causes baudolo-seed to fail
## [1.2.0] - 2025-12-29
* * Introduced **`--dump-only-sql`** mode for reliable, SQL-only database backups (replaces `--dump-only`).
* Database configuration in `databases.csv` is now **strict and explicit** (`*` or concrete database name only).
* **PostgreSQL cluster backups** are supported via `*`.
* SQL dumps are written **atomically** to avoid corrupted or empty files.
* Backups are **smarter and faster**: ignored volumes are skipped early, file backups run only when needed.
* Improved reliability through expanded end-to-end tests and safer defaults.
## [1.1.1] - 2025-12-28
* * **Backup:** In ***--dump-only*** mode, fall back to file backups with a warning when no database dump can be produced (e.g. missing `databases.csv` entry).
* * **Backup:** In ***--dump-only-sql*** mode, fall back to file backups with a warning when no database dump can be produced (e.g. missing `databases.csv` entry).
## [1.1.0] - 2025-12-28

View File

@@ -1,34 +1,37 @@
# syntax=docker/dockerfile:1
FROM python:3.11-slim
FROM python:3.14-slim
WORKDIR /app
# Runtime + build essentials:
# - rsync: required for file backup/restore
# - ca-certificates: TLS
# - docker-cli: needed if you want to control the host Docker engine (via /var/run/docker.sock mount)
# - make: to delegate install logic to Makefile
#
# Notes:
# - On Debian slim, the docker client package is typically "docker.io".
# - If you only want restore-without-docker, you can drop docker.io later.
# Base deps for build/runtime + docker repo key
RUN apt-get update && apt-get install -y --no-install-recommends \
make \
rsync \
ca-certificates \
docker-cli \
bash \
curl \
gnupg \
&& rm -rf /var/lib/apt/lists/*
# Install Docker CLI (docker-ce-cli) from Docker's official apt repo
RUN bash -lc "set -euo pipefail \
&& install -m 0755 -d /etc/apt/keyrings \
&& curl -fsSL https://download.docker.com/linux/debian/gpg \
| gpg --dearmor -o /etc/apt/keyrings/docker.gpg \
&& chmod a+r /etc/apt/keyrings/docker.gpg \
&& . /etc/os-release \
&& echo \"deb [arch=\$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \${VERSION_CODENAME} stable\" \
> /etc/apt/sources.list.d/docker.list \
&& apt-get update \
&& apt-get install -y --no-install-recommends docker-ce-cli \
&& rm -rf /var/lib/apt/lists/*"
# Fail fast if docker client is missing
RUN docker version || true
RUN command -v docker
COPY . .
# All install decisions are handled by the Makefile.
RUN make install
# Sensible defaults (can be overridden at runtime)
ENV PYTHONUNBUFFERED=1
# Default: show CLI help
CMD ["baudolo", "--help"]

View File

@@ -1,5 +1,6 @@
.PHONY: install build \
test-e2e test test-unit test-integration
.PHONY: install build clean \
test test-unit test-integration test-e2e \
test-unit-run test-integration-run test-e2e-run
# Default python if no venv is active
PY_DEFAULT ?= python3
@@ -33,25 +34,32 @@ build:
clean:
git clean -fdX .
# ------------------------------------------------------------
# Run E2E tests inside the container (Docker socket required)
# ------------------------------------------------------------
# E2E via isolated Docker-in-Docker (DinD)
# - depends on local image build
# - starts a DinD daemon container on a dedicated network
# - loads the freshly built image into DinD
# - runs the unittest suite inside a container that talks to DinD via DOCKER_HOST
test-e2e: clean build
@bash scripts/test-e2e.sh
# clean + build run once and in order, then the three suites run concurrently
# via -j3; the *-run targets carry no clean/build prereq so the sub-make cannot
# race a second clean against build.
test:
@$(MAKE) clean
@$(MAKE) build
@$(MAKE) -j3 test-unit-run test-integration-run test-e2e-run
test: test-unit test-integration test-e2e
test-unit: clean build test-unit-run
test-unit: clean build
test-integration: clean build test-integration-run
test-e2e: clean build test-e2e-run
test-unit-run:
@echo ">> Running unit tests"
@docker run --rm -t $(IMAGE) \
sh -lc 'python -m unittest discover -t . -s tests/unit -p "test_*.py" -v'
bash -lc 'python -m unittest discover -t . -s tests/unit -p "test_*.py" -v'
test-integration: clean build
test-integration-run:
@echo ">> Running integration tests"
@docker run --rm -t $(IMAGE) \
sh -lc 'python -m unittest discover -t . -s tests/integration -p "test_*.py" -v'
bash -lc 'python -m unittest discover -t . -s tests/integration -p "test_*.py" -v'
# E2E via isolated Docker-in-Docker (DinD): starts a DinD daemon on a dedicated
# network, loads the freshly built image into it, and runs tests/e2e inside a
# container that talks to DinD via DOCKER_HOST.
test-e2e-run:
@bash scripts/test-e2e.sh

View File

@@ -134,7 +134,7 @@ baudolo \
| Flag | Description |
| --------------- | ------------------------------------------- |
| `--everything` | Always stop containers and re-run rsync |
| `--dump-only` | Only create SQL dumps, skip file backups |
| `--dump-only-sql`| Skip file backups only for DB volumes when dumps succeed; non-DB volumes are still backed up; fallback to files if no dump. |
| `--shutdown` | Do not restart containers after backup |
| `--backups-dir` | Backup root directory (default: `/Backups`) |
| `--repo-name` | Backup namespace under machine hash |

View File

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "backup-docker-to-local"
version = "1.1.1"
version = "3.1.0"
description = "Backup Docker volumes to local with rsync and optional DB dumps."
readme = "README.md"
requires-python = ">=3.9"

View File

@@ -16,19 +16,30 @@ DIND="${E2E_DIND_NAME:-baudolo-e2e-dind}"
DIND_VOL="${E2E_DIND_VOL:-baudolo-e2e-dind-data}"
E2E_TMP_VOL="${E2E_TMP_VOL:-baudolo-e2e-tmp}"
DIND_HOST="${E2E_DIND_HOST:-tcp://127.0.0.1:2375}"
# Host-side access to the DinD daemon goes through `docker exec` (dind()
# below) instead of a host-published port: port publishing is not reachable
# from every environment (sandboxed runners, hosts with broken loopback
# publishing), while exec only needs the outer docker socket. The TCP
# listener stays for the test container inside the dedicated network.
DIND_HOST_IN_NET="${E2E_DIND_HOST_IN_NET:-tcp://${DIND}:2375}"
dind() { docker exec "${DIND}" docker "$@"; }
dind_stdin() { docker exec -i "${DIND}" docker "$@"; }
IMG="${E2E_IMAGE:-baudolo:local}"
RSYNC_IMG="${E2E_RSYNC_IMAGE:-ghcr.io/kevinveenbirkenbach/alpine-rsync}"
READY_TIMEOUT_SECONDS="${E2E_READY_TIMEOUT_SECONDS:-120}"
ARTIFACTS_DIR="${E2E_ARTIFACTS_DIR:-./artifacts}"
DIND_MTU="${E2E_DIND_MTU:-1280}"
KEEP_ON_FAIL="${E2E_KEEP_ON_FAIL:-0}"
KEEP_VOLUMES="${E2E_KEEP_VOLUMES:-0}"
DEBUG_SHELL="${E2E_DEBUG_SHELL:-0}"
# Override to run a subset, e.g. E2E_TEST_PATTERN=test_e2e_postgres_empty_drop_hard.py
TEST_PATTERN="${E2E_TEST_PATTERN:-test_*.py}"
FAILED=0
TS="$(date +%Y%m%d%H%M%S)"
@@ -46,8 +57,8 @@ dump_debug() {
echo "=== Host docker info ==="
docker info || true
echo
echo "=== DinD reachable? (docker -H ${DIND_HOST} version) ==="
docker -H "${DIND_HOST}" version || true
echo "=== DinD reachable? (docker exec ${DIND} docker version) ==="
dind version || true
echo
} > "${ARTIFACTS_DIR}/debug-host-${TS}.txt" 2>&1 || true
@@ -56,48 +67,34 @@ dump_debug() {
# DinD state
{
echo "=== docker -H ps -a ==="
docker -H "${DIND_HOST}" ps -a || true
echo "=== dind ps -a ==="
dind ps -a || true
echo
echo "=== docker -H images ==="
docker -H "${DIND_HOST}" images || true
echo "=== dind images ==="
dind images || true
echo
echo "=== docker -H network ls ==="
docker -H "${DIND_HOST}" network ls || true
echo "=== dind network ls ==="
dind network ls || true
echo
echo "=== docker -H volume ls ==="
docker -H "${DIND_HOST}" volume ls || true
echo "=== dind volume ls ==="
dind volume ls || true
echo
echo "=== docker -H system df ==="
docker -H "${DIND_HOST}" system df || true
echo "=== dind system df ==="
dind system df || true
} > "${ARTIFACTS_DIR}/debug-dind-${TS}.txt" 2>&1 || true
# Try to capture recent events (best effort; might be noisy)
docker -H "${DIND_HOST}" events --since 10m --until 0s \
dind events --since 10m --until 0s \
> "${ARTIFACTS_DIR}/dind-events-${TS}.txt" 2>&1 || true
# Dump shared /tmp content from the tmp volume:
# We create a temporary container that mounts the volume, then tar its content.
# (Does not rely on host filesystem paths.)
# The shared tmp volume is mounted at /tmp inside the DinD container
# itself, so tar it there and copy it out with the outer daemon.
log "DEBUG: archiving shared /tmp (volume ${E2E_TMP_VOL})"
docker -H "${DIND_HOST}" run --rm \
-v "${E2E_TMP_VOL}:/tmp" \
alpine:3.20 \
sh -lc 'cd /tmp && tar -czf /out.tar.gz . || true' \
>/dev/null 2>&1 || true
# The above writes inside the container FS, not to host. So do it properly:
# Use "docker cp" from a temp container.
local tmpc="baudolo-e2e-tmpdump-${TS}"
docker -H "${DIND_HOST}" rm -f "${tmpc}" >/dev/null 2>&1 || true
docker -H "${DIND_HOST}" create --name "${tmpc}" -v "${E2E_TMP_VOL}:/tmp" alpine:3.20 \
sh -lc 'cd /tmp && tar -czf /tmpdump.tar.gz . || true' >/dev/null
docker -H "${DIND_HOST}" start -a "${tmpc}" >/dev/null 2>&1 || true
docker -H "${DIND_HOST}" cp "${tmpc}:/tmpdump.tar.gz" "${ARTIFACTS_DIR}/e2e-tmp-${TS}.tar.gz" >/dev/null 2>&1 || true
docker -H "${DIND_HOST}" rm -f "${tmpc}" >/dev/null 2>&1 || true
docker exec "${DIND}" tar -czf "/tmpdump-${TS}.tar.gz" -C /tmp . >/dev/null 2>&1 || true
docker cp "${DIND}:/tmpdump-${TS}.tar.gz" "${ARTIFACTS_DIR}/e2e-tmp-${TS}.tar.gz" >/dev/null 2>&1 || true
log "DEBUG: artifacts written:"
ls -la "${ARTIFACTS_DIR}" | sed 's/^/ /' || true
find "${ARTIFACTS_DIR}" -maxdepth 1 -mindepth 1 -print | sed 's/^/ /' || true
}
cleanup() {
@@ -105,9 +102,9 @@ cleanup() {
log "KEEP_ON_FAIL=1 and failure detected -> skipping cleanup."
log "Next steps:"
echo " - Inspect DinD logs: docker logs ${DIND} | less"
echo " - Use DinD daemon: docker -H ${DIND_HOST} ps -a"
echo " - Shared tmp vol: docker -H ${DIND_HOST} run --rm -v ${E2E_TMP_VOL}:/tmp alpine:3.20 ls -la /tmp"
echo " - DinD docker root: docker -H ${DIND_HOST} run --rm -v ${DIND_VOL}:/var/lib/docker alpine:3.20 ls -la /var/lib/docker/volumes"
echo " - Use DinD daemon: docker exec ${DIND} docker ps -a"
echo " - Shared tmp vol: docker exec ${DIND} ls -la /tmp"
echo " - DinD docker root: docker exec ${DIND} ls -la /var/lib/docker/volumes"
return 0
fi
@@ -124,8 +121,11 @@ cleanup() {
}
trap cleanup EXIT INT TERM
log "Creating network ${NET} (if missing)"
docker network inspect "${NET}" >/dev/null 2>&1 || docker network create "${NET}" >/dev/null
log "(Re)creating network ${NET} with MTU ${DIND_MTU}"
docker network rm "${NET}" >/dev/null 2>&1 || true
docker network create \
--opt com.docker.network.driver.mtu="${DIND_MTU}" \
"${NET}" >/dev/null
log "Removing old ${DIND} (if any)"
docker rm -f "${DIND}" >/dev/null 2>&1 || true
@@ -145,14 +145,14 @@ docker run -d --privileged \
-e DOCKER_TLS_CERTDIR="" \
-v "${DIND_VOL}:/var/lib/docker" \
-v "${E2E_TMP_VOL}:/tmp" \
-p 2375:2375 \
docker:dind \
--host=tcp://0.0.0.0:2375 \
--tls=false >/dev/null
--tls=false \
--mtu="${DIND_MTU}" >/dev/null
log "Waiting for DinD to be ready..."
for i in $(seq 1 "${READY_TIMEOUT_SECONDS}"); do
if docker -H "${DIND_HOST}" version >/dev/null 2>&1; then
if dind version >/dev/null 2>&1; then
log "DinD is ready."
break
fi
@@ -166,15 +166,11 @@ for i in $(seq 1 "${READY_TIMEOUT_SECONDS}"); do
fi
done
log "Pre-pulling helper images in DinD..."
log " - Pulling: ${RSYNC_IMG}"
docker -H "${DIND_HOST}" pull "${RSYNC_IMG}"
log "Ensuring alpine exists in DinD (for debug helpers)"
docker -H "${DIND_HOST}" pull alpine:3.20 >/dev/null
dind pull alpine:3.20 >/dev/null
log "Loading ${IMG} image into DinD..."
docker save "${IMG}" | docker -H "${DIND_HOST}" load >/dev/null
docker save "${IMG}" | dind_stdin load >/dev/null
log "Running E2E tests inside DinD"
set +e
@@ -183,11 +179,10 @@ if [ "${DEBUG_SHELL}" = "1" ]; then
docker run --rm -it \
--network "${NET}" \
-e DOCKER_HOST="${DIND_HOST_IN_NET}" \
-e E2E_RSYNC_IMAGE="${RSYNC_IMG}" \
-v "${DIND_VOL}:/var/lib/docker:ro" \
-v "${DIND_VOL}:/var/lib/docker" \
-v "${E2E_TMP_VOL}:/tmp" \
"${IMG}" \
sh -lc '
bash -lc '
set -e
if [ ! -f /etc/machine-id ]; then
mkdir -p /etc
@@ -195,18 +190,18 @@ if [ "${DEBUG_SHELL}" = "1" ]; then
fi
echo ">> DOCKER_HOST=${DOCKER_HOST}"
docker ps -a || true
exec sh
exec bash
'
rc=$?
else
docker run --rm \
--network "${NET}" \
-e DOCKER_HOST="${DIND_HOST_IN_NET}" \
-e E2E_RSYNC_IMAGE="${RSYNC_IMG}" \
-v "${DIND_VOL}:/var/lib/docker:ro" \
-e E2E_TEST_PATTERN="${TEST_PATTERN}" \
-v "${DIND_VOL}:/var/lib/docker" \
-v "${E2E_TMP_VOL}:/tmp" \
"${IMG}" \
sh -lc '
bash -lc '
set -euo pipefail
set -x
export PYTHONUNBUFFERED=1
@@ -218,7 +213,7 @@ else
cat /proc/sys/kernel/random/uuid > /etc/machine-id
fi
python -m unittest discover -t . -s tests/e2e -p "test_*.py" -v -f
python -m unittest discover -t . -s tests/e2e -p "${E2E_TEST_PATTERN}" -v -f
'
rc=$?
fi

View File

@@ -2,10 +2,12 @@ from __future__ import annotations
import os
import pathlib
import sys
from datetime import datetime
import pandas
from dirval import create_stamp_file
from pandas.errors import EmptyDataError
from .cli import parse_args
from .compose import handle_docker_compose_services
@@ -14,8 +16,10 @@ from .docker import (
change_containers_status,
containers_using_volume,
docker_volume_names,
filter_stoppable,
get_image_info,
has_image,
is_swarm_task,
)
from .shell import execute_shell_command
from .volume import backup_volume
@@ -48,7 +52,7 @@ def is_image_ignored(container: str, images_no_backup_required: list[str]) -> bo
if not images_no_backup_required:
return False
img = get_image_info(container)
return any(pat in img for pat in images_no_backup_required)
return img in images_no_backup_required
def volume_is_fully_ignored(
@@ -64,14 +68,19 @@ def volume_is_fully_ignored(
def requires_stop(containers: list[str], images_no_stop_required: list[str]) -> bool:
"""
Stop is required if ANY container image is NOT in the whitelist patterns.
Stop is required if ANY stoppable container image is NOT in the exact
image whitelist. Swarm task containers never count: baudolo must
not cycle them (see docker.is_swarm_task).
"""
for c in containers:
if is_swarm_task(c):
continue
img = get_image_info(c)
if not any(pat in img for pat in images_no_stop_required):
if img not in images_no_stop_required:
return True
return False
def backup_mariadb_or_postgres(
*,
container: str,
@@ -95,6 +104,42 @@ def backup_mariadb_or_postgres(
return False, False
def _empty_databases_df() -> "pandas.DataFrame":
"""
Create an empty DataFrame with the expected schema for databases.csv.
This allows the backup to continue without DB dumps when the CSV is missing
or empty (pandas EmptyDataError).
"""
return pandas.DataFrame(columns=["instance", "database", "username", "password"])
def _load_databases_df(csv_path: str) -> "pandas.DataFrame":
"""
Load databases.csv robustly.
- Missing file -> warn, continue with empty df
- Empty file -> warn, continue with empty df
- Valid CSV -> return dataframe
"""
try:
return pandas.read_csv(csv_path, sep=";", keep_default_na=False, dtype=str)
except FileNotFoundError:
print(
f"WARNING: databases.csv not found: {csv_path}. Continuing without database dumps.",
file=sys.stderr,
flush=True,
)
return _empty_databases_df()
except EmptyDataError:
print(
f"WARNING: databases.csv exists but is empty: {csv_path}. Continuing without database dumps.",
file=sys.stderr,
flush=True,
)
return _empty_databases_df()
def _backup_dumps_for_volume(
*,
containers: list[str],
@@ -132,7 +177,13 @@ def main() -> int:
versions_dir = os.path.join(args.backups_dir, machine_id, args.repo_name)
version_dir = create_version_directory(versions_dir, backup_time)
databases_df = pandas.read_csv(args.databases_csv, sep=";")
# IMPORTANT:
# - keep_default_na=False prevents empty fields from turning into NaN
# - dtype=str keeps all columns stable for comparisons/validation
#
# Robust behavior:
# - if the file is missing or empty, we continue without DB dumps.
databases_df = _load_databases_df(args.databases_csv)
print("💾 Start volume backups...", flush=True)
@@ -140,6 +191,14 @@ def main() -> int:
print(f"Start backup routine for volume: {volume_name}", flush=True)
containers = containers_using_volume(volume_name)
# EARLY SKIP: if all linked containers are ignored, do not create any dirs
if volume_is_fully_ignored(containers, args.images_no_backup_required):
print(
f"Skipping volume '{volume_name}' entirely (all linked containers are ignored).",
flush=True,
)
continue
vol_dir = create_volume_directory(version_dir, volume_name)
found_db, dumped_any = _backup_dumps_for_volume(
@@ -149,51 +208,45 @@ def main() -> int:
database_containers=args.database_containers,
)
# dump-only logic:
if args.dump_only:
if found_db and not dumped_any:
# dump-only-sql logic:
if args.dump_only_sql:
if found_db:
if not dumped_any:
print(
f"WARNING: dump-only requested but no DB dump was produced for DB volume '{volume_name}'. Falling back to file backup.",
f"WARNING: dump-only-sql requested but no DB dump was produced for DB volume '{volume_name}'. "
"Falling back to file backup.",
flush=True,
)
# continue to file backup below
# fall through to file backup below
else:
# keep old behavior: skip file backups
continue
# skip file backup if all linked containers are ignored
if volume_is_fully_ignored(containers, args.images_no_backup_required):
print(
f"Skipping file backup for volume '{volume_name}' (all linked containers are ignored).",
flush=True,
)
# DB volume successfully dumped -> skip file backup
continue
# Non-DB volume -> always do file backup (fall through)
if args.everything:
# "everything": always do pre-rsync, then stop + rsync again
stoppable = filter_stoppable(containers)
backup_volume(versions_dir, volume_name, vol_dir)
change_containers_status(containers, "stop")
change_containers_status(stoppable, "stop")
backup_volume(versions_dir, volume_name, vol_dir)
if not args.shutdown:
change_containers_status(containers, "start")
change_containers_status(stoppable, "start")
continue
# default: rsync, and if needed stop + rsync
backup_volume(versions_dir, volume_name, vol_dir)
if requires_stop(containers, args.images_no_stop_required):
change_containers_status(containers, "stop")
stoppable = filter_stoppable(containers)
change_containers_status(stoppable, "stop")
backup_volume(versions_dir, volume_name, vol_dir)
if not args.shutdown:
change_containers_status(containers, "start")
change_containers_status(stoppable, "start")
# Stamp the backup version directory using dirval (python lib)
stamp_directory(version_dir)
print("Finished volume backups.", flush=True)
print("Handling Docker Compose services...", flush=True)
handle_docker_compose_services(
args.compose_dir, args.docker_compose_hard_restart_required
)
handle_docker_compose_services(args.compose_dir, args.hard_restart_projects)
return 0

View File

@@ -17,10 +17,10 @@ def parse_args() -> argparse.Namespace:
help="Path to the parent directory containing docker-compose setups",
)
p.add_argument(
"--docker-compose-hard-restart-required",
nargs="+",
default=["mailu"],
help="Compose dir names that require 'docker-compose down && up -d' (default: mailu)",
"--hard-restart-projects",
nargs="*",
default=[],
help="Compose dir names that require 'docker-compose down && up -d' (default: none; pass e.g. 'mailu' under compose where the DB cannot be backed up hot)",
)
p.add_argument(
@@ -35,27 +35,27 @@ def parse_args() -> argparse.Namespace:
)
p.add_argument(
"--backups-dir",
default="/var/lib/backup/",
help="Backup root directory (default: /var/lib/backup/)",
required=True,
help="Backup root directory (e.g. /var/lib/backup/)",
)
p.add_argument(
"--database-containers",
nargs="+",
required=True,
default=[],
help="Container names treated as special instances for database backups",
)
p.add_argument(
"--images-no-stop-required",
nargs="+",
required=True,
help="Image name patterns for which containers should not be stopped during file backup",
default=[],
help="Exact image references (repo:tag, incl. any registry prefix) whose containers must not be stopped during file backup",
)
p.add_argument(
"--images-no-backup-required",
nargs="+",
default=[],
help="Image name patterns for which no backup should be performed",
help="Exact image references (repo:tag, incl. any registry prefix) for which no backup should be performed",
)
p.add_argument(
@@ -68,10 +68,15 @@ def parse_args() -> argparse.Namespace:
action="store_true",
help="Do not restart containers after backup",
)
p.add_argument(
"--dump-only",
action="store_true",
help="Only create DB dumps (skip ALL file rsync backups)",
)
p.add_argument(
"--dump-only-sql",
action="store_true",
help=(
"Create database dumps only for DB volumes. "
"File backups are skipped for DB volumes if a dump succeeds, "
"but non-DB volumes are still backed up. "
"If a DB dump cannot be produced, baudolo falls back to a file backup."
),
)
return p.parse_args()

View File

@@ -1,17 +1,82 @@
from __future__ import annotations
import os
import shutil
import subprocess
from pathlib import Path
from typing import List, Optional
def _build_compose_cmd(project_dir: str, passthrough: List[str]) -> List[str]:
"""
Build the compose command for this project directory.
Policy:
- If `compose` wrapper exists (Infinito.Nexus): use it and delegate ALL logic to it.
- Else: use plain `docker compose` with --chdir.
- NO custom compose file/env detection in this project.
"""
pdir = Path(project_dir).resolve()
wrapper = shutil.which("compose")
if wrapper:
# "--" ensures wrapper stops parsing its own args.
return [wrapper, "--chdir", str(pdir), "--", *passthrough]
docker = shutil.which("docker")
if docker:
return [docker, "compose", "--chdir", str(pdir), *passthrough]
raise RuntimeError("Neither 'compose' nor 'docker' found in PATH")
def _find_compose_file(project_dir: str) -> Optional[Path]:
"""
Detect a compose file in `project_dir` (case-insensitive).
Supported names:
- compose.yml / compose.yaml
- docker-compose.yml / docker-compose.yaml
"""
pdir = Path(project_dir)
if not pdir.is_dir():
return None
# Map lowercase filename -> actual Path (preserves original casing)
by_lower = {p.name.lower(): p for p in pdir.iterdir() if p.is_file()}
# Preferred order (policy decision)
candidates = [
"docker-compose.yml",
"docker-compose.yaml",
"compose.yml",
"compose.yaml",
]
for name in candidates:
found = by_lower.get(name)
if found is not None:
return found
return None
def hard_restart_docker_services(dir_path: str) -> None:
print(f"Hard restart docker-compose services in: {dir_path}", flush=True)
subprocess.run(["docker-compose", "down"], cwd=dir_path, check=True)
subprocess.run(["docker-compose", "up", "-d"], cwd=dir_path, check=True)
print(f"Hard restart compose services in: {dir_path}", flush=True)
down_cmd = _build_compose_cmd(dir_path, ["down"])
up_cmd = _build_compose_cmd(dir_path, ["up", "-d"])
print(">>> " + " ".join(down_cmd), flush=True)
subprocess.run(down_cmd, check=True)
print(">>> " + " ".join(up_cmd), flush=True)
subprocess.run(up_cmd, check=True)
def handle_docker_compose_services(
parent_directory: str, hard_restart_required: list[str]
parent_directory: str,
hard_restart_required: list[str],
) -> None:
for entry in os.scandir(parent_directory):
if not entry.is_dir():
@@ -19,11 +84,12 @@ def handle_docker_compose_services(
dir_path = entry.path
name = os.path.basename(dir_path)
compose_file = os.path.join(dir_path, "docker-compose.yml")
print(f"Checking directory: {dir_path}", flush=True)
if not os.path.isfile(compose_file):
print("No docker-compose.yml found. Skipping.", flush=True)
compose_file = _find_compose_file(dir_path)
if compose_file is None:
print("No supported compose file found. Skipping.", flush=True)
continue
if name in hard_restart_required:

View File

@@ -4,6 +4,8 @@ import os
import pathlib
import re
import logging
from typing import Optional
import pandas
from .shell import BackupException, execute_shell_command
@@ -12,17 +14,55 @@ log = logging.getLogger(__name__)
def get_instance(container: str, database_containers: list[str]) -> str:
"""
Derive a stable instance name from the container name.
"""
if container in database_containers:
return container
return re.split(r"(_|-)(database|db|postgres)", container)[0]
def fallback_pg_dumpall(container: str, username: str, password: str, out_file: str) -> None:
def _validate_database_value(value: Optional[str], *, instance: str) -> str:
"""
Enforce explicit database semantics:
- "*" => dump ALL databases (cluster dump for Postgres)
- "<name>" => dump exactly this database
- "" => invalid configuration (would previously result in NaN / nan.backup.sql)
"""
v = (value or "").strip()
if v == "":
raise ValueError(
f"Invalid databases.csv entry for instance '{instance}': "
"column 'database' must be '*' or a concrete database name (not empty)."
)
return v
def _atomic_write_cmd(cmd: str, out_file: str) -> None:
"""
Write dump output atomically:
- write to <file>.tmp
- rename to <file> only on success
This prevents empty or partial dump files from being treated as valid backups.
"""
tmp = f"{out_file}.tmp"
execute_shell_command(f"{cmd} > {tmp}")
execute_shell_command(f"mv {tmp} {out_file}")
def fallback_pg_dumpall(
container: str, username: str, password: str, out_file: str
) -> None:
"""
Perform a full Postgres cluster dump using pg_dumpall.
"""
cmd = (
f"PGPASSWORD={password} docker exec -i {container} "
f"pg_dumpall -U {username} -h localhost > {out_file}"
f"pg_dumpall -U {username} -h localhost"
)
execute_shell_command(cmd)
_atomic_write_cmd(cmd, out_file)
def backup_database(
@@ -34,12 +74,15 @@ def backup_database(
database_containers: list[str],
) -> bool:
"""
Returns True if at least one dump file was produced, else False.
Backup databases for a given DB container.
Returns True if at least one dump was produced.
"""
instance_name = get_instance(container, database_containers)
entries = databases_df.loc[databases_df["instance"] == instance_name]
entries = databases_df[databases_df["instance"] == instance_name]
if entries.empty:
log.warning("No entry found for instance '%s' (skipping DB dump)", instance_name)
log.debug("No database entries for instance '%s'", instance_name)
return False
out_dir = os.path.join(volume_dir, "sql")
@@ -48,43 +91,57 @@ def backup_database(
produced = False
for row in entries.itertuples(index=False):
db_name = row.database
user = row.username
password = row.password
raw_db = getattr(row, "database", "")
user = (getattr(row, "username", "") or "").strip()
password = (getattr(row, "password", "") or "").strip()
db_value = _validate_database_value(raw_db, instance=instance_name)
# Explicit: dump ALL databases
if db_value == "*":
if db_type != "postgres":
raise ValueError(
f"databases.csv entry for instance '{instance_name}': "
"'*' is currently only supported for Postgres."
)
cluster_file = os.path.join(out_dir, f"{instance_name}.cluster.backup.sql")
fallback_pg_dumpall(container, user, password, cluster_file)
produced = True
continue
# Concrete database dump
db_name = db_value
dump_file = os.path.join(out_dir, f"{db_name}.backup.sql")
if db_type == "mariadb":
# Force TCP so auth matches '<user>'@'%' instead of socket -> 'localhost'.
cmd = (
f"docker exec {container} /usr/bin/mariadb-dump "
f"-u {user} -p{password} {db_name} > {dump_file}"
f"-h 127.0.0.1 --protocol=tcp "
f"-u {user} -p{password} {db_name}"
)
execute_shell_command(cmd)
_atomic_write_cmd(cmd, dump_file)
produced = True
continue
if db_type == "postgres":
cluster_file = os.path.join(out_dir, f"{instance_name}.cluster.backup.sql")
if not db_name:
fallback_pg_dumpall(container, user, password, cluster_file)
return True
try:
cmd = (
f"PGPASSWORD={password} docker exec -i {container} "
f"pg_dump -U {user} -d {db_name} -h localhost > {dump_file}"
f"pg_dump -U {user} -d {db_name} -h localhost "
f"--no-owner --no-privileges"
)
execute_shell_command(cmd)
_atomic_write_cmd(cmd, dump_file)
produced = True
except BackupException as e:
print(f"pg_dump failed: {e}", flush=True)
print(
f"Falling back to pg_dumpall for instance '{instance_name}'",
flush=True,
# Explicit DB dump failed -> hard error
raise BackupException(
f"Postgres dump failed for instance '{instance_name}', "
f"database '{db_name}'. This database was explicitly configured "
"and therefore must succeed.\n"
f"{e}"
)
fallback_pg_dumpall(container, user, password, cluster_file)
produced = True
continue
return produced

View File

@@ -1,6 +1,6 @@
from __future__ import annotations
from .shell import execute_shell_command
from .shell import BackupException, execute_shell_command
def get_image_info(container: str) -> str:
@@ -24,6 +24,43 @@ def containers_using_volume(volume_name: str) -> list[str]:
)
def is_swarm_task(container: str) -> bool:
"""Swarm-managed task containers must never be stopped or started
manually: the orchestrator replaces the stopped task and a later
`docker start` fails on the detached overlay network. A container that
vanished between listing and inspect (--rm one-shots, task-history GC)
counts as not stoppable instead of aborting the whole backup run; if the
container still exists the inspect failure re-raises, so a broken daemon
keeps failing the run loudly instead of silently skipping the stop."""
try:
out = execute_shell_command(
"docker inspect --format "
f"'{{{{index .Config.Labels \"com.docker.swarm.task.id\"}}}}' {container}"
)
except BackupException:
still_listed = execute_shell_command(
f"docker ps -a --filter name=^{container}$ --format '{{{{.Names}}}}'"
)
if still_listed and still_listed[0].strip():
raise
return True
return bool(out and out[0].strip())
def filter_stoppable(containers: list[str]) -> list[str]:
"""Containers baudolo may stop/start itself (everything but swarm tasks)."""
stoppable = []
for container in containers:
if is_swarm_task(container):
print(
f"Skipping stop/start for swarm task container '{container}'.",
flush=True,
)
continue
stoppable.append(container)
return stoppable
def change_containers_status(containers: list[str], status: str) -> None:
"""Stop or start a list of containers."""
if not containers:

View File

@@ -38,10 +38,6 @@ def main(argv: list[str] | None = None) -> int:
# ------------------------------------------------------------------
p_files = sub.add_parser("files", help="Restore files into a docker volume")
_add_common_backup_args(p_files)
p_files.add_argument(
"--rsync-image",
default="ghcr.io/kevinveenbirkenbach/alpine-rsync",
)
p_files.add_argument(
"--source-volume",
default=None,
@@ -95,7 +91,6 @@ def main(argv: list[str] | None = None) -> int:
return restore_volume_files(
args.volume_name,
bp_files.files_dir(),
rsync_image=args.rsync_image,
)
if args.cmd == "postgres":

View File

@@ -47,18 +47,6 @@ def restore_mariadb_sql(
# IMPORTANT:
# Do NOT hardcode 'mysql' here. Use the detected client.
# MariaDB 11 images may not contain the mysql binary at all.
docker_exec(
container,
[
client,
"-u",
user,
f"--password={password}",
"-e",
"SET FOREIGN_KEY_CHECKS=0;",
],
)
result = docker_exec(
container,
[
@@ -74,19 +62,16 @@ def restore_mariadb_sql(
)
tables = result.stdout.decode().split()
for tbl in tables:
docker_exec(
container,
[
client,
"-u",
user,
f"--password={password}",
"-e",
f"DROP TABLE IF EXISTS `{db_name}`.`{tbl}`;",
],
if tables:
# SET FOREIGN_KEY_CHECKS is session-scoped, so it must share one
# client session with the DROPs or FK constraints still fire.
drop_sql = (
"SET FOREIGN_KEY_CHECKS=0; "
+ " ".join(
f"DROP TABLE IF EXISTS `{db_name}`.`{tbl}`;" for tbl in tables
)
+ " SET FOREIGN_KEY_CHECKS=1;"
)
docker_exec(
container,
[
@@ -95,7 +80,7 @@ def restore_mariadb_sql(
user,
f"--password={password}",
"-e",
"SET FOREIGN_KEY_CHECKS=1;",
drop_sql,
],
)

View File

@@ -1,9 +1,41 @@
from __future__ import annotations
import os
import tempfile
from collections.abc import Iterable, Iterator
from ..run import docker_exec
_SUPERUSER_ONLY_PREFIXES = (b"COMMENT ON EXTENSION", b"ALTER DEFAULT PRIVILEGES")
def filter_superuser_only_lines(lines: Iterable[bytes]) -> Iterator[bytes]:
"""Drop superuser-only statements an app-level psql replay cannot run.
Args:
lines: dump lines including their trailing newlines.
Yields:
Every line except top-level statements starting with a superuser-only
prefix. Lines inside COPY ... FROM stdin data blocks are passed
through untouched: a data row may legally start with the same bytes,
and dropping it would silently corrupt the restored table.
"""
in_copy = False
for line in lines:
if in_copy:
yield line
if line.rstrip(b"\r\n") == b"\\.":
in_copy = False
continue
if line.startswith(b"COPY ") and line.rstrip(b"\r\n").endswith(b"FROM stdin;"):
in_copy = True
yield line
continue
if line.startswith(_SUPERUSER_ONLY_PREFIXES):
continue
yield line
def restore_postgres_sql(
*,
@@ -21,19 +53,58 @@ def restore_postgres_sql(
docker_env = {"PGPASSWORD": password}
if empty:
# Owner-filtered pre-clean emitted as one DROP per row and run via \gexec so each
# executes as its own top-level statement: a single DO-block runs every DROP in one
# transaction and exhausts max_locks_per_transaction on large schemas (e.g. gitlab).
# Also drop user-owned non-public schemas so a dump that CREATE SCHEMAs (e.g.
# discourse's discourse_functions) does not fail on an already-existing schema.
# Extension members (pg_trgm's set_limit) are superuser-owned; IF EXISTS absorbs CASCADE fallout.
drop_sql = r"""
DO $$ DECLARE r RECORD;
BEGIN
FOR r IN (
SELECT table_name AS name, 'TABLE' AS type FROM information_schema.tables WHERE table_schema='public'
SELECT format('DROP %s IF EXISTS public.%I CASCADE', obj.type, obj.name)
FROM (
SELECT c.relname AS name,
CASE c.relkind
WHEN 'v' THEN 'VIEW'
WHEN 'm' THEN 'MATERIALIZED VIEW'
WHEN 'f' THEN 'FOREIGN TABLE'
ELSE 'TABLE'
END AS type
FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'public' AND c.relkind IN ('r', 'p', 'v', 'm', 'f')
AND pg_get_userbyid(c.relowner) = current_user
UNION ALL
SELECT routine_name AS name, 'FUNCTION' AS type FROM information_schema.routines WHERE specific_schema='public'
SELECT p.proname AS name,
CASE p.prokind WHEN 'p' THEN 'PROCEDURE' ELSE 'FUNCTION' END AS type
FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace
WHERE n.nspname = 'public' AND p.prokind IN ('f', 'p', 'w')
AND pg_get_userbyid(p.proowner) = current_user
UNION ALL
SELECT sequence_name AS name, 'SEQUENCE' AS type FROM information_schema.sequences WHERE sequence_schema='public'
) LOOP
EXECUTE format('DROP %s public.%I CASCADE', r.type, r.name);
END LOOP;
END $$;
SELECT c.relname AS name, 'SEQUENCE' AS type
FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = 'public' AND c.relkind = 'S'
AND pg_get_userbyid(c.relowner) = current_user
UNION ALL
SELECT t.typname AS name, 'TYPE' AS type
FROM pg_type t JOIN pg_namespace n ON n.oid = t.typnamespace
WHERE n.nspname = 'public'
AND pg_get_userbyid(t.typowner) = current_user
AND (t.typtype IN ('e', 'd')
OR (t.typtype = 'c' AND EXISTS (
SELECT 1 FROM pg_class c2
WHERE c2.oid = t.typrelid AND c2.relkind = 'c')))
UNION ALL
SELECT col.collname AS name, 'COLLATION' AS type
FROM pg_collation col JOIN pg_namespace n ON n.oid = col.collnamespace
WHERE n.nspname = 'public'
AND pg_get_userbyid(col.collowner) = current_user
) obj
UNION ALL
SELECT format('DROP SCHEMA IF EXISTS %I CASCADE', n.nspname)
FROM pg_namespace n
WHERE NOT starts_with(n.nspname, 'pg_')
AND n.nspname NOT IN ('public', 'information_schema')
AND pg_get_userbyid(n.nspowner) = current_user
\gexec
"""
docker_exec(
container,
@@ -42,11 +113,17 @@ END $$;
docker_env=docker_env,
)
with open(sql_path, "rb") as f:
# Filter into a spooled temp file instead of building the whole dump in
# memory: production dumps reach many GB and the previous read/splitlines/
# join needed roughly three times the dump size in RSS.
with open(sql_path, "rb") as src, tempfile.TemporaryFile() as filtered:
for line in filter_superuser_only_lines(src):
filtered.write(line)
filtered.seek(0)
docker_exec(
container,
["psql", "-v", "ON_ERROR_STOP=1", "-U", user, "-d", db_name],
stdin=f,
stdin=filtered,
docker_env=docker_env,
)

View File

@@ -3,12 +3,10 @@ from __future__ import annotations
import os
import sys
from .run import run, docker_volume_exists
from .run import docker_volume_exists, run
def restore_volume_files(
volume_name: str, backup_files_dir: str, *, rsync_image: str
) -> int:
def restore_volume_files(volume_name: str, backup_files_dir: str) -> int:
if not os.path.isdir(backup_files_dir):
print(f"ERROR: backup files dir not found: {backup_files_dir}", file=sys.stderr)
return 2
@@ -19,21 +17,21 @@ def restore_volume_files(
else:
print(f"Volume {volume_name} already exists.")
# Keep behavior close to the old script: rsync -avv --delete
run(
[
"docker",
"run",
"--rm",
"-v",
f"{volume_name}:/recover/",
"-v",
f"{backup_files_dir}:/backup/",
rsync_image,
"sh",
"-lc",
"rsync -avv --delete /backup/ /recover/",
]
cp = run(
["docker", "volume", "inspect", "--format", "{{ .Mountpoint }}", volume_name],
capture=True,
)
raw = cp.stdout or b""
mountpoint = (raw.decode() if isinstance(raw, bytes) else raw).strip()
if not mountpoint:
print(
f"ERROR: could not resolve mountpoint for volume {volume_name}",
file=sys.stderr,
)
return 2
src = os.path.join(backup_files_dir, "")
dest = os.path.join(mountpoint, "")
run(["rsync", "-avv", "--delete", src, dest])
print("File restore complete.")
return 0

View File

@@ -1,67 +1,116 @@
import pandas as pd
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import os
import re
import sys
import pandas as pd
from typing import Optional
from pandas.errors import EmptyDataError
DB_NAME_RE = re.compile(r"^[a-zA-Z0-9_][a-zA-Z0-9_-]*$")
def check_and_add_entry(file_path, instance, database, username, password):
# Check if the file exists and is not empty
if os.path.exists(file_path) and os.path.getsize(file_path) > 0:
# Read the existing CSV file with header
df = pd.read_csv(file_path, sep=";")
def _validate_database_value(value: Optional[str], *, instance: str) -> str:
v = (value or "").strip()
if v == "":
raise ValueError(
f"Invalid databases.csv entry for instance '{instance}': "
"column 'database' must be '*' or a concrete database name (not empty)."
)
if v == "*":
return "*"
if v.lower() == "nan":
raise ValueError(
f"Invalid databases.csv entry for instance '{instance}': database must not be 'nan'."
)
if not DB_NAME_RE.match(v):
raise ValueError(
f"Invalid databases.csv entry for instance '{instance}': "
f"invalid database name '{v}'. Allowed: letters, numbers, '_' and '-'."
)
return v
def _empty_df() -> pd.DataFrame:
return pd.DataFrame(columns=["instance", "database", "username", "password"])
def check_and_add_entry(
file_path: str,
instance: str,
database: Optional[str],
username: str,
password: str,
) -> None:
"""
Add or update an entry in databases.csv.
The function enforces strict validation:
- database MUST be set
- database MUST be '*' or a valid database name
"""
database = _validate_database_value(database, instance=instance)
if os.path.exists(file_path):
try:
df = pd.read_csv(
file_path,
sep=";",
dtype=str,
keep_default_na=False,
)
except EmptyDataError:
print(
f"WARNING: databases.csv exists but is empty: {file_path}. Creating header columns.",
file=sys.stderr,
)
df = _empty_df()
else:
# Create a new DataFrame with columns if file does not exist
df = pd.DataFrame(columns=["instance", "database", "username", "password"])
df = _empty_df()
mask = (df["instance"] == instance) & (df["database"] == database)
# Check if the entry exists and remove it
mask = (
(df["instance"] == instance)
& (
(df["database"] == database)
| (((df["database"].isna()) | (df["database"] == "")) & (database == ""))
)
& (df["username"] == username)
)
if not df[mask].empty:
print("Replacing existing entry.")
df = df[~mask]
if mask.any():
print("Updating existing entry.")
df.loc[mask, ["username", "password"]] = [username, password]
else:
print("Adding new entry.")
# Create a new DataFrame for the new entry
new_entry = pd.DataFrame(
[
{
"instance": instance,
"database": database,
"username": username,
"password": password,
}
]
[[instance, database, username, password]],
columns=["instance", "database", "username", "password"],
)
# Add (or replace) the entry using concat
df = pd.concat([df, new_entry], ignore_index=True)
# Save the updated CSV file
df.to_csv(file_path, sep=";", index=False)
def main():
def main() -> None:
parser = argparse.ArgumentParser(
description="Check and replace (or add) a database entry in a CSV file."
description="Seed or update databases.csv for backup configuration."
)
parser.add_argument("file_path", help="Path to the CSV file")
parser.add_argument("instance", help="Database instance")
parser.add_argument("database", help="Database name")
parser.add_argument("username", help="Username")
parser.add_argument("password", nargs="?", default="", help="Password (optional)")
parser.add_argument("file", help="Path to databases.csv")
parser.add_argument("instance", help="Instance name (e.g. bigbluebutton)")
parser.add_argument(
"database",
help="Database name or '*' to dump all databases",
)
parser.add_argument("username", help="Database username")
parser.add_argument("password", help="Database password")
args = parser.parse_args()
try:
check_and_add_entry(
args.file_path, args.instance, args.database, args.username, args.password
file_path=args.file,
instance=args.instance,
database=args.database,
username=args.username,
password=args.password,
)
except Exception as exc:
print(f"ERROR: {exc}", file=sys.stderr)
sys.exit(1)
if __name__ == "__main__":

View File

@@ -7,6 +7,14 @@ import time
import uuid
from pathlib import Path
# SPOT for the database images and their in-container data dirs the e2e
# suite runs against. postgres:alpine tracks latest (18+), which mounts at
# /var/lib/postgresql (not /var/lib/postgresql/data); bump here only.
POSTGRES_IMAGE = "postgres:alpine"
POSTGRES_DATA_DIR = "/var/lib/postgresql"
MARIADB_IMAGE = "mariadb:latest"
MARIADB_DATA_DIR = "/var/lib/mysql"
def run(
cmd: list[str],
@@ -166,13 +174,13 @@ def backup_run(
database_containers: list[str],
images_no_stop_required: list[str],
images_no_backup_required: list[str] | None = None,
dump_only: bool = False,
dump_only_sql: bool = False,
) -> None:
cmd = [
"baudolo",
"--compose-dir",
compose_dir,
"--docker-compose-hard-restart-required",
"--hard-restart-projects",
"mailu",
"--repo-name",
repo_name,
@@ -187,8 +195,8 @@ def backup_run(
]
if images_no_backup_required:
cmd += ["--images-no-backup-required", *images_no_backup_required]
if dump_only:
cmd += ["--dump-only"]
if dump_only_sql:
cmd += ["--dump-only-sql"]
try:
run(cmd, capture=True, check=True)

View File

@@ -0,0 +1,29 @@
import unittest
from .helpers import run
class TestE2ECLIContractDumpOnlySql(unittest.TestCase):
def test_help_mentions_new_flag(self) -> None:
cp = run(["baudolo", "--help"], capture=True, check=True)
out = (cp.stdout or "") + "\n" + (cp.stderr or "")
self.assertIn(
"--dump-only-sql",
out,
f"Expected '--dump-only-sql' to appear in --help output. Output:\n{out}",
)
def test_old_flag_is_rejected(self) -> None:
cp = run(["baudolo", "--dump-only"], capture=True, check=False)
self.assertEqual(
cp.returncode,
2,
f"Expected exitcode 2 for unknown args, got {cp.returncode}\n"
f"STDOUT={cp.stdout}\nSTDERR={cp.stderr}",
)
err = (cp.stderr or "") + "\n" + (cp.stdout or "")
# Argparse typically prints "unrecognized arguments"
self.assertTrue(
("unrecognized arguments" in err) or ("usage:" in err.lower()),
f"Expected argparse-style error output. Output:\n{err}",
)

View File

@@ -2,6 +2,8 @@
import unittest
from .helpers import (
POSTGRES_IMAGE,
POSTGRES_DATA_DIR,
backup_path,
cleanup_docker,
create_minimal_compose_dir,
@@ -19,7 +21,7 @@ class TestE2EDumpOnlyFallbackToFiles(unittest.TestCase):
@classmethod
def setUpClass(cls) -> None:
require_docker()
cls.prefix = unique("baudolo-e2e-dump-only-fallback")
cls.prefix = unique("baudolo-e2e-dump-only-sql-fallback")
cls.backups_dir = f"/tmp/{cls.prefix}/Backups"
ensure_empty_dir(cls.backups_dir)
@@ -50,14 +52,14 @@ class TestE2EDumpOnlyFallbackToFiles(unittest.TestCase):
"-e",
"POSTGRES_USER=postgres",
"-v",
f"{cls.pg_volume}:/var/lib/postgresql/data",
"postgres:16",
f"{cls.pg_volume}:{POSTGRES_DATA_DIR}",
POSTGRES_IMAGE,
]
)
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)
# Add a deterministic marker file into the volume
cls.marker = "dump-only-fallback-marker"
cls.marker = "dump-only-sql-fallback-marker"
run(
[
"docker",
@@ -65,7 +67,7 @@ class TestE2EDumpOnlyFallbackToFiles(unittest.TestCase):
cls.pg_container,
"sh",
"-lc",
f"echo '{cls.marker}' > /var/lib/postgresql/data/marker.txt",
f"echo '{cls.marker}' > {POSTGRES_DATA_DIR}/marker.txt",
]
)
@@ -73,13 +75,13 @@ class TestE2EDumpOnlyFallbackToFiles(unittest.TestCase):
cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv"
write_databases_csv(cls.databases_csv, []) # empty except header
# Run baudolo with --dump-only and a DB container present:
# Run baudolo with --dump-only-sql and a DB container present:
# Expected: WARNING + FALLBACK to file backup (files/ must exist)
cmd = [
"baudolo",
"--compose-dir",
cls.compose_dir,
"--docker-compose-hard-restart-required",
"--hard-restart-projects",
"mailu",
"--repo-name",
cls.repo_name,
@@ -90,11 +92,8 @@ class TestE2EDumpOnlyFallbackToFiles(unittest.TestCase):
"--database-containers",
cls.pg_container,
"--images-no-stop-required",
"postgres",
"mariadb",
"mysql",
"alpine",
"--dump-only",
POSTGRES_IMAGE,
"--dump-only-sql",
]
cp = run(cmd, capture=True, check=True)
@@ -116,8 +115,6 @@ class TestE2EDumpOnlyFallbackToFiles(unittest.TestCase):
cls.repo_name,
"--source-volume",
cls.pg_volume,
"--rsync-image",
"ghcr.io/kevinveenbirkenbach/alpine-rsync",
]
)
@@ -127,28 +124,34 @@ class TestE2EDumpOnlyFallbackToFiles(unittest.TestCase):
def test_warns_about_missing_dump_in_dump_only_mode(self) -> None:
self.assertIn(
"WARNING: dump-only requested but no DB dump was produced",
"WARNING: dump-only-sql requested but no DB dump was produced",
self.stdout,
f"Expected warning in baudolo output. STDOUT:\n{self.stdout}",
)
def test_files_backup_exists_due_to_fallback(self) -> None:
p = backup_path(
p = (
backup_path(
self.backups_dir,
self.repo_name,
self.version,
self.pg_volume,
) / "files"
)
/ "files"
)
self.assertTrue(p.is_dir(), f"Expected files backup dir at: {p}")
def test_sql_dump_not_present(self) -> None:
# There should be no sql dumps because databases.csv had no matching entry.
sql_dir = backup_path(
sql_dir = (
backup_path(
self.backups_dir,
self.repo_name,
self.version,
self.pg_volume,
) / "sql"
)
/ "sql"
)
# Could exist (dir created) in some edge cases, but should contain no *.sql dumps.
if sql_dir.exists():
dumps = list(sql_dir.glob("*.sql"))

View File

@@ -0,0 +1,183 @@
import unittest
from .helpers import (
POSTGRES_IMAGE,
POSTGRES_DATA_DIR,
backup_path,
cleanup_docker,
create_minimal_compose_dir,
ensure_empty_dir,
latest_version_dir,
require_docker,
run,
unique,
wait_for_postgres,
write_databases_csv,
)
class TestE2EDumpOnlySqlMixedRun(unittest.TestCase):
@classmethod
def setUpClass(cls) -> None:
require_docker()
cls.prefix = unique("baudolo-e2e-dump-only-sql-mixed-run")
cls.backups_dir = f"/tmp/{cls.prefix}/Backups"
ensure_empty_dir(cls.backups_dir)
cls.compose_dir = create_minimal_compose_dir(f"/tmp/{cls.prefix}")
cls.repo_name = cls.prefix
# --- Volumes ---
cls.db_volume = f"{cls.prefix}-vol-db"
cls.files_volume = f"{cls.prefix}-vol-files"
# Track for cleanup
cls.containers: list[str] = []
cls.volumes = [cls.db_volume, cls.files_volume]
# Create volumes
run(["docker", "volume", "create", cls.db_volume])
run(["docker", "volume", "create", cls.files_volume])
# Put a marker into the non-db volume
run(
[
"docker",
"run",
"--rm",
"-v",
f"{cls.files_volume}:/data",
"alpine:3.20",
"sh",
"-lc",
"echo 'hello-non-db' > /data/hello.txt",
]
)
# --- Start Postgres container using the DB volume ---
cls.pg_container = f"{cls.prefix}-pg"
cls.containers.append(cls.pg_container)
cls.pg_password = "postgres"
cls.pg_db = "testdb"
cls.pg_user = "postgres"
run(
[
"docker",
"run",
"-d",
"--name",
cls.pg_container,
"-e",
f"POSTGRES_PASSWORD={cls.pg_password}",
"-v",
f"{cls.db_volume}:{POSTGRES_DATA_DIR}",
POSTGRES_IMAGE,
]
)
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)
# Create deterministic content in DB so dump is non-empty
run(
[
"docker",
"exec",
cls.pg_container,
"sh",
"-lc",
f'psql -U postgres -c "CREATE DATABASE {cls.pg_db};" || true',
],
check=True,
)
run(
[
"docker",
"exec",
cls.pg_container,
"sh",
"-lc",
(
f"psql -U postgres -d {cls.pg_db} -c "
'"CREATE TABLE IF NOT EXISTS t (id INT PRIMARY KEY, v TEXT);'
"INSERT INTO t(id,v) VALUES (1,'hello-db') "
'ON CONFLICT (id) DO UPDATE SET v=EXCLUDED.v;"'
),
],
check=True,
)
# databases.csv with an entry => dump should succeed
cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv"
write_databases_csv(
cls.databases_csv,
[(cls.pg_container, cls.pg_db, cls.pg_user, cls.pg_password)],
)
# Run baudolo with dump-only-sql
cmd = [
"baudolo",
"--compose-dir",
cls.compose_dir,
"--databases-csv",
cls.databases_csv,
"--database-containers",
cls.pg_container,
"--images-no-stop-required",
POSTGRES_IMAGE,
"--dump-only-sql",
"--backups-dir",
cls.backups_dir,
"--repo-name",
cls.repo_name,
]
cp = run(cmd, capture=True, check=True)
cls.stdout = cp.stdout
cls.stderr = cp.stderr
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)
@classmethod
def tearDownClass(cls) -> None:
cleanup_docker(containers=cls.containers, volumes=cls.volumes)
def test_db_volume_has_dump_and_no_files_dir(self) -> None:
base = backup_path(
self.backups_dir, self.repo_name, self.version, self.db_volume
)
dumps = base / "sql"
files = base / "files"
self.assertTrue(dumps.exists(), f"Expected dumps dir for DB volume at: {dumps}")
self.assertFalse(
files.exists(),
f"Did not expect files dir for DB volume when dump succeeded at: {files}",
)
# Optional: at least one dump file exists
dump_files = list(dumps.glob("*.sql")) + list(dumps.glob("*.sql.gz"))
self.assertTrue(
dump_files,
f"Expected at least one SQL dump file in {dumps}, found none.",
)
def test_non_db_volume_has_files_dir(self) -> None:
base = backup_path(
self.backups_dir, self.repo_name, self.version, self.files_volume
)
files = base / "files"
self.assertTrue(
files.exists(),
f"Expected files dir for non-DB volume at: {files}",
)
def test_dump_only_sql_does_not_disable_non_db_files_backup(self) -> None:
# Regression guard: even with --dump-only-sql, non-DB volumes must still be backed up as files
base = backup_path(
self.backups_dir, self.repo_name, self.version, self.files_volume
)
self.assertTrue(
(base / "files").exists(),
f"Expected non-DB volume files backup to exist at: {base / 'files'}",
)

View File

@@ -57,7 +57,7 @@ class TestE2EFilesFull(unittest.TestCase):
compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv,
database_containers=["dummy-db"],
images_no_stop_required=["alpine", "postgres", "mariadb", "mysql"],
images_no_stop_required=["alpine:3.20"],
)
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)
@@ -94,8 +94,6 @@ class TestE2EFilesFull(unittest.TestCase):
self.repo_name,
"--source-volume",
self.volume_src,
"--rsync-image",
"ghcr.io/kevinveenbirkenbach/alpine-rsync",
]
)

View File

@@ -26,10 +26,10 @@ class TestE2EFilesNoCopy(unittest.TestCase):
cls.repo_name = cls.prefix
cls.volume_src = f"{cls.prefix}-vol-src"
cls.volume_dst = f"{cls.prefix}-vol-dst"
cls.containers = []
cls.volumes = [cls.volume_src, cls.volume_dst]
cls.containers: list[str] = []
cls.volumes = [cls.volume_src]
# Create source volume and write a marker file
run(["docker", "volume", "create", cls.volume_src])
run(
[
@@ -48,36 +48,40 @@ class TestE2EFilesNoCopy(unittest.TestCase):
cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv"
write_databases_csv(cls.databases_csv, [])
# dump-only => NO file rsync backups
# dump-only-sql => non-DB volumes are STILL backed up as files
backup_run(
backups_dir=cls.backups_dir,
repo_name=cls.repo_name,
compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv,
database_containers=["dummy-db"],
images_no_stop_required=["alpine", "postgres", "mariadb", "mysql"],
dump_only=True,
images_no_stop_required=["alpine:3.20"],
dump_only_sql=True,
)
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)
# Wipe the volume to ensure restore actually restores something
run(["docker", "volume", "rm", "-f", cls.volume_src])
run(["docker", "volume", "create", cls.volume_src])
@classmethod
def tearDownClass(cls) -> None:
cleanup_docker(containers=cls.containers, volumes=cls.volumes)
def test_files_backup_not_present(self) -> None:
def test_files_backup_present_for_non_db_volume(self) -> None:
p = (
backup_path(self.backups_dir, self.repo_name, self.version, self.volume_src)
/ "files"
)
self.assertFalse(p.exists(), f"Did not expect files backup dir at: {p}")
self.assertTrue(p.exists(), f"Expected files backup dir at: {p}")
def test_restore_files_fails_expected(self) -> None:
def test_restore_files_succeeds_and_restores_content(self) -> None:
p = run(
[
"baudolo-restore",
"files",
self.volume_dst,
self.volume_src,
self.hash,
self.version,
"--backups-dir",
@@ -89,6 +93,27 @@ class TestE2EFilesNoCopy(unittest.TestCase):
)
self.assertEqual(
p.returncode,
2,
f"Expected exitcode 2, got {p.returncode}\nSTDOUT={p.stdout}\nSTDERR={p.stderr}",
0,
f"Expected exitcode 0, got {p.returncode}\nSTDOUT={p.stdout}\nSTDERR={p.stderr}",
)
cp = run(
[
"docker",
"run",
"--rm",
"-v",
f"{self.volume_src}:/data",
"alpine:3.20",
"sh",
"-lc",
"cat /data/hello.txt",
],
capture=True,
check=True,
)
self.assertEqual(
cp.stdout.strip(),
"hello",
f"Unexpected restored content. STDOUT={cp.stdout}\nSTDERR={cp.stderr}",
)

View File

@@ -0,0 +1,127 @@
# tests/e2e/test_e2e_images_no_backup_required_early_skip.py
import unittest
from .helpers import (
backup_path,
cleanup_docker,
create_minimal_compose_dir,
ensure_empty_dir,
latest_version_dir,
require_docker,
run,
unique,
write_databases_csv,
)
class TestE2EImagesNoBackupRequiredEarlySkip(unittest.TestCase):
@classmethod
def setUpClass(cls) -> None:
require_docker()
cls.prefix = unique("baudolo-e2e-early-skip-no-backup-required")
cls.backups_dir = f"/tmp/{cls.prefix}/Backups"
ensure_empty_dir(cls.backups_dir)
cls.compose_dir = create_minimal_compose_dir(f"/tmp/{cls.prefix}")
cls.repo_name = cls.prefix
# --- Docker resources ---
cls.redis_container = f"{cls.prefix}-redis"
cls.ignored_volume = f"{cls.prefix}-redis-vol"
cls.normal_volume = f"{cls.prefix}-files-vol"
cls.containers = [cls.redis_container]
cls.volumes = [cls.ignored_volume, cls.normal_volume]
# Create volumes
run(["docker", "volume", "create", cls.ignored_volume])
run(["docker", "volume", "create", cls.normal_volume])
# Start redis container using the ignored volume
run(
[
"docker",
"run",
"-d",
"--name",
cls.redis_container,
"-v",
f"{cls.ignored_volume}:/data",
"redis:alpine",
]
)
# Put deterministic content into the normal volume
run(
[
"docker",
"run",
"--rm",
"-v",
f"{cls.normal_volume}:/data",
"alpine:3.20",
"sh",
"-lc",
"mkdir -p /data && echo 'hello' > /data/hello.txt",
]
)
# databases.csv required by CLI (can be empty)
cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv"
write_databases_csv(cls.databases_csv, [])
# Run baudolo with images-no-backup-required redis
cmd = [
"baudolo",
"--compose-dir",
cls.compose_dir,
"--hard-restart-projects",
"mailu",
"--repo-name",
cls.repo_name,
"--databases-csv",
cls.databases_csv,
"--backups-dir",
cls.backups_dir,
"--database-containers",
"dummy-db",
"--images-no-stop-required",
"redis:alpine",
"--images-no-backup-required",
"redis:alpine",
]
cp = run(cmd, capture=True, check=True)
cls.stdout = cp.stdout or ""
cls.stderr = cp.stderr or ""
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)
@classmethod
def tearDownClass(cls) -> None:
cleanup_docker(containers=cls.containers, volumes=cls.volumes)
def test_ignored_volume_has_no_backup_directory_at_all(self) -> None:
p = backup_path(
self.backups_dir,
self.repo_name,
self.version,
self.ignored_volume,
)
self.assertFalse(
p.exists(),
f"Expected NO backup directory to be created for ignored volume, but found: {p}",
)
def test_normal_volume_is_still_backed_up(self) -> None:
p = (
backup_path(
self.backups_dir,
self.repo_name,
self.version,
self.normal_volume,
)
/ "files"
/ "hello.txt"
)
self.assertTrue(p.is_file(), f"Expected backed up file at: {p}")

View File

@@ -0,0 +1,164 @@
"""
Bug-repro for: mariadb-dump fails with `ERROR 1045 Access denied for user
'<u>'@'localhost' (using password: YES)` when only '<u>'@'%' is granted and a
preempting ''@'localhost' user is present.
The fix forces TCP loopback in baudolo.backup.db so the dump matches the
'<u>'@'%' grant instead of the socket->localhost auth row.
This file:
- builds the exact preconditions that triggered the production failure,
- as a NEGATIVE control, runs a socket-based mariadb-dump (== the old code path)
and asserts that it fails with the literal 1045 / @'localhost' error,
- as a POSITIVE proof, calls backup_database() (where the fix lives) against
the same DB container and asserts the dump file is produced and contains the
seed data.
Note: the volume-rsync stage of baudolo is intentionally NOT exercised here.
That stage needs root on /var/lib/docker/volumes, which is provided by the
DinD wrapper in `make test-e2e` but not by an on-host invocation. The bug we
are verifying is in the DB-dump stage, so testing backup_database() directly
keeps the assertion focused and the test runnable both on-host and in DinD.
"""
import os
import tempfile
import unittest
import pandas
from baudolo.backup import db as db_mod
from .helpers import (
MARIADB_IMAGE,
MARIADB_DATA_DIR,
cleanup_docker,
require_docker,
run,
unique,
wait_for_mariadb,
wait_for_mariadb_sql,
)
class TestE2EMariaDBAnonymousPreemption(unittest.TestCase):
@classmethod
def setUpClass(cls) -> None:
require_docker()
cls.prefix = unique("baudolo-e2e-mariadb-anon")
cls.db_container = f"{cls.prefix}-mariadb"
cls.db_volume = f"{cls.prefix}-mariadb-vol"
cls.containers = [cls.db_container]
cls.volumes = [cls.db_volume]
cls.db_name = "appdb"
cls.db_user = "tcponly"
cls.db_password = "tcponlypw"
cls.root_password = "rootpw"
run(["docker", "volume", "create", cls.db_volume])
# Boot WITHOUT MARIADB_USER/MARIADB_PASSWORD/MARIADB_DATABASE so the
# entrypoint does not auto-create '<u>'@'%'. We provision the user
# explicitly below to mirror the SQL path used by svc-db-mariadb.
run(
[
"docker",
"run",
"-d",
"--name",
cls.db_container,
"-e",
f"MARIADB_ROOT_PASSWORD={cls.root_password}",
"-v",
f"{cls.db_volume}:{MARIADB_DATA_DIR}",
MARIADB_IMAGE,
]
)
wait_for_mariadb(
cls.db_container, root_password=cls.root_password, timeout_s=120
)
# Provision: '<u>'@'%' (the app/backup grant) + anonymous ''@'localhost'
# (the preemption trigger). Mirrors the production state that produced
# `ERROR 1045 ... '<u>'@'localhost' (using password: YES)`.
bootstrap_sql = (
f"CREATE DATABASE {cls.db_name};"
f"CREATE USER '{cls.db_user}'@'%' IDENTIFIED BY '{cls.db_password}';"
f"GRANT ALL PRIVILEGES ON {cls.db_name}.* TO '{cls.db_user}'@'%';"
f"CREATE USER ''@'localhost' IDENTIFIED BY 'anonpw-not-{cls.db_password}';"
"FLUSH PRIVILEGES;"
f"CREATE TABLE {cls.db_name}.t (id INT PRIMARY KEY, v VARCHAR(50));"
f"INSERT INTO {cls.db_name}.t VALUES (1,'ok');"
)
run(
[
"docker",
"exec",
cls.db_container,
"sh",
"-lc",
f'mariadb -uroot --protocol=socket -e "{bootstrap_sql}"',
]
)
# Sanity: '<u>' can log in over TCP (matches '%'). If THIS fails,
# the precondition for the fix to even apply is broken.
wait_for_mariadb_sql(
cls.db_container, user=cls.db_user, password=cls.db_password, timeout_s=60
)
@classmethod
def tearDownClass(cls) -> None:
cleanup_docker(containers=cls.containers, volumes=cls.volumes)
def test_negative_control_socket_dump_fails_with_1045(self) -> None:
# Reproduces the OLD code path (no -h/--protocol). MUST fail with 1045
# under the configured preemption. If this ever starts passing, either
# the MariaDB auth semantics changed or the anonymous-user setup did
# not take effect — in both cases the positive test below loses its
# ability to discriminate "fix works" vs "bug never reproduced".
p = run(
[
"docker",
"exec",
self.db_container,
"sh",
"-lc",
f"mariadb-dump -u{self.db_user} -p{self.db_password} {self.db_name}",
],
capture=True,
check=False,
)
self.assertNotEqual(p.returncode, 0, "socket-based dump unexpectedly succeeded")
self.assertIn("1045", (p.stderr or "") + (p.stdout or ""))
self.assertIn("@'localhost'", (p.stderr or "") + (p.stdout or ""))
def test_backup_database_succeeds_with_tcp_fix(self) -> None:
# Drives the function where the fix lives. No rsync, no privileged
# paths — just the dump that the negative-control proved is failing
# under the same preemption setup.
with tempfile.TemporaryDirectory() as volume_dir:
df = pandas.DataFrame(
[(self.db_container, self.db_name, self.db_user, self.db_password)],
columns=["instance", "database", "username", "password"],
)
produced = db_mod.backup_database(
container=self.db_container,
volume_dir=volume_dir,
db_type="mariadb",
databases_df=df,
database_containers=[self.db_container],
)
self.assertTrue(produced, "backup_database did not produce a dump")
dump_path = os.path.join(volume_dir, "sql", f"{self.db_name}.backup.sql")
self.assertTrue(os.path.isfile(dump_path), f"expected dump at {dump_path}")
with open(dump_path, "r", encoding="utf-8", errors="replace") as f:
content = f.read()
self.assertIn("INSERT INTO", content)
self.assertIn("'ok'", content)
if __name__ == "__main__":
unittest.main(verbosity=2)

View File

@@ -2,6 +2,8 @@
import unittest
from .helpers import (
MARIADB_IMAGE,
MARIADB_DATA_DIR,
backup_run,
backup_path,
cleanup_docker,
@@ -56,8 +58,8 @@ class TestE2EMariaDBFull(unittest.TestCase):
"-e",
f"MARIADB_PASSWORD={cls.db_password}",
"-v",
f"{cls.db_volume}:/var/lib/mysql",
"mariadb:11",
f"{cls.db_volume}:{MARIADB_DATA_DIR}",
MARIADB_IMAGE,
]
)
@@ -97,7 +99,7 @@ class TestE2EMariaDBFull(unittest.TestCase):
compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv,
database_containers=[cls.db_container],
images_no_stop_required=["mariadb", "mysql", "alpine", "postgres"],
images_no_stop_required=[MARIADB_IMAGE],
)
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)

View File

@@ -2,6 +2,8 @@
import unittest
from .helpers import (
MARIADB_IMAGE,
MARIADB_DATA_DIR,
backup_run,
backup_path,
cleanup_docker,
@@ -55,8 +57,8 @@ class TestE2EMariaDBNoCopy(unittest.TestCase):
"-e",
f"MARIADB_PASSWORD={cls.db_password}",
"-v",
f"{cls.db_volume}:/var/lib/mysql",
"mariadb:11",
f"{cls.db_volume}:{MARIADB_DATA_DIR}",
MARIADB_IMAGE,
]
)
@@ -87,15 +89,15 @@ class TestE2EMariaDBNoCopy(unittest.TestCase):
[(cls.db_container, cls.db_name, cls.db_user, cls.db_password)],
)
# dump-only => no files
# dump-only-sql => no files
backup_run(
backups_dir=cls.backups_dir,
repo_name=cls.repo_name,
compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv,
database_containers=[cls.db_container],
images_no_stop_required=["mariadb", "mysql", "alpine", "postgres"],
dump_only=True,
images_no_stop_required=[MARIADB_IMAGE],
dump_only_sql=True,
)
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)

View File

@@ -0,0 +1,157 @@
# tests/e2e/test_e2e_postgres_empty_drop_hard.py
import unittest
from .helpers import (
POSTGRES_IMAGE,
POSTGRES_DATA_DIR,
backup_run,
cleanup_docker,
create_minimal_compose_dir,
ensure_empty_dir,
latest_version_dir,
require_docker,
run,
unique,
wait_for_postgres,
write_databases_csv,
)
# The scenario the --empty pre-clean must survive: a non-public user schema
# plus one object of every class the discovery SELECT enumerates. Restore
# --empty runs against the still-populated DB (no wipe), so the pre-clean must
# drop discourse_functions too or the dump's CREATE SCHEMA aborts the replay
# under ON_ERROR_STOP; the old public-only DROP left it and broke discourse.
SCENARIO_SQL = (
"CREATE SCHEMA discourse_functions;"
"CREATE TABLE discourse_functions.helper (id int);"
"INSERT INTO discourse_functions.helper VALUES (1);"
"CREATE TABLE public.t (id int primary key, v text);"
"INSERT INTO public.t VALUES (1, 'ok');"
"CREATE VIEW public.t_view AS SELECT * FROM public.t;"
"CREATE SEQUENCE public.s;"
"CREATE TYPE public.mood AS ENUM ('ok', 'bad');"
"CREATE FUNCTION public.f() RETURNS int LANGUAGE sql AS 'SELECT 1';"
"CREATE COLLATION public.c (locale = 'C');"
)
class TestE2EPostgresEmptyDropHard(unittest.TestCase):
@classmethod
def setUpClass(cls) -> None:
require_docker()
cls.prefix = unique("baudolo-e2e-postgres-empty-drop-hard")
cls.backups_dir = f"/tmp/{cls.prefix}/Backups"
ensure_empty_dir(cls.backups_dir)
cls.compose_dir = create_minimal_compose_dir(f"/tmp/{cls.prefix}")
cls.repo_name = cls.prefix
cls.pg_container = f"{cls.prefix}-pg"
cls.pg_volume = f"{cls.prefix}-pg-vol"
cls.containers = [cls.pg_container]
cls.volumes = [cls.pg_volume]
run(["docker", "volume", "create", cls.pg_volume])
run(
[
"docker",
"run",
"-d",
"--name",
cls.pg_container,
"-e",
"POSTGRES_PASSWORD=pgpw",
"-e",
"POSTGRES_DB=appdb",
"-e",
"POSTGRES_USER=postgres",
"-v",
f"{cls.pg_volume}:{POSTGRES_DATA_DIR}",
POSTGRES_IMAGE,
]
)
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)
run(
[
"docker",
"exec",
cls.pg_container,
"sh",
"-lc",
f'psql -U postgres -d appdb -v ON_ERROR_STOP=1 -c "{SCENARIO_SQL}"',
]
)
cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv"
write_databases_csv(
cls.databases_csv, [(cls.pg_container, "appdb", "postgres", "pgpw")]
)
backup_run(
backups_dir=cls.backups_dir,
repo_name=cls.repo_name,
compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv,
database_containers=[cls.pg_container],
images_no_stop_required=[POSTGRES_IMAGE],
)
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)
# No wipe: restore --empty must pre-clean the fully-populated DB
# (incl. the non-public schema) before replaying the dump.
run(
[
"baudolo-restore",
"postgres",
cls.pg_volume,
cls.hash,
cls.version,
"--backups-dir",
cls.backups_dir,
"--repo-name",
cls.repo_name,
"--container",
cls.pg_container,
"--db-name",
"appdb",
"--db-user",
"postgres",
"--db-password",
"pgpw",
"--empty",
]
)
@classmethod
def tearDownClass(cls) -> None:
cleanup_docker(containers=cls.containers, volumes=cls.volumes)
def _scalar(self, sql: str) -> str:
p = run(
[
"docker",
"exec",
self.pg_container,
"sh",
"-lc",
f'psql -U postgres -d appdb -t -A -c "{sql}"',
]
)
return (p.stdout or "").strip()
def test_public_data_restored(self) -> None:
self.assertEqual(self._scalar("SELECT v FROM public.t WHERE id=1;"), "ok")
def test_view_restored(self) -> None:
self.assertEqual(self._scalar("SELECT count(*) FROM public.t_view;"), "1")
def test_non_public_schema_restored(self) -> None:
self.assertEqual(
self._scalar(
"SELECT count(*) FROM pg_namespace WHERE nspname='discourse_functions';"
),
"1",
)
if __name__ == "__main__":
unittest.main()

View File

@@ -2,6 +2,8 @@
import unittest
from .helpers import (
POSTGRES_IMAGE,
POSTGRES_DATA_DIR,
backup_run,
backup_path,
cleanup_docker,
@@ -47,8 +49,8 @@ class TestE2EPostgresFull(unittest.TestCase):
"-e",
"POSTGRES_USER=postgres",
"-v",
f"{cls.pg_volume}:/var/lib/postgresql/data",
"postgres:16",
f"{cls.pg_volume}:{POSTGRES_DATA_DIR}",
POSTGRES_IMAGE,
]
)
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)
@@ -76,7 +78,7 @@ class TestE2EPostgresFull(unittest.TestCase):
compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv,
database_containers=[cls.pg_container],
images_no_stop_required=["postgres", "mariadb", "mysql", "alpine"],
images_no_stop_required=[POSTGRES_IMAGE],
)
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)

View File

@@ -2,6 +2,8 @@
import unittest
from .helpers import (
POSTGRES_IMAGE,
POSTGRES_DATA_DIR,
backup_run,
backup_path,
cleanup_docker,
@@ -46,8 +48,8 @@ class TestE2EPostgresNoCopy(unittest.TestCase):
"-e",
"POSTGRES_USER=postgres",
"-v",
f"{cls.pg_volume}:/var/lib/postgresql/data",
"postgres:16",
f"{cls.pg_volume}:{POSTGRES_DATA_DIR}",
POSTGRES_IMAGE,
]
)
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)
@@ -74,8 +76,8 @@ class TestE2EPostgresNoCopy(unittest.TestCase):
compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv,
database_containers=[cls.pg_container],
images_no_stop_required=["postgres", "mariadb", "mysql", "alpine"],
dump_only=True,
images_no_stop_required=[POSTGRES_IMAGE],
dump_only_sql=True,
)
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)

View File

@@ -0,0 +1,231 @@
import unittest
from .helpers import (
POSTGRES_IMAGE,
POSTGRES_DATA_DIR,
backup_path,
cleanup_docker,
create_minimal_compose_dir,
ensure_empty_dir,
latest_version_dir,
require_docker,
run,
unique,
wait_for_postgres,
)
class TestE2ESeedStarAndDbEntriesBackupPostgres(unittest.TestCase):
@classmethod
def setUpClass(cls) -> None:
require_docker()
cls.prefix = unique("baudolo-e2e-seed-star-and-db")
cls.backups_dir = f"/tmp/{cls.prefix}/Backups"
ensure_empty_dir(cls.backups_dir)
cls.compose_dir = create_minimal_compose_dir(f"/tmp/{cls.prefix}")
cls.repo_name = cls.prefix
# --- Volumes ---
cls.db_volume = f"{cls.prefix}-vol-db"
cls.files_volume = f"{cls.prefix}-vol-files"
cls.volumes = [cls.db_volume, cls.files_volume]
run(["docker", "volume", "create", cls.db_volume])
run(["docker", "volume", "create", cls.files_volume])
# Put a marker into the non-db volume
cls.marker = "hello-non-db-seed-star"
run(
[
"docker",
"run",
"--rm",
"-v",
f"{cls.files_volume}:/data",
"alpine:3.20",
"sh",
"-lc",
f"echo '{cls.marker}' > /data/hello.txt",
]
)
# --- Start Postgres container using the DB volume ---
cls.pg_container = f"{cls.prefix}-pg"
cls.containers = [cls.pg_container]
cls.pg_password = "postgres"
cls.pg_user = "postgres"
run(
[
"docker",
"run",
"-d",
"--name",
cls.pg_container,
"-e",
f"POSTGRES_PASSWORD={cls.pg_password}",
"-v",
f"{cls.db_volume}:{POSTGRES_DATA_DIR}",
POSTGRES_IMAGE,
]
)
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)
# Create two DBs and deterministic content, so pg_dumpall is meaningful
cls.pg_db1 = "testdb1"
cls.pg_db2 = "testdb2"
run(
[
"docker",
"exec",
cls.pg_container,
"sh",
"-lc",
(
f'psql -U {cls.pg_user} -c "CREATE DATABASE {cls.pg_db1};" || true; '
f'psql -U {cls.pg_user} -c "CREATE DATABASE {cls.pg_db2};" || true; '
),
],
check=True,
)
run(
[
"docker",
"exec",
cls.pg_container,
"sh",
"-lc",
(
f"psql -U {cls.pg_user} -d {cls.pg_db1} -c "
'"CREATE TABLE IF NOT EXISTS t (id INT PRIMARY KEY, v TEXT);'
"INSERT INTO t(id,v) VALUES (1,'hello-db1') "
'ON CONFLICT (id) DO UPDATE SET v=EXCLUDED.v;"'
),
],
check=True,
)
run(
[
"docker",
"exec",
cls.pg_container,
"sh",
"-lc",
(
f"psql -U {cls.pg_user} -d {cls.pg_db2} -c "
'"CREATE TABLE IF NOT EXISTS t (id INT PRIMARY KEY, v TEXT);'
"INSERT INTO t(id,v) VALUES (1,'hello-db2') "
'ON CONFLICT (id) DO UPDATE SET v=EXCLUDED.v;"'
),
],
check=True,
)
# --- Seed databases.csv using CLI (star + concrete db) ---
cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv"
# IMPORTANT: because we pass --database-containers <container>,
# get_instance() will use the container name as instance key.
instance = cls.pg_container
# Seed star entry (pg_dumpall)
run(
[
"baudolo-seed",
cls.databases_csv,
instance,
"*",
cls.pg_user,
cls.pg_password,
]
)
# Seed concrete DB entry (pg_dump)
run(
[
"baudolo-seed",
cls.databases_csv,
instance,
cls.pg_db1,
cls.pg_user,
cls.pg_password,
]
)
# --- Run baudolo with dump-only-sql ---
cmd = [
"baudolo",
"--compose-dir",
cls.compose_dir,
"--databases-csv",
cls.databases_csv,
"--database-containers",
cls.pg_container,
"--images-no-stop-required",
POSTGRES_IMAGE,
"--dump-only-sql",
"--backups-dir",
cls.backups_dir,
"--repo-name",
cls.repo_name,
]
cp = run(cmd, capture=True, check=True)
cls.stdout = cp.stdout or ""
cls.stderr = cp.stderr or ""
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)
@classmethod
def tearDownClass(cls) -> None:
cleanup_docker(containers=cls.containers, volumes=cls.volumes)
def test_db_volume_has_cluster_dump_and_concrete_db_dump_and_no_files(self) -> None:
base = backup_path(
self.backups_dir, self.repo_name, self.version, self.db_volume
)
sql_dir = base / "sql"
files_dir = base / "files"
self.assertTrue(sql_dir.exists(), f"Expected sql dir at: {sql_dir}")
self.assertFalse(
files_dir.exists(),
f"Did not expect files dir for DB volume when dump-only-sql succeeded: {files_dir}",
)
# Cluster dump file produced by '*' entry
cluster = sql_dir / f"{self.pg_container}.cluster.backup.sql"
self.assertTrue(cluster.is_file(), f"Expected cluster dump file at: {cluster}")
# Concrete DB dump produced by normal entry
db1 = sql_dir / f"{self.pg_db1}.backup.sql"
self.assertTrue(db1.is_file(), f"Expected db dump file at: {db1}")
# Basic sanity: cluster dump usually contains CREATE DATABASE statements
txt = cluster.read_text(encoding="utf-8", errors="ignore")
self.assertIn(
"CREATE DATABASE",
txt,
"Expected cluster dump to contain CREATE DATABASE statements",
)
def test_non_db_volume_still_has_files_backup(self) -> None:
base = backup_path(
self.backups_dir, self.repo_name, self.version, self.files_volume
)
files_dir = base / "files"
self.assertTrue(
files_dir.exists(), f"Expected files dir for non-DB volume at: {files_dir}"
)
marker = files_dir / "hello.txt"
self.assertTrue(marker.is_file(), f"Expected marker file at: {marker}")
self.assertEqual(
marker.read_text(encoding="utf-8").strip(),
self.marker,
)

View File

@@ -0,0 +1,196 @@
# tests/e2e/test_e2e_swarm_task_skip.py
#
# Reproduces the swarm flake fixed on this branch: baudolo used to stop a
# swarm task container around the volume file backup because its image was
# not whitelisted; the orchestrator immediately replaced the stopped task and
# the later `docker start` failed on the detached overlay network, killing
# the backup run. With the fix the task container is skipped (backed up hot):
# the backup succeeds, the very same container instance keeps running, and
# the service never has to replace a task.
import time
import unittest
from .helpers import (
backup_path,
backup_run,
create_minimal_compose_dir,
ensure_empty_dir,
latest_version_dir,
require_docker,
run,
unique,
write_databases_csv,
)
def _swarm_state() -> str:
return run(
["docker", "info", "--format", "{{.Swarm.LocalNodeState}}"]
).stdout.strip()
def _task_container_id(service: str, timeout_s: int = 60) -> str:
deadline = time.time() + timeout_s
while time.time() < deadline:
out = run(
[
"docker",
"ps",
"--filter",
f"label=com.docker.swarm.service.name={service}",
"--format",
"{{.ID}}",
]
).stdout.strip()
if out:
return out.splitlines()[0]
time.sleep(2)
raise RuntimeError(f"No running task container for service {service}")
def _started_at(container_id: str) -> str:
return run(
["docker", "inspect", "--format", "{{.State.StartedAt}}", container_id]
).stdout.strip()
class TestE2ESwarmTaskSkip(unittest.TestCase):
@classmethod
def setUpClass(cls) -> None:
require_docker()
cls.prefix = unique("baudolo-e2e-swarm-skip")
cls.backups_dir = f"/tmp/{cls.prefix}/Backups"
ensure_empty_dir(cls.backups_dir)
cls.compose_dir = create_minimal_compose_dir(f"/tmp/{cls.prefix}")
cls.repo_name = cls.prefix
cls.swarm_initted = False
if _swarm_state() != "active":
run(["docker", "swarm", "init", "--advertise-addr", "127.0.0.1"])
cls.swarm_initted = True
cls.volume = f"{cls.prefix}-vol"
cls.service = f"{cls.prefix}-svc"
cls.volumes = [cls.volume]
run(["docker", "volume", "create", cls.volume])
run(
[
"docker",
"run",
"--rm",
"-v",
f"{cls.volume}:/data",
"alpine:3.20",
"sh",
"-lc",
"echo 'swarm-payload' > /data/payload.txt",
]
)
run(
[
"docker",
"service",
"create",
"--name",
cls.service,
"--replicas",
"1",
"--mount",
f"type=volume,source={cls.volume},target=/data",
"alpine:3.20",
"sleep",
"3600",
]
)
cls.task_cid = _task_container_id(cls.service)
cls.task_started_at = _started_at(cls.task_cid)
cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv"
write_databases_csv(cls.databases_csv, [])
# Whitelist that matches nothing: on main this forces a stop of every
# container at the volume, i.e. exactly the flake; on this branch the
# swarm task must be skipped instead. (An empty list would leave the
# --images-no-stop-required flag without arguments and argparse-fail.)
backup_run(
backups_dir=cls.backups_dir,
repo_name=cls.repo_name,
compose_dir=cls.compose_dir,
databases_csv=cls.databases_csv,
database_containers=["dummy-db"],
images_no_stop_required=["image-that-matches-nothing"],
)
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)
@classmethod
def tearDownClass(cls) -> None:
run(["docker", "service", "rm", cls.service], check=False)
deadline = time.time() + 30
while time.time() < deadline:
out = run(
[
"docker",
"ps",
"-aq",
"--filter",
f"label=com.docker.swarm.service.name={cls.service}",
],
check=False,
).stdout.strip()
if not out:
break
time.sleep(2)
for v in cls.volumes:
run(["docker", "volume", "rm", "-f", v], check=False)
if cls.swarm_initted:
run(["docker", "swarm", "leave", "--force"], check=False)
def test_volume_backed_up_hot(self) -> None:
p = (
backup_path(
self.backups_dir,
self.repo_name,
self.version,
self.volume,
)
/ "files"
/ "payload.txt"
)
self.assertTrue(p.is_file(), f"Expected backed up file at: {p}")
def test_task_container_never_stopped(self) -> None:
out = run(
["docker", "ps", "-q", "--no-trunc", "--filter", f"id={self.task_cid}"]
).stdout.strip()
self.assertTrue(
out.startswith(self.task_cid) or self.task_cid.startswith(out.strip()[:12]),
f"Task container {self.task_cid} is no longer running",
)
self.assertEqual(
self.task_started_at,
_started_at(self.task_cid),
"Task container was restarted during the backup",
)
def test_service_never_replaced_the_task(self) -> None:
states = run(
[
"docker",
"service",
"ps",
self.service,
"--format",
"{{.DesiredState}} {{.CurrentState}}",
]
).stdout.strip()
lines = [line for line in states.splitlines() if line.strip()]
self.assertEqual(
len(lines), 1, f"Service task history shows replacements:\n{states}"
)
self.assertIn("Running", lines[0])
if __name__ == "__main__":
unittest.main()

View File

@@ -7,9 +7,47 @@ from pathlib import Path
def run_seed(
csv_path: Path, instance: str, database: str, username: str, password: str = ""
csv_path: Path, instance: str, database: str, username: str, password: str
) -> subprocess.CompletedProcess:
# Run the real CLI module (integration-style).
"""
Run the real CLI module (E2E-style) using subprocess.
Seed contract (current):
- database must be "*" or a valid name (non-empty, matches allowed charset)
- password is required
- entry is keyed by (instance, database); username/password get updated
"""
cp = subprocess.run(
[
sys.executable,
"-m",
"baudolo.seed",
str(csv_path),
instance,
database,
username,
password,
],
text=True,
capture_output=True,
check=False,
)
if cp.returncode != 0:
raise AssertionError(
"seed command failed unexpectedly.\n"
f"returncode: {cp.returncode}\n"
f"stdout:\n{cp.stdout}\n"
f"stderr:\n{cp.stderr}\n"
)
return cp
def run_seed_expect_fail(
csv_path: Path, instance: str, database: str, username: str, password: str
) -> subprocess.CompletedProcess:
"""
Same as run_seed, but expects non-zero exit. Returns CompletedProcess for inspection.
"""
return subprocess.run(
[
sys.executable,
@@ -23,7 +61,7 @@ def run_seed(
],
text=True,
capture_output=True,
check=True,
check=False,
)
@@ -33,6 +71,10 @@ def read_csv_semicolon(path: Path) -> list[dict]:
return list(reader)
def read_text(path: Path) -> str:
return path.read_text(encoding="utf-8")
class TestSeedIntegration(unittest.TestCase):
def test_creates_file_and_adds_entry_when_missing(self) -> None:
with tempfile.TemporaryDirectory() as td:
@@ -41,7 +83,7 @@ class TestSeedIntegration(unittest.TestCase):
cp = run_seed(p, "docker.test", "appdb", "alice", "secret")
self.assertEqual(cp.returncode, 0, cp.stderr)
self.assertEqual(cp.returncode, 0)
self.assertTrue(p.exists())
rows = read_csv_semicolon(p)
@@ -51,40 +93,121 @@ class TestSeedIntegration(unittest.TestCase):
self.assertEqual(rows[0]["username"], "alice")
self.assertEqual(rows[0]["password"], "secret")
def test_replaces_existing_entry_same_keys(self) -> None:
def test_replaces_existing_entry_same_instance_and_database_updates_username_and_password(
self,
) -> None:
"""
Replacement semantics:
- Key is (instance, database)
- username/password are updated in-place
"""
with tempfile.TemporaryDirectory() as td:
p = Path(td) / "databases.csv"
# First add
run_seed(p, "docker.test", "appdb", "alice", "oldpw")
rows = read_csv_semicolon(p)
self.assertEqual(len(rows), 1)
self.assertEqual(rows[0]["username"], "alice")
self.assertEqual(rows[0]["password"], "oldpw")
# Replace (same instance+database+username)
run_seed(p, "docker.test", "appdb", "alice", "newpw")
run_seed(p, "docker.test", "appdb", "bob", "newpw")
rows = read_csv_semicolon(p)
self.assertEqual(len(rows), 1, "Expected replacement, not a duplicate row")
self.assertEqual(rows[0]["instance"], "docker.test")
self.assertEqual(rows[0]["database"], "appdb")
self.assertEqual(rows[0]["username"], "alice")
self.assertEqual(rows[0]["username"], "bob")
self.assertEqual(rows[0]["password"], "newpw")
def test_database_empty_string_matches_existing_empty_database(self) -> None:
def test_allows_star_database_for_dump_all(self) -> None:
with tempfile.TemporaryDirectory() as td:
p = Path(td) / "databases.csv"
# Add with empty database
run_seed(p, "docker.test", "", "alice", "pw1")
cp = run_seed(p, "bigbluebutton", "*", "postgres", "pw")
self.assertEqual(cp.returncode, 0)
rows = read_csv_semicolon(p)
self.assertEqual(len(rows), 1)
self.assertEqual(rows[0]["database"], "")
self.assertEqual(rows[0]["instance"], "bigbluebutton")
self.assertEqual(rows[0]["database"], "*")
self.assertEqual(rows[0]["username"], "postgres")
self.assertEqual(rows[0]["password"], "pw")
def test_replaces_existing_star_entry(self) -> None:
with tempfile.TemporaryDirectory() as td:
p = Path(td) / "databases.csv"
run_seed(p, "bigbluebutton", "*", "postgres", "pw1")
run_seed(p, "bigbluebutton", "*", "postgres", "pw2")
# Replace with empty database again
run_seed(p, "docker.test", "", "alice", "pw2")
rows = read_csv_semicolon(p)
self.assertEqual(len(rows), 1)
self.assertEqual(rows[0]["database"], "")
self.assertEqual(rows[0]["database"], "*")
self.assertEqual(rows[0]["password"], "pw2")
def test_rejects_empty_database_value(self) -> None:
with tempfile.TemporaryDirectory() as td:
p = Path(td) / "databases.csv"
cp = run_seed_expect_fail(p, "docker.test", "", "alice", "pw")
self.assertNotEqual(cp.returncode, 0)
combined = ((cp.stdout or "") + "\n" + (cp.stderr or "")).lower()
self.assertIn("error:", combined)
self.assertIn("database", combined)
self.assertIn("not empty", combined)
self.assertFalse(p.exists(), "Should not create file on invalid input")
def test_rejects_invalid_database_name_characters(self) -> None:
with tempfile.TemporaryDirectory() as td:
p = Path(td) / "databases.csv"
cp = run_seed_expect_fail(p, "docker.test", "app db", "alice", "pw")
self.assertNotEqual(cp.returncode, 0)
combined = ((cp.stdout or "") + "\n" + (cp.stderr or "")).lower()
self.assertIn("error:", combined)
self.assertIn("invalid database name", combined)
self.assertFalse(p.exists(), "Should not create file on invalid input")
def test_rejects_nan_database_name(self) -> None:
with tempfile.TemporaryDirectory() as td:
p = Path(td) / "databases.csv"
cp = run_seed_expect_fail(p, "docker.test", "nan", "alice", "pw")
self.assertNotEqual(cp.returncode, 0)
combined = ((cp.stdout or "") + "\n" + (cp.stderr or "")).lower()
self.assertIn("error:", combined)
self.assertIn("must not be 'nan'", combined)
self.assertFalse(p.exists(), "Should not create file on invalid input")
def test_accepts_hyphen_and_underscore_database_names(self) -> None:
with tempfile.TemporaryDirectory() as td:
p = Path(td) / "databases.csv"
run_seed(p, "docker.test", "my_db-1", "alice", "pw")
rows = read_csv_semicolon(p)
self.assertEqual(len(rows), 1)
self.assertEqual(rows[0]["database"], "my_db-1")
def test_file_is_semicolon_delimited_and_has_header(self) -> None:
with tempfile.TemporaryDirectory() as td:
p = Path(td) / "databases.csv"
run_seed(p, "docker.test", "appdb", "alice", "pw")
txt = read_text(p)
self.assertTrue(
txt.startswith("instance;database;username;password"),
f"Unexpected header / delimiter in file:\n{txt}",
)
self.assertIn(";", txt)
if __name__ == "__main__":
unittest.main()

View File

View File

@@ -0,0 +1,77 @@
import io
import os
import tempfile
import unittest
from contextlib import redirect_stderr
import pandas as pd
# Adjust if your package name/import path differs.
from baudolo.backup.app import _load_databases_df
EXPECTED_COLUMNS = ["instance", "database", "username", "password"]
class TestLoadDatabasesDf(unittest.TestCase):
def test_missing_csv_is_handled_with_warning_and_empty_df(self) -> None:
with tempfile.TemporaryDirectory() as td:
missing_path = os.path.join(td, "does-not-exist.csv")
buf = io.StringIO()
with redirect_stderr(buf):
df = _load_databases_df(missing_path)
stderr = buf.getvalue()
self.assertIn("WARNING:", stderr)
self.assertIn("databases.csv not found", stderr)
self.assertIsInstance(df, pd.DataFrame)
self.assertListEqual(list(df.columns), EXPECTED_COLUMNS)
self.assertTrue(df.empty)
def test_empty_csv_is_handled_with_warning_and_empty_df(self) -> None:
with tempfile.TemporaryDirectory() as td:
empty_path = os.path.join(td, "databases.csv")
# Create an empty file (0 bytes)
with open(empty_path, "w", encoding="utf-8") as f:
f.write("")
buf = io.StringIO()
with redirect_stderr(buf):
df = _load_databases_df(empty_path)
stderr = buf.getvalue()
self.assertIn("WARNING:", stderr)
self.assertIn("exists but is empty", stderr)
self.assertIsInstance(df, pd.DataFrame)
self.assertListEqual(list(df.columns), EXPECTED_COLUMNS)
self.assertTrue(df.empty)
def test_valid_csv_loads_without_warning(self) -> None:
with tempfile.TemporaryDirectory() as td:
csv_path = os.path.join(td, "databases.csv")
content = "instance;database;username;password\nmyapp;*;dbuser;secret\n"
with open(csv_path, "w", encoding="utf-8") as f:
f.write(content)
buf = io.StringIO()
with redirect_stderr(buf):
df = _load_databases_df(csv_path)
stderr = buf.getvalue()
self.assertEqual(stderr, "") # no warning expected
self.assertIsInstance(df, pd.DataFrame)
self.assertListEqual(list(df.columns), EXPECTED_COLUMNS)
self.assertEqual(len(df), 1)
self.assertEqual(df.loc[0, "instance"], "myapp")
self.assertEqual(df.loc[0, "database"], "*")
self.assertEqual(df.loc[0, "username"], "dbuser")
self.assertEqual(df.loc[0, "password"], "secret")
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,311 @@
from __future__ import annotations
import shutil
import tempfile
import unittest
from pathlib import Path
from typing import List
from unittest.mock import patch
def _touch(p: Path) -> None:
p.parent.mkdir(parents=True, exist_ok=True)
# If the path already exists as a directory (e.g. ".env" created by ".env/env"),
# remove it so we can create a file with the same name.
if p.exists() and p.is_dir():
shutil.rmtree(p)
p.write_text("x", encoding="utf-8")
def _setup_compose_dir(
tmp_path: Path,
name: str = "mailu",
*,
compose_name: str = "docker-compose.yml",
with_override: bool = False,
with_ca_override: bool = False,
env_layout: str | None = None, # None | ".env" | ".env/env"
) -> Path:
d = tmp_path / name
d.mkdir(parents=True, exist_ok=True)
_touch(d / compose_name)
if with_override:
_touch(d / "docker-compose.override.yml")
if with_ca_override:
_touch(d / "docker-compose.ca.override.yml")
if env_layout == ".env":
_touch(d / ".env")
elif env_layout == ".env/env":
_touch(d / ".env" / "env")
return d
class TestCompose(unittest.TestCase):
@classmethod
def setUpClass(cls) -> None:
from baudolo.backup import compose as mod
cls.compose_mod = mod
def test_find_compose_file_supports_all_valid_names_case_insensitive(self) -> None:
with tempfile.TemporaryDirectory() as td:
tmp_path = Path(td)
variants = [
"compose.yml",
"compose.yaml",
"docker-compose.yml",
"docker-compose.yaml",
"docker-compose.yAml",
]
for i, name in enumerate(variants):
d = _setup_compose_dir(
tmp_path,
name=f"project{i}",
compose_name=name,
)
found = self.compose_mod._find_compose_file(str(d))
self.assertIsNotNone(found)
self.assertEqual(found.name, name)
def test_find_compose_file_returns_none_when_missing(self) -> None:
with tempfile.TemporaryDirectory() as td:
tmp_path = Path(td)
d = tmp_path / "empty"
d.mkdir(parents=True, exist_ok=True)
found = self.compose_mod._find_compose_file(str(d))
self.assertIsNone(found)
def test_build_cmd_uses_wrapper_when_present(self) -> None:
with tempfile.TemporaryDirectory() as td:
tmp_path = Path(td)
d = _setup_compose_dir(
tmp_path,
with_override=True,
with_ca_override=True,
env_layout=".env",
)
def fake_which(name: str):
if name == "compose":
return "/usr/local/bin/compose"
return None
with patch.object(self.compose_mod.shutil, "which", fake_which):
cmd = self.compose_mod._build_compose_cmd(str(d), ["up", "-d"])
self.assertEqual(
cmd,
[
"/usr/local/bin/compose",
"--chdir",
str(d.resolve()),
"--",
"up",
"-d",
],
)
def test_build_cmd_fallback_uses_plain_docker_compose_chdir(self) -> None:
with tempfile.TemporaryDirectory() as td:
tmp_path = Path(td)
d = _setup_compose_dir(
tmp_path,
with_override=True,
with_ca_override=True,
env_layout=".env",
)
def fake_which(name: str):
if name == "compose":
return None
if name == "docker":
return "/usr/bin/docker"
return None
with patch.object(self.compose_mod.shutil, "which", fake_which):
cmd = self.compose_mod._build_compose_cmd(
str(d), ["up", "-d", "--force-recreate"]
)
expected: List[str] = [
"/usr/bin/docker",
"compose",
"--chdir",
str(d.resolve()),
"up",
"-d",
"--force-recreate",
]
self.assertEqual(cmd, expected)
def test_hard_restart_calls_run_twice_with_correct_cmds_wrapper(self) -> None:
with tempfile.TemporaryDirectory() as td:
tmp_path = Path(td)
d = _setup_compose_dir(tmp_path, name="mailu", env_layout=".env")
def fake_which(name: str):
if name == "compose":
return "/usr/local/bin/compose"
return None
with patch.object(self.compose_mod.shutil, "which", fake_which):
calls = []
def fake_run(cmd, check: bool):
calls.append((cmd, check))
return 0
with patch.object(self.compose_mod.subprocess, "run", fake_run):
self.compose_mod.hard_restart_docker_services(str(d))
self.assertEqual(
calls,
[
(
[
"/usr/local/bin/compose",
"--chdir",
str(d.resolve()),
"--",
"down",
],
True,
),
(
[
"/usr/local/bin/compose",
"--chdir",
str(d.resolve()),
"--",
"up",
"-d",
],
True,
),
],
)
def test_hard_restart_calls_run_twice_with_correct_cmds_fallback(self) -> None:
with tempfile.TemporaryDirectory() as td:
tmp_path = Path(td)
d = _setup_compose_dir(
tmp_path,
name="mailu",
with_override=True,
with_ca_override=True,
env_layout=".env/env",
)
def fake_which(name: str):
if name == "compose":
return None
if name == "docker":
return "/usr/bin/docker"
return None
with patch.object(self.compose_mod.shutil, "which", fake_which):
calls = []
def fake_run(cmd, check: bool):
calls.append((cmd, check))
return 0
with patch.object(self.compose_mod.subprocess, "run", fake_run):
self.compose_mod.hard_restart_docker_services(str(d))
self.assertEqual(
calls,
[
(
[
"/usr/bin/docker",
"compose",
"--chdir",
str(d.resolve()),
"down",
],
True,
),
(
[
"/usr/bin/docker",
"compose",
"--chdir",
str(d.resolve()),
"up",
"-d",
],
True,
),
],
)
class HardRestartArgTests(unittest.TestCase):
"""The hard-restart list defaults to empty (no compose down/up); callers
opt in per dir, e.g. compose hosts pass 'mailu' while swarm hosts, where
the dir is a stack whose overlay network collides with compose up, pass
nothing."""
def _parse(self, extra: List[str]):
import sys
from baudolo.backup import cli
argv = [
"baudolo",
"--compose-dir",
"/tmp",
"--backups-dir",
"/tmp/backup",
"--database-containers",
"postgres",
"--images-no-stop-required",
"redis",
*extra,
]
with patch.object(sys, "argv", argv):
return cli.parse_args()
def test_default_is_empty(self) -> None:
args = self._parse([])
self.assertEqual(args.hard_restart_projects, [])
def test_empty_flag_stays_empty(self) -> None:
args = self._parse(["--hard-restart-projects"])
self.assertEqual(args.hard_restart_projects, [])
def test_explicit_names_preserved(self) -> None:
args = self._parse(["--hard-restart-projects", "mailu", "foo"])
self.assertEqual(args.hard_restart_projects, ["mailu", "foo"])
def test_backups_dir_is_required(self) -> None:
import sys
from baudolo.backup import cli
argv = [
"baudolo",
"--compose-dir",
"/tmp",
"--database-containers",
"postgres",
"--images-no-stop-required",
"redis",
]
with patch.object(sys, "argv", argv), self.assertRaises(SystemExit):
cli.parse_args()
if __name__ == "__main__":
unittest.main(verbosity=2)

View File

@@ -0,0 +1,70 @@
import tempfile
import unittest
from unittest.mock import patch
import pandas
from baudolo.backup import db as db_mod
def _df(rows):
return pandas.DataFrame(
rows, columns=["instance", "database", "username", "password"]
)
def _capture_commands(*, db_type, rows, container):
captured = []
def _capture(cmd):
captured.append(cmd)
return []
with tempfile.TemporaryDirectory() as td:
with patch.object(db_mod, "execute_shell_command", side_effect=_capture):
db_mod.backup_database(
container=container,
volume_dir=td,
db_type=db_type,
databases_df=_df(rows),
database_containers=[container],
)
return captured
class TestMariaDBDumpUsesTCP(unittest.TestCase):
# Regression guard for 'Access denied for user <user>@localhost' when only
# '<user>'@'%' is granted: the in-container mariadb-dump MUST force TCP so
# the connection is auth-matched against '%' instead of socket->localhost.
def test_mariadb_dump_forces_tcp_loopback(self):
captured = _capture_commands(
db_type="mariadb",
rows=[("mariadb", "appdb", "appuser", "s3cret")],
container="mariadb",
)
dump_cmds = [c for c in captured if "mariadb-dump" in c]
self.assertEqual(
len(dump_cmds), 1, f"expected one dump command, got: {captured}"
)
cmd = dump_cmds[0]
self.assertIn("-h 127.0.0.1", cmd)
self.assertIn("--protocol=tcp", cmd)
self.assertIn("-u appuser", cmd)
self.assertIn("-ps3cret", cmd)
self.assertIn(" appdb", cmd)
def test_postgres_dump_unaffected(self):
captured = _capture_commands(
db_type="postgres",
rows=[("pg", "appdb", "appuser", "s3cret")],
container="pg",
)
dump_cmds = [c for c in captured if "pg_dump" in c and "pg_dumpall" not in c]
self.assertEqual(len(dump_cmds), 1)
self.assertNotIn("--protocol=tcp", dump_cmds[0])
if __name__ == "__main__":
unittest.main(verbosity=2)

View File

@@ -0,0 +1,53 @@
import unittest
from unittest.mock import patch
from baudolo.backup import docker as docker_mod
from baudolo.backup.shell import BackupException
class TestIsSwarmTask(unittest.TestCase):
@patch.object(docker_mod, "execute_shell_command", return_value=["task-id-123"])
def test_true_when_task_label_present(self, _mock) -> None:
self.assertTrue(docker_mod.is_swarm_task("c1"))
@patch.object(docker_mod, "execute_shell_command", return_value=[""])
def test_false_when_label_empty(self, _mock) -> None:
self.assertFalse(docker_mod.is_swarm_task("c1"))
@patch.object(docker_mod, "execute_shell_command", return_value=[])
def test_false_when_no_output(self, _mock) -> None:
self.assertFalse(docker_mod.is_swarm_task("c1"))
@patch.object(
docker_mod,
"execute_shell_command",
side_effect=[BackupException("gone"), []],
)
def test_vanished_container_counts_as_not_stoppable(self, _mock) -> None:
# A container removed between listing and inspect must not abort the
# whole backup run; treating it as a swarm task keeps it out of every
# stop/start and image-inspect path.
self.assertTrue(docker_mod.is_swarm_task("gone-container"))
@patch.object(
docker_mod,
"execute_shell_command",
side_effect=[BackupException("daemon hiccup"), ["still-here"]],
)
def test_inspect_failure_on_existing_container_still_fails(self, _mock) -> None:
# If the container still exists, an inspect failure must keep failing
# the run: silently skipping the stop would back up a hot volume and
# report green without the stop guarantee.
with self.assertRaises(BackupException):
docker_mod.is_swarm_task("still-here")
class TestFilterStoppable(unittest.TestCase):
@patch.object(docker_mod, "is_swarm_task", side_effect=[False, True, False])
def test_mixed_list_keeps_order_and_drops_tasks(self, _mock) -> None:
result = docker_mod.filter_stoppable(["plain-1", "swarm-task", "plain-2"])
self.assertEqual(result, ["plain-1", "plain-2"])
if __name__ == "__main__":
unittest.main()

View File

View File

@@ -0,0 +1,44 @@
import tempfile
import unittest
from unittest.mock import MagicMock, patch
from baudolo.restore.db import mariadb as mariadb_mod
class TestMariadbEmptyDrop(unittest.TestCase):
def test_drops_run_in_one_session_with_fk_checks_off(self) -> None:
calls = []
def _capture(container, argv, **kwargs):
calls.append(argv)
result = MagicMock()
result.stdout = b"users\nfetches\n"
return result
with tempfile.NamedTemporaryFile(suffix=".sql") as sql:
sql.write(b"CREATE TABLE t (id int);\n")
sql.flush()
with (
patch.object(mariadb_mod, "docker_exec", side_effect=_capture),
patch.object(mariadb_mod, "_pick_client", return_value="mariadb"),
):
mariadb_mod.restore_mariadb_sql(
container="db",
db_name="mailu",
user="mailu",
password="pw",
sql_path=sql.name,
empty=True,
)
drop_calls = [argv for argv in calls if any("DROP TABLE" in a for a in argv)]
self.assertEqual(len(drop_calls), 1, f"expected ONE drop session: {calls}")
drop_sql = drop_calls[0][-1]
self.assertTrue(drop_sql.startswith("SET FOREIGN_KEY_CHECKS=0; "))
self.assertIn("DROP TABLE IF EXISTS `mailu`.`users`;", drop_sql)
self.assertIn("DROP TABLE IF EXISTS `mailu`.`fetches`;", drop_sql)
self.assertTrue(drop_sql.endswith("SET FOREIGN_KEY_CHECKS=1;"))
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,43 @@
"""The --empty pre-clean drop must cover every object class a dump can
re-CREATE ahead of tables; a missed class aborts the replay under
ON_ERROR_STOP (OpenProject's ICU collation public.versions_name did)."""
import tempfile
import unittest
from pathlib import Path
from unittest import mock
from baudolo.restore.db import postgres
class TestPostgresEmptyDrop(unittest.TestCase):
def _drop_sql(self) -> str:
with tempfile.NamedTemporaryFile(suffix=".sql") as fh:
Path(fh.name).write_bytes(b"SELECT 1;\n")
with mock.patch.object(postgres, "docker_exec") as run:
postgres.restore_postgres_sql(
container="c",
db_name="db",
user="u",
password="p",
sql_path=fh.name,
empty=True,
)
first_call = run.call_args_list[0]
return first_call.kwargs["stdin"].decode()
def test_drop_covers_collations(self) -> None:
sql = self._drop_sql()
self.assertIn("pg_collation", sql)
self.assertIn("'COLLATION' AS type", sql)
self.assertIn("pg_get_userbyid(col.collowner) = current_user", sql)
def test_drop_still_covers_the_other_classes(self) -> None:
sql = self._drop_sql()
for marker in ("pg_class", "pg_proc", "'SEQUENCE' AS type", "'TYPE' AS type"):
self.assertIn(marker, sql)
self.assertIn("DROP %s IF EXISTS public.%I CASCADE", sql)
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,63 @@
import unittest
from baudolo.restore.db.postgres import filter_superuser_only_lines
def _filter(raw: bytes) -> bytes:
return b"".join(filter_superuser_only_lines(raw.splitlines(keepends=True)))
class TestFilterSuperuserOnlyLines(unittest.TestCase):
def test_drops_superuser_only_statements(self) -> None:
raw = (
b"CREATE TABLE t (id int);\n"
b"COMMENT ON EXTENSION pg_trgm IS 'trigram';\n"
b"ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO x;\n"
b"INSERT INTO t VALUES (1);\n"
)
self.assertEqual(
_filter(raw),
b"CREATE TABLE t (id int);\nINSERT INTO t VALUES (1);\n",
)
def test_copy_data_rows_are_never_filtered(self) -> None:
raw = (
b"COPY public.snippets (body) FROM stdin;\n"
b"COMMENT ON EXTENSION looks like sql but is data\n"
b"ALTER DEFAULT PRIVILEGES stored as text\n"
b"\\.\n"
b"ALTER DEFAULT PRIVILEGES IN SCHEMA public REVOKE ALL ON TABLES FROM y;\n"
)
self.assertEqual(
_filter(raw),
b"COPY public.snippets (body) FROM stdin;\n"
b"COMMENT ON EXTENSION looks like sql but is data\n"
b"ALTER DEFAULT PRIVILEGES stored as text\n"
b"\\.\n",
)
def test_consecutive_copy_blocks_keep_state(self) -> None:
raw = (
b"COPY public.a (v) FROM stdin;\n"
b"row-a\n"
b"\\.\n"
b"COMMENT ON EXTENSION dropme IS 'x';\n"
b"COPY public.b (v) FROM stdin;\n"
b"COMMENT ON EXTENSION kept-as-data\n"
b"\\.\n"
)
out = _filter(raw)
self.assertNotIn(b"dropme", out)
self.assertIn(b"COMMENT ON EXTENSION kept-as-data\n", out)
def test_everything_else_passes_through_verbatim(self) -> None:
raw = (
b"SET statement_timeout = 0;\n"
b"CREATE EXTENSION IF NOT EXISTS pg_trgm;\n"
b"GRANT ALL ON SCHEMA public TO app;\n"
)
self.assertEqual(_filter(raw), raw)
if __name__ == "__main__":
unittest.main()

View File

View File

@@ -0,0 +1,215 @@
from __future__ import annotations
import unittest
from unittest.mock import MagicMock, patch
import pandas as pd
from pandas.errors import EmptyDataError
import baudolo.seed.__main__ as seed_main
class TestSeedMain(unittest.TestCase):
@patch("baudolo.seed.__main__.pd.DataFrame")
def test_empty_df_creates_expected_columns(self, df_ctor: MagicMock) -> None:
seed_main._empty_df()
df_ctor.assert_called_once_with(
columns=["instance", "database", "username", "password"]
)
def test_validate_database_value_rejects_empty(self) -> None:
with self.assertRaises(ValueError):
seed_main._validate_database_value("", instance="x")
def test_validate_database_value_accepts_star(self) -> None:
self.assertEqual(seed_main._validate_database_value("*", instance="x"), "*")
def test_validate_database_value_rejects_nan(self) -> None:
with self.assertRaises(ValueError):
seed_main._validate_database_value("nan", instance="x")
def test_validate_database_value_rejects_invalid_name(self) -> None:
with self.assertRaises(ValueError):
seed_main._validate_database_value("bad name", instance="x")
def _mock_df_mask_any(self, *, any_value: bool) -> MagicMock:
"""
Build a DataFrame-like mock such that:
mask = (df["instance"] == instance) & (df["database"] == database)
mask.any() returns any_value
"""
df = MagicMock(spec=pd.DataFrame)
left = MagicMock()
right = MagicMock()
mask = MagicMock()
mask.any.return_value = any_value
# (left & right) => mask
left.__and__.return_value = mask
# df["instance"] / df["database"] => return objects whose == produces left/right
col = MagicMock()
col.__eq__.side_effect = [left, right]
df.__getitem__.return_value = col
return df
@patch("baudolo.seed.__main__.os.path.exists", return_value=False)
@patch("baudolo.seed.__main__.pd.read_csv")
@patch("baudolo.seed.__main__._empty_df")
@patch("baudolo.seed.__main__.pd.concat")
def test_check_and_add_entry_file_missing_adds_entry(
self,
concat: MagicMock,
empty_df: MagicMock,
read_csv: MagicMock,
exists: MagicMock,
) -> None:
df_existing = self._mock_df_mask_any(any_value=False)
empty_df.return_value = df_existing
df_out = MagicMock(spec=pd.DataFrame)
concat.return_value = df_out
seed_main.check_and_add_entry(
file_path="/tmp/databases.csv",
instance="inst",
database="db",
username="user",
password="pass",
)
read_csv.assert_not_called()
empty_df.assert_called_once()
concat.assert_called_once()
df_out.to_csv.assert_called_once_with(
"/tmp/databases.csv", sep=";", index=False
)
@patch("baudolo.seed.__main__.os.path.exists", return_value=True)
@patch("baudolo.seed.__main__.pd.read_csv", side_effect=EmptyDataError("empty"))
@patch("baudolo.seed.__main__._empty_df")
@patch("baudolo.seed.__main__.pd.concat")
@patch("baudolo.seed.__main__.print")
def test_check_and_add_entry_empty_file_warns_and_creates_columns_and_adds(
self,
print_: MagicMock,
concat: MagicMock,
empty_df: MagicMock,
read_csv: MagicMock,
exists: MagicMock,
) -> None:
df_existing = self._mock_df_mask_any(any_value=False)
empty_df.return_value = df_existing
df_out = MagicMock(spec=pd.DataFrame)
concat.return_value = df_out
seed_main.check_and_add_entry(
file_path="/tmp/databases.csv",
instance="inst",
database="db",
username="user",
password="pass",
)
exists.assert_called_once_with("/tmp/databases.csv")
read_csv.assert_called_once()
empty_df.assert_called_once()
concat.assert_called_once()
# Assert: at least one print call contains the WARNING and prints to stderr
warning_calls = []
for call in print_.call_args_list:
args, kwargs = call
if args and "WARNING: databases.csv exists but is empty" in str(args[0]):
warning_calls.append((args, kwargs))
self.assertTrue(
warning_calls,
"Expected a WARNING print when databases.csv is empty, but none was found.",
)
# Ensure the warning goes to stderr
_, warn_kwargs = warning_calls[0]
self.assertEqual(warn_kwargs.get("file"), seed_main.sys.stderr)
df_out.to_csv.assert_called_once_with(
"/tmp/databases.csv", sep=";", index=False
)
@patch("baudolo.seed.__main__.os.path.exists", return_value=True)
@patch("baudolo.seed.__main__.pd.read_csv")
def test_check_and_add_entry_updates_existing_row(
self,
read_csv: MagicMock,
exists: MagicMock,
) -> None:
df = self._mock_df_mask_any(any_value=True)
read_csv.return_value = df
seed_main.check_and_add_entry(
file_path="/tmp/databases.csv",
instance="inst",
database="db",
username="user",
password="pass",
)
df.to_csv.assert_called_once_with("/tmp/databases.csv", sep=";", index=False)
@patch("baudolo.seed.__main__.check_and_add_entry")
@patch("baudolo.seed.__main__.argparse.ArgumentParser.parse_args")
def test_main_calls_check_and_add_entry(
self, parse_args: MagicMock, cae: MagicMock
) -> None:
ns = MagicMock()
ns.file = "/tmp/databases.csv"
ns.instance = "inst"
ns.database = "db"
ns.username = "user"
ns.password = "pass"
parse_args.return_value = ns
seed_main.main()
cae.assert_called_once_with(
file_path="/tmp/databases.csv",
instance="inst",
database="db",
username="user",
password="pass",
)
@patch("baudolo.seed.__main__.sys.exit")
@patch("baudolo.seed.__main__.print")
@patch(
"baudolo.seed.__main__.check_and_add_entry", side_effect=RuntimeError("boom")
)
@patch("baudolo.seed.__main__.argparse.ArgumentParser.parse_args")
def test_main_exits_nonzero_on_error(
self,
parse_args: MagicMock,
cae: MagicMock,
print_: MagicMock,
exit_: MagicMock,
) -> None:
ns = MagicMock()
ns.file = "/tmp/databases.csv"
ns.instance = "inst"
ns.database = "db"
ns.username = "user"
ns.password = "pass"
parse_args.return_value = ns
seed_main.main()
self.assertTrue(print_.called)
_, kwargs = print_.call_args
self.assertEqual(kwargs.get("file"), seed_main.sys.stderr)
exit_.assert_called_once_with(1)
if __name__ == "__main__":
unittest.main(verbosity=2)

View File

@@ -4,37 +4,58 @@ from unittest.mock import patch
from baudolo.backup.app import requires_stop
@patch("baudolo.backup.app.is_swarm_task", return_value=False)
class TestRequiresStop(unittest.TestCase):
@patch("baudolo.backup.app.get_image_info")
def test_requires_stop_false_when_all_images_are_whitelisted(
self, mock_get_image_info
self, mock_get_image_info, _mock_is_swarm_task
):
# All containers use images containing allowed substrings
mock_get_image_info.side_effect = [
"repo/mastodon:v4",
"repo/wordpress:latest",
]
containers = ["c1", "c2"]
whitelist = ["mastodon", "wordpress"]
whitelist = ["repo/mastodon:v4", "repo/wordpress:latest"]
self.assertFalse(requires_stop(containers, whitelist))
@patch("baudolo.backup.app.get_image_info")
def test_requires_stop_true_when_any_image_is_not_whitelisted(
self, mock_get_image_info
self, mock_get_image_info, _mock_is_swarm_task
):
mock_get_image_info.side_effect = [
"repo/mastodon:v4",
"repo/nginx:latest",
]
containers = ["c1", "c2"]
whitelist = ["mastodon", "wordpress"]
whitelist = ["repo/mastodon:v4", "repo/wordpress:latest"]
self.assertTrue(requires_stop(containers, whitelist))
@patch("baudolo.backup.app.get_image_info")
def test_requires_stop_true_when_whitelist_empty(self, mock_get_image_info):
def test_requires_stop_true_on_substring_only_match(
self, mock_get_image_info, _mock_is_swarm_task
):
mock_get_image_info.return_value = "reg:5000/repo/mastodon:v4"
self.assertTrue(requires_stop(["c1"], ["mastodon"]))
self.assertTrue(requires_stop(["c1"], ["repo/mastodon:v4"]))
@patch("baudolo.backup.app.get_image_info")
def test_requires_stop_true_when_whitelist_empty(
self, mock_get_image_info, _mock_is_swarm_task
):
mock_get_image_info.return_value = "repo/anything:latest"
self.assertTrue(requires_stop(["c1"], []))
class TestRequiresStopSwarm(unittest.TestCase):
@patch("baudolo.backup.app.get_image_info")
@patch("baudolo.backup.app.is_swarm_task", return_value=True)
def test_swarm_tasks_never_require_stop(
self, _mock_is_swarm_task, mock_get_image_info
):
mock_get_image_info.return_value = "repo/not-whitelisted:latest"
self.assertFalse(requires_stop(["c1"], []))
mock_get_image_info.assert_not_called()
if __name__ == "__main__":
unittest.main()