11 Commits

Author SHA1 Message Date
0222f7f109 Release version 1.1.0 2025-12-28 22:16:41 +01:00
6adafe6b1f fix(backup): log missing db config instead of raising
- Use module logger in backup/db.py
- Skip db dump when no databases.csv entry is present
- Apply black/formatting cleanup across backup/restore/tests

https://chatgpt.com/share/69519d45-b0dc-800f-acb6-6fb8504e9b46
2025-12-28 22:12:31 +01:00
88b35ee923 backup(cli): use FHS-compliant default backup directory
- Replace dynamic repo name detection with stable default
- Switch default backups directory from /Backups to /var/lib/backup
- Align CLI defaults with Linux FHS best practices

https://chatgpt.com/share/69515eed-001c-800f-b1da-aee8d8683e63
2025-12-28 17:46:31 +01:00
71f79929be Changedf pi update mirror 2025-12-27 12:49:24 +01:00
0fb8efba4f Ignored .egg-info 2025-12-27 09:33:59 +01:00
3b39a6ef02 Release version 1.0.0 2025-12-27 09:30:38 +01:00
e0b2e8934e docs(readme): rewrite README to reflect deterministic backup design
- clarify separation between file backups (always) and SQL dumps (explicit only)
- document correct nested backup directory layout
- remove legacy script-based usage and outdated sections
- add explicit explanation of database definition scope
- update usage examples to current baudolo CLI

https://chatgpt.com/share/694ef6d2-7584-800f-a32b-27367f234d1d
2025-12-26 21:57:46 +01:00
bbb2dd1732 Removed .travis 2025-12-26 21:03:00 +01:00
159502af5e Added mirros 2025-12-26 20:50:29 +01:00
698d1e7a9e ci: add Makefile-driven CI with unit, integration and e2e tests
- add GitHub Actions CI workflow using Makefile targets exclusively
- run unit, integration and e2e tests via `make test`
- publish Docker image to GHCR on SemVer tags
- force-update `stable` git tag after successful release
- add integration test for seed CLI (CSV upsert behavior)
- extend Makefile with test-unit and test-integration targets

https://chatgpt.com/share/694ee54f-b814-800f-a714-e87563e538b7
2025-12-26 20:43:06 +01:00
f8420c8bea renamed configure to seed 2025-12-26 19:58:39 +01:00
32 changed files with 790 additions and 236 deletions

91
.github/workflows/ci.yml vendored Normal file
View File

@@ -0,0 +1,91 @@
name: CI (make tests, stable, publish)
on:
push:
branches: ["**"]
tags: ["v*.*.*"] # SemVer tags like v1.2.3
pull_request:
permissions:
contents: write # push/update 'stable' tag
packages: write # push to GHCR
env:
IMAGE_NAME: baudolo
REGISTRY: ghcr.io
IMAGE_REPO: ${{ github.repository }}
jobs:
test:
name: make test
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Show docker info
run: |
docker version
docker info
- name: Run all tests via Makefile
run: |
make test
- name: Upload E2E artifacts (always)
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-artifacts
path: artifacts
if-no-files-found: ignore
stable_and_publish:
name: Mark stable + publish image (SemVer tags only)
needs: [test]
runs-on: ubuntu-latest
if: startsWith(github.ref, 'refs/tags/v')
steps:
- name: Checkout (full history for tags)
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Derive version from tag
id: ver
run: |
TAG="${GITHUB_REF#refs/tags/}" # v1.2.3
echo "tag=${TAG}" >> "$GITHUB_OUTPUT"
- name: Mark 'stable' git tag (force update)
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
git tag -f stable "${GITHUB_SHA}"
git push -f origin stable
- name: Login to GHCR
uses: docker/login-action@v3
with:
registry: ${{ env.REGISTRY }}
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Build image (Makefile)
run: |
make build
- name: Tag image for registry
run: |
# local image built by Makefile is: baudolo:local
docker tag "${IMAGE_NAME}:local" "${REGISTRY}/${IMAGE_REPO}:${{ steps.ver.outputs.tag }}"
docker tag "${IMAGE_NAME}:local" "${REGISTRY}/${IMAGE_REPO}:stable"
docker tag "${IMAGE_NAME}:local" "${REGISTRY}/${IMAGE_REPO}:sha-${GITHUB_SHA::12}"
- name: Push image
run: |
docker push "${REGISTRY}/${IMAGE_REPO}:${{ steps.ver.outputs.tag }}"
docker push "${REGISTRY}/${IMAGE_REPO}:stable"
docker push "${REGISTRY}/${IMAGE_REPO}:sha-${GITHUB_SHA::12}"

3
.gitignore vendored
View File

@@ -1,2 +1,3 @@
__pycache__ __pycache__
artifacts/ artifacts/
*.egg-info

View File

@@ -1,2 +0,0 @@
language: shell
script: shellcheck $(find . -type f -name '*.sh')

11
CHANGELOG.md Normal file
View File

@@ -0,0 +1,11 @@
## [1.1.0] - 2025-12-28
* * **Backup:** Log a warning and skip database dumps when no databases.csv entry is present instead of raising an exception; introduce module-level logging and apply formatting cleanups across backup/restore code and tests.
* **CLI:** Switch to an FHS-compliant default backup directory (/var/lib/backup) and use a stable default repository name instead of dynamic detection.
* **Maintenance:** Update mirror configuration and ignore generated .egg-info files.
## [1.0.0] - 2025-12-27
* Official Release 🥳

4
MIRRORS Normal file
View File

@@ -0,0 +1,4 @@
git@github.com:kevinveenbirkenbach/backup-docker-to-local.git
ssh://git@git.veen.world:2201/kevinveenbirkenbach/backup-docker-to-local.git
ssh://git@code.infinito.nexus:2201/kevinveenbirkenbach/backup-docker-to-local.git
https://pypi.org/project/backup-docker-to-local/

View File

@@ -1,4 +1,5 @@
.PHONY: install build test-e2e .PHONY: install build \
test-e2e test test-unit test-integration
# Default python if no venv is active # Default python if no venv is active
PY_DEFAULT ?= python3 PY_DEFAULT ?= python3
@@ -41,4 +42,16 @@ clean:
# - loads the freshly built image into DinD # - loads the freshly built image into DinD
# - runs the unittest suite inside a container that talks to DinD via DOCKER_HOST # - runs the unittest suite inside a container that talks to DinD via DOCKER_HOST
test-e2e: clean build test-e2e: clean build
@bash scripts/test-e2e.sh @bash scripts/test-e2e.sh
test: test-unit test-integration test-e2e
test-unit: clean build
@echo ">> Running unit tests"
@docker run --rm -t $(IMAGE) \
sh -lc 'python -m unittest discover -t . -s tests/unit -p "test_*.py" -v'
test-integration: clean build
@echo ">> Running integration tests"
@docker run --rm -t $(IMAGE) \
sh -lc 'python -m unittest discover -t . -s tests/integration -p "test_*.py" -v'

196
README.md
View File

@@ -1,80 +1,196 @@
# Backup Docker Volumes to Local (baudolo) 📦🔄 # baudolo Deterministic Backup & Restore for Docker Volumes 📦🔄
[![GitHub Sponsors](https://img.shields.io/badge/Sponsor-GitHub%20Sponsors-blue?logo=github)](https://github.com/sponsors/kevinveenbirkenbach) [![Patreon](https://img.shields.io/badge/Support-Patreon-orange?logo=patreon)](https://www.patreon.com/c/kevinveenbirkenbach) [![Buy Me a Coffee](https://img.shields.io/badge/Buy%20me%20a%20Coffee-Funding-yellow?logo=buymeacoffee)](https://buymeacoffee.com/kevinveenbirkenbach) [![PayPal](https://img.shields.io/badge/Donate-PayPal-blue?logo=paypal)](https://s.veen.world/paypaldonate) [![GitHub Sponsors](https://img.shields.io/badge/Sponsor-GitHub%20Sponsors-blue?logo=github)](https://github.com/sponsors/kevinveenbirkenbach) [![Patreon](https://img.shields.io/badge/Support-Patreon-orange?logo=patreon)](https://www.patreon.com/c/kevinveenbirkenbach) [![Buy Me a Coffee](https://img.shields.io/badge/Buy%20me%20a%20Coffee-Funding-yellow?logo=buymeacoffee)](https://buymeacoffee.com/kevinveenbirkenbach) [![PayPal](https://img.shields.io/badge/Donate-PayPal-blue?logo=paypal)](https://s.veen.world/paypaldonate) [![License: AGPL v3](https://img.shields.io/badge/License-AGPL%20v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0) [![Docker Version](https://img.shields.io/badge/Docker-Yes-blue.svg)](https://www.docker.com) [![Python Version](https://img.shields.io/badge/Python-3.x-blue.svg)](https://www.python.org) [![GitHub stars](https://img.shields.io/github/stars/kevinveenbirkenbach/backup-docker-to-local.svg?style=social)](https://github.com/kevinveenbirkenbach/backup-docker-to-local/stargazers)
**Backup Docker Volumes to Local** is a set of Python and shell scripts that enable you to perform incremental backups of all your Docker volumes using rsync. It is designed to integrate seamlessly with [Kevin's Package Manager](https://github.com/kevinveenbirkenbach/package-manager) under the alias **baudolo**, making it easy to install and manage. The tool supports both file and database recoveries with a clear, automated backup scheme. `baudolo` is a backup and restore system for Docker volumes with
**mandatory file backups** and **explicit, deterministic database dumps**.
It is designed for environments with many Docker services where:
- file-level backups must always exist
- database dumps must be intentional, predictable, and auditable
[![License: AGPL v3](https://img.shields.io/badge/License-AGPL%20v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0) [![Docker Version](https://img.shields.io/badge/Docker-Yes-blue.svg)](https://www.docker.com) [![Python Version](https://img.shields.io/badge/Python-3.x-blue.svg)](https://www.python.org) [![GitHub stars](https://img.shields.io/github/stars/kevinveenbirkenbach/backup-docker-to-local.svg?style=social)](https://github.com/kevinveenbirkenbach/backup-docker-to-local/stargazers) ## ✨ Key Features
## 🎯 Goal - 📦 Incremental Docker volume backups using `rsync --link-dest`
- 🗄 Optional SQL dumps for:
- PostgreSQL
- MariaDB / MySQL
- 🌱 Explicit database definition for SQL backups (no auto-discovery)
- 🧾 Backup integrity stamping via `dirval` (Python API)
- ⏸ Automatic container stop/start when required for consistency
- 🚫 Whitelisting of containers that do not require stopping
- ♻️ Modular, maintainable Python architecture
This project automates the backup of Docker volumes using incremental backups (rsync) and supports recovering both files and database dumps (MariaDB/PostgreSQL). A robust directory stamping mechanism ensures data integrity, and the tool also handles restarting Docker Compose services when necessary.
## 🚀 Features ## 🧠 Core Concept (Important!)
- **Incremental Backups:** Uses rsync with `--link-dest` for efficient, versioned backups. `baudolo` **separates file backups from database dumps**.
- **Database Backup Support:** Backs up MariaDB and PostgreSQL databases from running containers.
- **Volume Recovery:** Provides scripts to recover volumes and databases from backups.
- **Docker Compose Integration:** Option to automatically restart Docker Compose services after backup.
- **Flexible Configuration:** Easily integrated with your Docker environment with minimal setup.
- **Comprehensive Logging:** Detailed command output and error handling for safe operations.
## 🛠 Requirements - **Docker volumes are always backed up at file level**
- **SQL dumps are created only for explicitly defined databases**
- **Linux Operating System** (with Docker installed) 🐧 This results in the following behavior:
- **Python 3.x** 🐍
- **Docker & Docker Compose** 🔧
- **rsync** installed on your system
## 📥 Installation | Database defined | File backup | SQL dump |
|------------------|-------------|----------|
| No | ✔ yes | ✘ no |
| Yes | ✔ yes | ✔ yes |
You can install **Backup Docker Volumes to Local** easily via [Kevin's Package Manager](https://github.com/kevinveenbirkenbach/package-manager) using the alias **baudolo**: ## 📁 Backup Layout
```bash Backups are stored in a deterministic, fully nested structure:
pkgmgr install baudolo
```text
<backups-dir>/
└── <machine-hash>/
└── <repo-name>/
└── <timestamp>/
└── <volume-name>/
├── files/
└── sql/
└── <database>.backup.sql
``` ```
Alternatively, clone the repository directly: ### Meaning of each level
* `<machine-hash>`
SHA256 hash of `/etc/machine-id` (host separation)
* `<repo-name>`
Logical backup namespace (project / stack)
* `<timestamp>`
Backup generation (`YYYYMMDDHHMMSS`)
* `<volume-name>`
Docker volume name
* `files/`
Incremental file backup (rsync)
* `sql/`
Optional SQL dumps (only for defined databases)
## 🚀 Installation
### Local (editable install)
```bash ```bash
git clone https://github.com/kevinveenbirkenbach/backup-docker-to-local.git python3 -m venv .venv
cd backup-docker-to-local source .venv/bin/activate
pip install -e .
``` ```
## 🚀 Usage ## 🌱 Database Definition (SQL Backup Scope)
### Backup All Volumes ### How SQL backups are defined
To backup all Docker volumes, simply run: `baudolo` creates SQL dumps **only** for databases that are **explicitly defined**
via configuration (e.g. a databases definition file or seeding step).
If a database is **not defined**:
* its Docker volume is still backed up (files)
* **no SQL dump is created**
> No database definition → file backup only
> Database definition present → file backup + SQL dump
### Why explicit definition?
`baudolo` does **not** inspect running containers to guess databases.
Databases must be explicitly defined to guarantee:
* deterministic backups
* predictable restore behavior
* reproducible environments
* zero accidental production data exposure
### Required database metadata
Each database definition provides:
* database instance (container or logical instance)
* database name
* database user
* database password
This information is used by `baudolo` to execute
`pg_dump`, `pg_dumpall`, or `mariadb-dump`.
## 💾 Running a Backup
```bash ```bash
./backup-docker-to-local.sh baudolo \
--compose-dir /srv/docker \
--databases-csv /etc/baudolo/databases.csv \
--database-containers central-postgres central-mariadb \
--images-no-stop-required alpine postgres mariadb mysql \
--images-no-backup-required redis busybox
``` ```
### Recovery ### Common Backup Flags
#### Recover Volume Files | Flag | Description |
| --------------- | ------------------------------------------- |
| `--everything` | Always stop containers and re-run rsync |
| `--dump-only` | Only create SQL dumps, skip file backups |
| `--shutdown` | Do not restart containers after backup |
| `--backups-dir` | Backup root directory (default: `/Backups`) |
| `--repo-name` | Backup namespace under machine hash |
## ♻️ Restore Operations
### Restore Volume Files
```bash ```bash
bash ./recover-docker-from-local.sh "{{volume_name}}" "$(sha256sum /etc/machine-id | head -c 64)" "{{version_to_recover}}" baudolo-restore files \
my-volume \
<machine-hash> \
<version> \
--backups-dir /Backups \
--repo-name my-repo
``` ```
#### Recover Database Restore into a **different target volume**:
For example, to recover a MySQL/MariaDB database:
```bash ```bash
docker exec -i mysql_container mysql -uroot -psecret database < db.sql baudolo-restore files \
target-volume \
<machine-hash> \
<version> \
--source-volume source-volume
``` ```
#### Debug Mode ### Restore PostgreSQL
To inspect whats happening inside a container:
```bash ```bash
docker run -it --entrypoint /bin/sh --rm --volumes-from {{container_name}} -v /Backups/:/Backups/ kevinveenbirkenbach/alpine-rsync baudolo-restore postgres \
my-volume \
<machine-hash> \
<version> \
--container postgres \
--db-name appdb \
--db-password secret \
--empty
``` ```
### Restore MariaDB / MySQL
```bash
baudolo-restore mariadb \
my-volume \
<machine-hash> \
<version> \
--container mariadb \
--db-name shopdb \
--db-password secret \
--empty
```
> `baudolo` automatically detects whether `mariadb` or `mysql`
> is available inside the container
## 🔍 Backup Scheme ## 🔍 Backup Scheme
The backup mechanism uses incremental backups with rsync and stamps directories with a unique hash. For more details on the backup scheme, check out [this blog post](https://blog.veen.world/blog/2020/12/26/how-i-backup-dedicated-root-servers/). The backup mechanism uses incremental backups with rsync and stamps directories with a unique hash. For more details on the backup scheme, check out [this blog post](https://blog.veen.world/blog/2020/12/26/how-i-backup-dedicated-root-servers/).

Binary file not shown.

BIN
dist/backup_docker_to_local-1.0.0.tar.gz vendored Normal file

Binary file not shown.

View File

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "backup-docker-to-local" name = "backup-docker-to-local"
version = "0.1.0" version = "1.1.0"
description = "Backup Docker volumes to local with rsync and optional DB dumps." description = "Backup Docker volumes to local with rsync and optional DB dumps."
readme = "README.md" readme = "README.md"
requires-python = ">=3.9" requires-python = ">=3.9"
@@ -19,7 +19,7 @@ dependencies = [
[project.scripts] [project.scripts]
baudolo = "baudolo.backup.__main__:main" baudolo = "baudolo.backup.__main__:main"
baudolo-restore = "baudolo.restore.__main__:main" baudolo-restore = "baudolo.restore.__main__:main"
baudolo-configure = "baudolo.configure.__main__:main" baudolo-seed = "baudolo.seed.__main__:main"
[tool.setuptools] [tool.setuptools]
package-dir = { "" = "src" } package-dir = { "" = "src" }

View File

@@ -51,7 +51,9 @@ def is_image_ignored(container: str, images_no_backup_required: list[str]) -> bo
return any(pat in img for pat in images_no_backup_required) return any(pat in img for pat in images_no_backup_required)
def volume_is_fully_ignored(containers: list[str], images_no_backup_required: list[str]) -> bool: def volume_is_fully_ignored(
containers: list[str], images_no_backup_required: list[str]
) -> bool:
""" """
Skip file backup only if all containers linked to the volume are ignored. Skip file backup only if all containers linked to the volume are ignored.
""" """
@@ -178,6 +180,8 @@ def main() -> int:
print("Finished volume backups.", flush=True) print("Finished volume backups.", flush=True)
print("Handling Docker Compose services...", flush=True) print("Handling Docker Compose services...", flush=True)
handle_docker_compose_services(args.compose_dir, args.docker_compose_hard_restart_required) handle_docker_compose_services(
args.compose_dir, args.docker_compose_hard_restart_required
)
return 0 return 0

View File

@@ -2,22 +2,6 @@ from __future__ import annotations
import argparse import argparse
import os import os
from pathlib import Path
def _default_repo_name() -> str:
"""
Derive the repository name from the folder that contains `src/`.
Expected layout:
<repo-root>/src/baudolo/backup/cli.py
=> parents[0]=backup, [1]=baudolo, [2]=src, [3]=repo-root
"""
try:
return Path(__file__).resolve().parents[3].name
except Exception:
return "backup-docker-to-local"
def parse_args() -> argparse.Namespace: def parse_args() -> argparse.Namespace:
@@ -41,7 +25,7 @@ def parse_args() -> argparse.Namespace:
p.add_argument( p.add_argument(
"--repo-name", "--repo-name",
default=_default_repo_name(), default="backup-docker-to-local",
help="Backup repo folder name under <backups-dir>/<machine-id>/ (default: git repo folder name)", help="Backup repo folder name under <backups-dir>/<machine-id>/ (default: git repo folder name)",
) )
p.add_argument( p.add_argument(
@@ -51,8 +35,8 @@ def parse_args() -> argparse.Namespace:
) )
p.add_argument( p.add_argument(
"--backups-dir", "--backups-dir",
default="/Backups", default="/var/lib/backup/",
help="Backup root directory (default: /Backups)", help="Backup root directory (default: /var/lib/backup/)",
) )
p.add_argument( p.add_argument(

View File

@@ -10,7 +10,9 @@ def hard_restart_docker_services(dir_path: str) -> None:
subprocess.run(["docker-compose", "up", "-d"], cwd=dir_path, check=True) subprocess.run(["docker-compose", "up", "-d"], cwd=dir_path, check=True)
def handle_docker_compose_services(parent_directory: str, hard_restart_required: list[str]) -> None: def handle_docker_compose_services(
parent_directory: str, hard_restart_required: list[str]
) -> None:
for entry in os.scandir(parent_directory): for entry in os.scandir(parent_directory):
if not entry.is_dir(): if not entry.is_dir():
continue continue

View File

@@ -5,9 +5,12 @@ import pathlib
import re import re
import pandas import pandas
import logging
from .shell import BackupException, execute_shell_command from .shell import BackupException, execute_shell_command
log = logging.getLogger(__name__)
def get_instance(container: str, database_containers: list[str]) -> str: def get_instance(container: str, database_containers: list[str]) -> str:
if container in database_containers: if container in database_containers:
@@ -15,7 +18,9 @@ def get_instance(container: str, database_containers: list[str]) -> str:
return re.split(r"(_|-)(database|db|postgres)", container)[0] return re.split(r"(_|-)(database|db|postgres)", container)[0]
def fallback_pg_dumpall(container: str, username: str, password: str, out_file: str) -> None: def fallback_pg_dumpall(
container: str, username: str, password: str, out_file: str
) -> None:
cmd = ( cmd = (
f"PGPASSWORD={password} docker exec -i {container} " f"PGPASSWORD={password} docker exec -i {container} "
f"pg_dumpall -U {username} -h localhost > {out_file}" f"pg_dumpall -U {username} -h localhost > {out_file}"
@@ -34,7 +39,8 @@ def backup_database(
instance_name = get_instance(container, database_containers) instance_name = get_instance(container, database_containers)
entries = databases_df.loc[databases_df["instance"] == instance_name] entries = databases_df.loc[databases_df["instance"] == instance_name]
if entries.empty: if entries.empty:
raise BackupException(f"No entry found for instance '{instance_name}'") log.warning("No entry found for instance '%s'", instance_name)
return
out_dir = os.path.join(volume_dir, "sql") out_dir = os.path.join(volume_dir, "sql")
pathlib.Path(out_dir).mkdir(parents=True, exist_ok=True) pathlib.Path(out_dir).mkdir(parents=True, exist_ok=True)
@@ -68,6 +74,9 @@ def backup_database(
execute_shell_command(cmd) execute_shell_command(cmd)
except BackupException as e: except BackupException as e:
print(f"pg_dump failed: {e}", flush=True) print(f"pg_dump failed: {e}", flush=True)
print(f"Falling back to pg_dumpall for instance '{instance_name}'", flush=True) print(
f"Falling back to pg_dumpall for instance '{instance_name}'",
flush=True,
)
fallback_pg_dumpall(container, user, password, cluster_file) fallback_pg_dumpall(container, user, password, cluster_file)
continue continue

View File

@@ -37,7 +37,9 @@ def change_containers_status(containers: list[str], status: str) -> None:
def docker_volume_exists(volume: str) -> bool: def docker_volume_exists(volume: str) -> bool:
# Avoid throwing exceptions for exists checks. # Avoid throwing exceptions for exists checks.
try: try:
execute_shell_command(f"docker volume inspect {volume} >/dev/null 2>&1 && echo OK") execute_shell_command(
f"docker volume inspect {volume} >/dev/null 2>&1 && echo OK"
)
return True return True
except Exception: except Exception:
return False return False

View File

@@ -13,7 +13,9 @@ def get_storage_path(volume_name: str) -> str:
return f"{path}/" return f"{path}/"
def get_last_backup_dir(versions_dir: str, volume_name: str, current_backup_dir: str) -> str | None: def get_last_backup_dir(
versions_dir: str, volume_name: str, current_backup_dir: str
) -> str | None:
versions = sorted(os.listdir(versions_dir), reverse=True) versions = sorted(os.listdir(versions_dir), reverse=True)
for version in versions: for version in versions:
candidate = os.path.join(versions_dir, version, volume_name, "files", "") candidate = os.path.join(versions_dir, version, volume_name, "files", "")
@@ -37,6 +39,8 @@ def backup_volume(versions_dir: str, volume_name: str, volume_dir: str) -> None:
execute_shell_command(cmd) execute_shell_command(cmd)
except BackupException as e: except BackupException as e:
if "file has vanished" in str(e): if "file has vanished" in str(e):
print("Warning: Some files vanished before transfer. Continuing.", flush=True) print(
"Warning: Some files vanished before transfer. Continuing.", flush=True
)
else: else:
raise raise

View File

@@ -1 +1 @@
__all__ = ["main"] __all__ = ["main"]

View File

@@ -66,7 +66,9 @@ def main(argv: list[str] | None = None) -> int:
# ------------------------------------------------------------------ # ------------------------------------------------------------------
# mariadb # mariadb
# ------------------------------------------------------------------ # ------------------------------------------------------------------
p_mdb = sub.add_parser("mariadb", help="Restore a single MariaDB/MySQL-compatible dump") p_mdb = sub.add_parser(
"mariadb", help="Restore a single MariaDB/MySQL-compatible dump"
)
_add_common_backup_args(p_mdb) _add_common_backup_args(p_mdb)
p_mdb.add_argument("--container", required=True) p_mdb.add_argument("--container", required=True)
p_mdb.add_argument("--db-name", required=True) p_mdb.add_argument("--db-name", required=True)

View File

@@ -1 +1 @@
"""Database restore handlers (Postgres, MariaDB/MySQL).""" """Database restore handlers (Postgres, MariaDB/MySQL)."""

View File

@@ -23,7 +23,9 @@ exit 42
raise RuntimeError("empty client detection output") raise RuntimeError("empty client detection output")
return out return out
except Exception as e: except Exception as e:
print("ERROR: neither 'mariadb' nor 'mysql' found in container.", file=sys.stderr) print(
"ERROR: neither 'mariadb' nor 'mysql' found in container.", file=sys.stderr
)
raise e raise e
@@ -47,7 +49,14 @@ def restore_mariadb_sql(
# MariaDB 11 images may not contain the mysql binary at all. # MariaDB 11 images may not contain the mysql binary at all.
docker_exec( docker_exec(
container, container,
[client, "-u", user, f"--password={password}", "-e", "SET FOREIGN_KEY_CHECKS=0;"], [
client,
"-u",
user,
f"--password={password}",
"-e",
"SET FOREIGN_KEY_CHECKS=0;",
],
) )
result = docker_exec( result = docker_exec(
@@ -80,10 +89,19 @@ def restore_mariadb_sql(
docker_exec( docker_exec(
container, container,
[client, "-u", user, f"--password={password}", "-e", "SET FOREIGN_KEY_CHECKS=1;"], [
client,
"-u",
user,
f"--password={password}",
"-e",
"SET FOREIGN_KEY_CHECKS=1;",
],
) )
with open(sql_path, "rb") as f: with open(sql_path, "rb") as f:
docker_exec(container, [client, "-u", user, f"--password={password}", db_name], stdin=f) docker_exec(
container, [client, "-u", user, f"--password={password}", db_name], stdin=f
)
print(f"MariaDB/MySQL restore complete for db '{db_name}'.") print(f"MariaDB/MySQL restore complete for db '{db_name}'.")

View File

@@ -6,7 +6,9 @@ import sys
from .run import run, docker_volume_exists from .run import run, docker_volume_exists
def restore_volume_files(volume_name: str, backup_files_dir: str, *, rsync_image: str) -> int: def restore_volume_files(
volume_name: str, backup_files_dir: str, *, rsync_image: str
) -> int:
if not os.path.isdir(backup_files_dir): if not os.path.isdir(backup_files_dir):
print(f"ERROR: backup files dir not found: {backup_files_dir}", file=sys.stderr) print(f"ERROR: backup files dir not found: {backup_files_dir}", file=sys.stderr)
return 2 return 2

View File

@@ -2,21 +2,24 @@ import pandas as pd
import argparse import argparse
import os import os
def check_and_add_entry(file_path, instance, database, username, password): def check_and_add_entry(file_path, instance, database, username, password):
# Check if the file exists and is not empty # Check if the file exists and is not empty
if os.path.exists(file_path) and os.path.getsize(file_path) > 0: if os.path.exists(file_path) and os.path.getsize(file_path) > 0:
# Read the existing CSV file with header # Read the existing CSV file with header
df = pd.read_csv(file_path, sep=';') df = pd.read_csv(file_path, sep=";")
else: else:
# Create a new DataFrame with columns if file does not exist # Create a new DataFrame with columns if file does not exist
df = pd.DataFrame(columns=['instance', 'database', 'username', 'password']) df = pd.DataFrame(columns=["instance", "database", "username", "password"])
# Check if the entry exists and remove it # Check if the entry exists and remove it
mask = ( mask = (
(df['instance'] == instance) & (df["instance"] == instance)
((df['database'] == database) | & (
(((df['database'].isna()) | (df['database'] == '')) & (database == ''))) & (df["database"] == database)
(df['username'] == username) | (((df["database"].isna()) | (df["database"] == "")) & (database == ""))
)
& (df["username"] == username)
) )
if not df[mask].empty: if not df[mask].empty:
@@ -26,25 +29,40 @@ def check_and_add_entry(file_path, instance, database, username, password):
print("Adding new entry.") print("Adding new entry.")
# Create a new DataFrame for the new entry # Create a new DataFrame for the new entry
new_entry = pd.DataFrame([{'instance': instance, 'database': database, 'username': username, 'password': password}]) new_entry = pd.DataFrame(
[
{
"instance": instance,
"database": database,
"username": username,
"password": password,
}
]
)
# Add (or replace) the entry using concat # Add (or replace) the entry using concat
df = pd.concat([df, new_entry], ignore_index=True) df = pd.concat([df, new_entry], ignore_index=True)
# Save the updated CSV file # Save the updated CSV file
df.to_csv(file_path, sep=';', index=False) df.to_csv(file_path, sep=";", index=False)
def main(): def main():
parser = argparse.ArgumentParser(description="Check and replace (or add) a database entry in a CSV file.") parser = argparse.ArgumentParser(
description="Check and replace (or add) a database entry in a CSV file."
)
parser.add_argument("file_path", help="Path to the CSV file") parser.add_argument("file_path", help="Path to the CSV file")
parser.add_argument("instance", help="Database instance") parser.add_argument("instance", help="Database instance")
parser.add_argument("database", help="Database name") parser.add_argument("database", help="Database name")
parser.add_argument("username", help="Username") parser.add_argument("username", help="Username")
parser.add_argument("password", nargs='?', default="", help="Password (optional)") parser.add_argument("password", nargs="?", default="", help="Password (optional)")
args = parser.parse_args() args = parser.parse_args()
check_and_add_entry(args.file_path, args.instance, args.database, args.username, args.password) check_and_add_entry(
args.file_path, args.instance, args.database, args.username, args.password
)
if __name__ == "__main__": if __name__ == "__main__":
main() main()

View File

@@ -34,7 +34,9 @@ def run(
raise raise
def sh(cmd: str, *, capture: bool = True, check: bool = True) -> subprocess.CompletedProcess: def sh(
cmd: str, *, capture: bool = True, check: bool = True
) -> subprocess.CompletedProcess:
return run(["sh", "-lc", cmd], capture=capture, check=check) return run(["sh", "-lc", cmd], capture=capture, check=check)
@@ -63,24 +65,37 @@ def wait_for_log(container: str, pattern: str, timeout_s: int = 60) -> None:
raise TimeoutError(f"Timed out waiting for log pattern '{pattern}' in {container}") raise TimeoutError(f"Timed out waiting for log pattern '{pattern}' in {container}")
def wait_for_postgres(container: str, *, user: str = "postgres", timeout_s: int = 90) -> None: def wait_for_postgres(
container: str, *, user: str = "postgres", timeout_s: int = 90
) -> None:
""" """
Docker-outside-of-Docker friendly readiness: check from inside the DB container. Docker-outside-of-Docker friendly readiness: check from inside the DB container.
""" """
deadline = time.time() + timeout_s deadline = time.time() + timeout_s
while time.time() < deadline: while time.time() < deadline:
p = run( p = run(
["docker", "exec", container, "sh", "-lc", f"pg_isready -U {user} -h localhost"], [
"docker",
"exec",
container,
"sh",
"-lc",
f"pg_isready -U {user} -h localhost",
],
capture=True, capture=True,
check=False, check=False,
) )
if p.returncode == 0: if p.returncode == 0:
return return
time.sleep(1) time.sleep(1)
raise TimeoutError(f"Timed out waiting for Postgres readiness in container {container}") raise TimeoutError(
f"Timed out waiting for Postgres readiness in container {container}"
)
def wait_for_mariadb(container: str, *, root_password: str, timeout_s: int = 90) -> None: def wait_for_mariadb(
container: str, *, root_password: str, timeout_s: int = 90
) -> None:
""" """
Liveness probe for MariaDB. Liveness probe for MariaDB.
@@ -92,17 +107,28 @@ def wait_for_mariadb(container: str, *, root_password: str, timeout_s: int = 90)
deadline = time.time() + timeout_s deadline = time.time() + timeout_s
while time.time() < deadline: while time.time() < deadline:
p = run( p = run(
["docker", "exec", container, "sh", "-lc", "mariadb -uroot --protocol=socket -e \"SELECT 1;\""], [
"docker",
"exec",
container,
"sh",
"-lc",
'mariadb -uroot --protocol=socket -e "SELECT 1;"',
],
capture=True, capture=True,
check=False, check=False,
) )
if p.returncode == 0: if p.returncode == 0:
return return
time.sleep(1) time.sleep(1)
raise TimeoutError(f"Timed out waiting for MariaDB readiness in container {container}") raise TimeoutError(
f"Timed out waiting for MariaDB readiness in container {container}"
)
def wait_for_mariadb_sql(container: str, *, user: str, password: str, timeout_s: int = 90) -> None: def wait_for_mariadb_sql(
container: str, *, user: str, password: str, timeout_s: int = 90
) -> None:
""" """
SQL login readiness for the *dedicated test user* over TCP. SQL login readiness for the *dedicated test user* over TCP.
@@ -118,7 +144,7 @@ def wait_for_mariadb_sql(container: str, *, user: str, password: str, timeout_s:
container, container,
"sh", "sh",
"-lc", "-lc",
f"mariadb -h 127.0.0.1 -u{user} -p{password} -e \"SELECT 1;\"", f'mariadb -h 127.0.0.1 -u{user} -p{password} -e "SELECT 1;"',
], ],
capture=True, capture=True,
check=False, check=False,
@@ -126,7 +152,9 @@ def wait_for_mariadb_sql(container: str, *, user: str, password: str, timeout_s:
if p.returncode == 0: if p.returncode == 0:
return return
time.sleep(1) time.sleep(1)
raise TimeoutError(f"Timed out waiting for MariaDB SQL login readiness in container {container}") raise TimeoutError(
f"Timed out waiting for MariaDB SQL login readiness in container {container}"
)
def backup_run( def backup_run(
@@ -142,13 +170,20 @@ def backup_run(
) -> None: ) -> None:
cmd = [ cmd = [
"baudolo", "baudolo",
"--compose-dir", compose_dir, "--compose-dir",
"--docker-compose-hard-restart-required", "mailu", compose_dir,
"--repo-name", repo_name, "--docker-compose-hard-restart-required",
"--databases-csv", databases_csv, "mailu",
"--backups-dir", backups_dir, "--repo-name",
"--database-containers", *database_containers, repo_name,
"--images-no-stop-required", *images_no_stop_required, "--databases-csv",
databases_csv,
"--backups-dir",
backups_dir,
"--database-containers",
*database_containers,
"--images-no-stop-required",
*images_no_stop_required,
] ]
if images_no_backup_required: if images_no_backup_required:
cmd += ["--images-no-backup-required", *images_no_backup_required] cmd += ["--images-no-backup-required", *images_no_backup_required]

View File

@@ -1,5 +1,4 @@
import unittest import unittest
from pathlib import Path
from .helpers import ( from .helpers import (
backup_run, backup_run,
@@ -33,12 +32,19 @@ class TestE2EFilesFull(unittest.TestCase):
# create source volume with a file # create source volume with a file
run(["docker", "volume", "create", cls.volume_src]) run(["docker", "volume", "create", cls.volume_src])
run([ run(
"docker", "run", "--rm", [
"-v", f"{cls.volume_src}:/data", "docker",
"alpine:3.20", "run",
"sh", "-lc", "mkdir -p /data && echo 'hello' > /data/hello.txt", "--rm",
]) "-v",
f"{cls.volume_src}:/data",
"alpine:3.20",
"sh",
"-lc",
"mkdir -p /data && echo 'hello' > /data/hello.txt",
]
)
# databases.csv (unused, but required by CLI) # databases.csv (unused, but required by CLI)
cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv" cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv"
@@ -75,20 +81,36 @@ class TestE2EFilesFull(unittest.TestCase):
def test_restore_files_into_new_volume(self) -> None: def test_restore_files_into_new_volume(self) -> None:
# restore files from volume_src backup into volume_dst # restore files from volume_src backup into volume_dst
run([ run(
"baudolo-restore", "files", [
self.volume_dst, self.hash, self.version, "baudolo-restore",
"--backups-dir", self.backups_dir, "files",
"--repo-name", self.repo_name, self.volume_dst,
"--source-volume", self.volume_src, self.hash,
"--rsync-image", "ghcr.io/kevinveenbirkenbach/alpine-rsync", self.version,
]) "--backups-dir",
self.backups_dir,
"--repo-name",
self.repo_name,
"--source-volume",
self.volume_src,
"--rsync-image",
"ghcr.io/kevinveenbirkenbach/alpine-rsync",
]
)
# verify restored file exists in dst volume # verify restored file exists in dst volume
p = run([ p = run(
"docker", "run", "--rm", [
"-v", f"{self.volume_dst}:/data", "docker",
"alpine:3.20", "run",
"sh", "-lc", "cat /data/hello.txt", "--rm",
]) "-v",
f"{self.volume_dst}:/data",
"alpine:3.20",
"sh",
"-lc",
"cat /data/hello.txt",
]
)
self.assertEqual((p.stdout or "").strip(), "hello") self.assertEqual((p.stdout or "").strip(), "hello")

View File

@@ -31,12 +31,19 @@ class TestE2EFilesNoCopy(unittest.TestCase):
cls.volumes = [cls.volume_src, cls.volume_dst] cls.volumes = [cls.volume_src, cls.volume_dst]
run(["docker", "volume", "create", cls.volume_src]) run(["docker", "volume", "create", cls.volume_src])
run([ run(
"docker", "run", "--rm", [
"-v", f"{cls.volume_src}:/data", "docker",
"alpine:3.20", "run",
"sh", "-lc", "echo 'hello' > /data/hello.txt", "--rm",
]) "-v",
f"{cls.volume_src}:/data",
"alpine:3.20",
"sh",
"-lc",
"echo 'hello' > /data/hello.txt",
]
)
cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv" cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv"
write_databases_csv(cls.databases_csv, []) write_databases_csv(cls.databases_csv, [])
@@ -59,14 +66,29 @@ class TestE2EFilesNoCopy(unittest.TestCase):
cleanup_docker(containers=cls.containers, volumes=cls.volumes) cleanup_docker(containers=cls.containers, volumes=cls.volumes)
def test_files_backup_not_present(self) -> None: def test_files_backup_not_present(self) -> None:
p = backup_path(self.backups_dir, self.repo_name, self.version, self.volume_src) / "files" p = (
backup_path(self.backups_dir, self.repo_name, self.version, self.volume_src)
/ "files"
)
self.assertFalse(p.exists(), f"Did not expect files backup dir at: {p}") self.assertFalse(p.exists(), f"Did not expect files backup dir at: {p}")
def test_restore_files_fails_expected(self) -> None: def test_restore_files_fails_expected(self) -> None:
p = run([ p = run(
"baudolo-restore", "files", [
self.volume_dst, self.hash, self.version, "baudolo-restore",
"--backups-dir", self.backups_dir, "files",
"--repo-name", self.repo_name, self.volume_dst,
], check=False) self.hash,
self.assertEqual(p.returncode, 2, f"Expected exitcode 2, got {p.returncode}\nSTDOUT={p.stdout}\nSTDERR={p.stderr}") self.version,
"--backups-dir",
self.backups_dir,
"--repo-name",
self.repo_name,
],
check=False,
)
self.assertEqual(
p.returncode,
2,
f"Expected exitcode 2, got {p.returncode}\nSTDOUT={p.stdout}\nSTDERR={p.stderr}",
)

View File

@@ -62,8 +62,12 @@ class TestE2EMariaDBFull(unittest.TestCase):
) )
# Liveness + actual SQL login readiness (TCP) # Liveness + actual SQL login readiness (TCP)
wait_for_mariadb(cls.db_container, root_password=cls.root_password, timeout_s=90) wait_for_mariadb(
wait_for_mariadb_sql(cls.db_container, user=cls.db_user, password=cls.db_password, timeout_s=90) cls.db_container, root_password=cls.root_password, timeout_s=90
)
wait_for_mariadb_sql(
cls.db_container, user=cls.db_user, password=cls.db_password, timeout_s=90
)
# Create table + data via the dedicated user (TCP) # Create table + data via the dedicated user (TCP)
run( run(
@@ -74,14 +78,17 @@ class TestE2EMariaDBFull(unittest.TestCase):
"sh", "sh",
"-lc", "-lc",
f"mariadb -h 127.0.0.1 -u{cls.db_user} -p{cls.db_password} " f"mariadb -h 127.0.0.1 -u{cls.db_user} -p{cls.db_password} "
f"-e \"CREATE TABLE {cls.db_name}.t (id INT PRIMARY KEY, v VARCHAR(50)); " f'-e "CREATE TABLE {cls.db_name}.t (id INT PRIMARY KEY, v VARCHAR(50)); '
f"INSERT INTO {cls.db_name}.t VALUES (1,'ok');\"", f"INSERT INTO {cls.db_name}.t VALUES (1,'ok');\"",
] ]
) )
cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv" cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv"
# IMPORTANT: baudolo backup expects credentials for the DB dump. # IMPORTANT: baudolo backup expects credentials for the DB dump.
write_databases_csv(cls.databases_csv, [(cls.db_container, cls.db_name, cls.db_user, cls.db_password)]) write_databases_csv(
cls.databases_csv,
[(cls.db_container, cls.db_name, cls.db_user, cls.db_password)],
)
# Backup with file+dump # Backup with file+dump
backup_run( backup_run(
@@ -104,7 +111,7 @@ class TestE2EMariaDBFull(unittest.TestCase):
"sh", "sh",
"-lc", "-lc",
f"mariadb -h 127.0.0.1 -u{cls.db_user} -p{cls.db_password} " f"mariadb -h 127.0.0.1 -u{cls.db_user} -p{cls.db_password} "
f"-e \"DROP TABLE {cls.db_name}.t;\"", f'-e "DROP TABLE {cls.db_name}.t;"',
] ]
) )
@@ -137,7 +144,11 @@ class TestE2EMariaDBFull(unittest.TestCase):
cleanup_docker(containers=cls.containers, volumes=cls.volumes) cleanup_docker(containers=cls.containers, volumes=cls.volumes)
def test_dump_file_exists(self) -> None: def test_dump_file_exists(self) -> None:
p = backup_path(self.backups_dir, self.repo_name, self.version, self.db_volume) / "sql" / f"{self.db_name}.backup.sql" p = (
backup_path(self.backups_dir, self.repo_name, self.version, self.db_volume)
/ "sql"
/ f"{self.db_name}.backup.sql"
)
self.assertTrue(p.is_file(), f"Expected dump file at: {p}") self.assertTrue(p.is_file(), f"Expected dump file at: {p}")
def test_data_restored(self) -> None: def test_data_restored(self) -> None:
@@ -149,7 +160,7 @@ class TestE2EMariaDBFull(unittest.TestCase):
"sh", "sh",
"-lc", "-lc",
f"mariadb -h 127.0.0.1 -u{self.db_user} -p{self.db_password} " f"mariadb -h 127.0.0.1 -u{self.db_user} -p{self.db_password} "
f"-N -e \"SELECT v FROM {self.db_name}.t WHERE id=1;\"", f'-N -e "SELECT v FROM {self.db_name}.t WHERE id=1;"',
] ]
) )
self.assertEqual((p.stdout or "").strip(), "ok") self.assertEqual((p.stdout or "").strip(), "ok")

View File

@@ -60,8 +60,12 @@ class TestE2EMariaDBNoCopy(unittest.TestCase):
] ]
) )
wait_for_mariadb(cls.db_container, root_password=cls.root_password, timeout_s=90) wait_for_mariadb(
wait_for_mariadb_sql(cls.db_container, user=cls.db_user, password=cls.db_password, timeout_s=90) cls.db_container, root_password=cls.root_password, timeout_s=90
)
wait_for_mariadb_sql(
cls.db_container, user=cls.db_user, password=cls.db_password, timeout_s=90
)
# Create table + data (TCP) # Create table + data (TCP)
run( run(
@@ -72,13 +76,16 @@ class TestE2EMariaDBNoCopy(unittest.TestCase):
"sh", "sh",
"-lc", "-lc",
f"mariadb -h 127.0.0.1 -u{cls.db_user} -p{cls.db_password} " f"mariadb -h 127.0.0.1 -u{cls.db_user} -p{cls.db_password} "
f"-e \"CREATE TABLE {cls.db_name}.t (id INT PRIMARY KEY, v VARCHAR(50)); " f'-e "CREATE TABLE {cls.db_name}.t (id INT PRIMARY KEY, v VARCHAR(50)); '
f"INSERT INTO {cls.db_name}.t VALUES (1,'ok');\"", f"INSERT INTO {cls.db_name}.t VALUES (1,'ok');\"",
] ]
) )
cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv" cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv"
write_databases_csv(cls.databases_csv, [(cls.db_container, cls.db_name, cls.db_user, cls.db_password)]) write_databases_csv(
cls.databases_csv,
[(cls.db_container, cls.db_name, cls.db_user, cls.db_password)],
)
# dump-only => no files # dump-only => no files
backup_run( backup_run(
@@ -102,7 +109,7 @@ class TestE2EMariaDBNoCopy(unittest.TestCase):
"sh", "sh",
"-lc", "-lc",
f"mariadb -h 127.0.0.1 -u{cls.db_user} -p{cls.db_password} " f"mariadb -h 127.0.0.1 -u{cls.db_user} -p{cls.db_password} "
f"-e \"DROP TABLE {cls.db_name}.t;\"", f'-e "DROP TABLE {cls.db_name}.t;"',
] ]
) )
@@ -135,7 +142,10 @@ class TestE2EMariaDBNoCopy(unittest.TestCase):
cleanup_docker(containers=cls.containers, volumes=cls.volumes) cleanup_docker(containers=cls.containers, volumes=cls.volumes)
def test_files_backup_not_present(self) -> None: def test_files_backup_not_present(self) -> None:
p = backup_path(self.backups_dir, self.repo_name, self.version, self.db_volume) / "files" p = (
backup_path(self.backups_dir, self.repo_name, self.version, self.db_volume)
/ "files"
)
self.assertFalse(p.exists(), f"Did not expect files backup dir at: {p}") self.assertFalse(p.exists(), f"Did not expect files backup dir at: {p}")
def test_data_restored(self) -> None: def test_data_restored(self) -> None:
@@ -147,7 +157,7 @@ class TestE2EMariaDBNoCopy(unittest.TestCase):
"sh", "sh",
"-lc", "-lc",
f"mariadb -h 127.0.0.1 -u{self.db_user} -p{self.db_password} " f"mariadb -h 127.0.0.1 -u{self.db_user} -p{self.db_password} "
f"-N -e \"SELECT v FROM {self.db_name}.t WHERE id=1;\"", f'-N -e "SELECT v FROM {self.db_name}.t WHERE id=1;"',
] ]
) )
self.assertEqual((p.stdout or "").strip(), "ok") self.assertEqual((p.stdout or "").strip(), "ok")

View File

@@ -33,26 +33,42 @@ class TestE2EPostgresFull(unittest.TestCase):
run(["docker", "volume", "create", cls.pg_volume]) run(["docker", "volume", "create", cls.pg_volume])
run([ run(
"docker", "run", "-d", [
"--name", cls.pg_container, "docker",
"-e", "POSTGRES_PASSWORD=pgpw", "run",
"-e", "POSTGRES_DB=appdb", "-d",
"-e", "POSTGRES_USER=postgres", "--name",
"-v", f"{cls.pg_volume}:/var/lib/postgresql/data", cls.pg_container,
"postgres:16", "-e",
]) "POSTGRES_PASSWORD=pgpw",
"-e",
"POSTGRES_DB=appdb",
"-e",
"POSTGRES_USER=postgres",
"-v",
f"{cls.pg_volume}:/var/lib/postgresql/data",
"postgres:16",
]
)
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90) wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)
# Create a table + data # Create a table + data
run([ run(
"docker", "exec", cls.pg_container, [
"sh", "-lc", "docker",
"psql -U postgres -d appdb -c \"CREATE TABLE t (id int primary key, v text); INSERT INTO t VALUES (1,'ok');\"", "exec",
]) cls.pg_container,
"sh",
"-lc",
"psql -U postgres -d appdb -c \"CREATE TABLE t (id int primary key, v text); INSERT INTO t VALUES (1,'ok');\"",
]
)
cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv" cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv"
write_databases_csv(cls.databases_csv, [(cls.pg_container, "appdb", "postgres", "pgpw")]) write_databases_csv(
cls.databases_csv, [(cls.pg_container, "appdb", "postgres", "pgpw")]
)
backup_run( backup_run(
backups_dir=cls.backups_dir, backups_dir=cls.backups_dir,
@@ -66,37 +82,62 @@ class TestE2EPostgresFull(unittest.TestCase):
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name) cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)
# Wipe schema # Wipe schema
run([ run(
"docker", "exec", cls.pg_container, [
"sh", "-lc", "docker",
"psql -U postgres -d appdb -c \"DROP TABLE t;\"", "exec",
]) cls.pg_container,
"sh",
"-lc",
'psql -U postgres -d appdb -c "DROP TABLE t;"',
]
)
# Restore # Restore
run([ run(
"baudolo-restore", "postgres", [
cls.pg_volume, cls.hash, cls.version, "baudolo-restore",
"--backups-dir", cls.backups_dir, "postgres",
"--repo-name", cls.repo_name, cls.pg_volume,
"--container", cls.pg_container, cls.hash,
"--db-name", "appdb", cls.version,
"--db-user", "postgres", "--backups-dir",
"--db-password", "pgpw", cls.backups_dir,
"--empty", "--repo-name",
]) cls.repo_name,
"--container",
cls.pg_container,
"--db-name",
"appdb",
"--db-user",
"postgres",
"--db-password",
"pgpw",
"--empty",
]
)
@classmethod @classmethod
def tearDownClass(cls) -> None: def tearDownClass(cls) -> None:
cleanup_docker(containers=cls.containers, volumes=cls.volumes) cleanup_docker(containers=cls.containers, volumes=cls.volumes)
def test_dump_file_exists(self) -> None: def test_dump_file_exists(self) -> None:
p = backup_path(self.backups_dir, self.repo_name, self.version, self.pg_volume) / "sql" / "appdb.backup.sql" p = (
backup_path(self.backups_dir, self.repo_name, self.version, self.pg_volume)
/ "sql"
/ "appdb.backup.sql"
)
self.assertTrue(p.is_file(), f"Expected dump file at: {p}") self.assertTrue(p.is_file(), f"Expected dump file at: {p}")
def test_data_restored(self) -> None: def test_data_restored(self) -> None:
p = run([ p = run(
"docker", "exec", self.pg_container, [
"sh", "-lc", "docker",
"psql -U postgres -d appdb -t -c \"SELECT v FROM t WHERE id=1;\"", "exec",
]) self.pg_container,
"sh",
"-lc",
'psql -U postgres -d appdb -t -c "SELECT v FROM t WHERE id=1;"',
]
)
self.assertEqual((p.stdout or "").strip(), "ok") self.assertEqual((p.stdout or "").strip(), "ok")

View File

@@ -32,25 +32,41 @@ class TestE2EPostgresNoCopy(unittest.TestCase):
cls.volumes = [cls.pg_volume] cls.volumes = [cls.pg_volume]
run(["docker", "volume", "create", cls.pg_volume]) run(["docker", "volume", "create", cls.pg_volume])
run([ run(
"docker", "run", "-d", [
"--name", cls.pg_container, "docker",
"-e", "POSTGRES_PASSWORD=pgpw", "run",
"-e", "POSTGRES_DB=appdb", "-d",
"-e", "POSTGRES_USER=postgres", "--name",
"-v", f"{cls.pg_volume}:/var/lib/postgresql/data", cls.pg_container,
"postgres:16", "-e",
]) "POSTGRES_PASSWORD=pgpw",
"-e",
"POSTGRES_DB=appdb",
"-e",
"POSTGRES_USER=postgres",
"-v",
f"{cls.pg_volume}:/var/lib/postgresql/data",
"postgres:16",
]
)
wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90) wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)
run([ run(
"docker", "exec", cls.pg_container, [
"sh", "-lc", "docker",
"psql -U postgres -d appdb -c \"CREATE TABLE t (id int primary key, v text); INSERT INTO t VALUES (1,'ok');\"", "exec",
]) cls.pg_container,
"sh",
"-lc",
"psql -U postgres -d appdb -c \"CREATE TABLE t (id int primary key, v text); INSERT INTO t VALUES (1,'ok');\"",
]
)
cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv" cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv"
write_databases_csv(cls.databases_csv, [(cls.pg_container, "appdb", "postgres", "pgpw")]) write_databases_csv(
cls.databases_csv, [(cls.pg_container, "appdb", "postgres", "pgpw")]
)
backup_run( backup_run(
backups_dir=cls.backups_dir, backups_dir=cls.backups_dir,
@@ -64,36 +80,60 @@ class TestE2EPostgresNoCopy(unittest.TestCase):
cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name) cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)
run([ run(
"docker", "exec", cls.pg_container, [
"sh", "-lc", "docker",
"psql -U postgres -d appdb -c \"DROP TABLE t;\"", "exec",
]) cls.pg_container,
"sh",
"-lc",
'psql -U postgres -d appdb -c "DROP TABLE t;"',
]
)
run([ run(
"baudolo-restore", "postgres", [
cls.pg_volume, cls.hash, cls.version, "baudolo-restore",
"--backups-dir", cls.backups_dir, "postgres",
"--repo-name", cls.repo_name, cls.pg_volume,
"--container", cls.pg_container, cls.hash,
"--db-name", "appdb", cls.version,
"--db-user", "postgres", "--backups-dir",
"--db-password", "pgpw", cls.backups_dir,
"--empty", "--repo-name",
]) cls.repo_name,
"--container",
cls.pg_container,
"--db-name",
"appdb",
"--db-user",
"postgres",
"--db-password",
"pgpw",
"--empty",
]
)
@classmethod @classmethod
def tearDownClass(cls) -> None: def tearDownClass(cls) -> None:
cleanup_docker(containers=cls.containers, volumes=cls.volumes) cleanup_docker(containers=cls.containers, volumes=cls.volumes)
def test_files_backup_not_present(self) -> None: def test_files_backup_not_present(self) -> None:
p = backup_path(self.backups_dir, self.repo_name, self.version, self.pg_volume) / "files" p = (
backup_path(self.backups_dir, self.repo_name, self.version, self.pg_volume)
/ "files"
)
self.assertFalse(p.exists(), f"Did not expect files backup dir at: {p}") self.assertFalse(p.exists(), f"Did not expect files backup dir at: {p}")
def test_data_restored(self) -> None: def test_data_restored(self) -> None:
p = run([ p = run(
"docker", "exec", self.pg_container, [
"sh", "-lc", "docker",
"psql -U postgres -d appdb -t -c \"SELECT v FROM t WHERE id=1;\"", "exec",
]) self.pg_container,
"sh",
"-lc",
'psql -U postgres -d appdb -t -c "SELECT v FROM t WHERE id=1;"',
]
)
self.assertEqual((p.stdout or "").strip(), "ok") self.assertEqual((p.stdout or "").strip(), "ok")

View File

View File

@@ -0,0 +1,90 @@
import csv
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
def run_seed(
csv_path: Path, instance: str, database: str, username: str, password: str = ""
) -> subprocess.CompletedProcess:
# Run the real CLI module (integration-style).
return subprocess.run(
[
sys.executable,
"-m",
"baudolo.seed",
str(csv_path),
instance,
database,
username,
password,
],
text=True,
capture_output=True,
check=True,
)
def read_csv_semicolon(path: Path) -> list[dict]:
with path.open("r", encoding="utf-8", newline="") as f:
reader = csv.DictReader(f, delimiter=";")
return list(reader)
class TestSeedIntegration(unittest.TestCase):
def test_creates_file_and_adds_entry_when_missing(self) -> None:
with tempfile.TemporaryDirectory() as td:
p = Path(td) / "databases.csv"
self.assertFalse(p.exists())
cp = run_seed(p, "docker.test", "appdb", "alice", "secret")
self.assertEqual(cp.returncode, 0, cp.stderr)
self.assertTrue(p.exists())
rows = read_csv_semicolon(p)
self.assertEqual(len(rows), 1)
self.assertEqual(rows[0]["instance"], "docker.test")
self.assertEqual(rows[0]["database"], "appdb")
self.assertEqual(rows[0]["username"], "alice")
self.assertEqual(rows[0]["password"], "secret")
def test_replaces_existing_entry_same_keys(self) -> None:
with tempfile.TemporaryDirectory() as td:
p = Path(td) / "databases.csv"
# First add
run_seed(p, "docker.test", "appdb", "alice", "oldpw")
rows = read_csv_semicolon(p)
self.assertEqual(len(rows), 1)
self.assertEqual(rows[0]["password"], "oldpw")
# Replace (same instance+database+username)
run_seed(p, "docker.test", "appdb", "alice", "newpw")
rows = read_csv_semicolon(p)
self.assertEqual(len(rows), 1, "Expected replacement, not a duplicate row")
self.assertEqual(rows[0]["instance"], "docker.test")
self.assertEqual(rows[0]["database"], "appdb")
self.assertEqual(rows[0]["username"], "alice")
self.assertEqual(rows[0]["password"], "newpw")
def test_database_empty_string_matches_existing_empty_database(self) -> None:
with tempfile.TemporaryDirectory() as td:
p = Path(td) / "databases.csv"
# Add with empty database
run_seed(p, "docker.test", "", "alice", "pw1")
rows = read_csv_semicolon(p)
self.assertEqual(len(rows), 1)
self.assertEqual(rows[0]["database"], "")
# Replace with empty database again
run_seed(p, "docker.test", "", "alice", "pw2")
rows = read_csv_semicolon(p)
self.assertEqual(len(rows), 1)
self.assertEqual(rows[0]["database"], "")
self.assertEqual(rows[0]["password"], "pw2")

View File

@@ -6,7 +6,9 @@ from baudolo.backup.app import requires_stop
class TestRequiresStop(unittest.TestCase): class TestRequiresStop(unittest.TestCase):
@patch("baudolo.backup.app.get_image_info") @patch("baudolo.backup.app.get_image_info")
def test_requires_stop_false_when_all_images_are_whitelisted(self, mock_get_image_info): def test_requires_stop_false_when_all_images_are_whitelisted(
self, mock_get_image_info
):
# All containers use images containing allowed substrings # All containers use images containing allowed substrings
mock_get_image_info.side_effect = [ mock_get_image_info.side_effect = [
"repo/mastodon:v4", "repo/mastodon:v4",
@@ -17,7 +19,9 @@ class TestRequiresStop(unittest.TestCase):
self.assertFalse(requires_stop(containers, whitelist)) self.assertFalse(requires_stop(containers, whitelist))
@patch("baudolo.backup.app.get_image_info") @patch("baudolo.backup.app.get_image_info")
def test_requires_stop_true_when_any_image_is_not_whitelisted(self, mock_get_image_info): def test_requires_stop_true_when_any_image_is_not_whitelisted(
self, mock_get_image_info
):
mock_get_image_info.side_effect = [ mock_get_image_info.side_effect = [
"repo/mastodon:v4", "repo/mastodon:v4",
"repo/nginx:latest", "repo/nginx:latest",