Release version 1.0.0

docs(readme): rewrite README to reflect deterministic backup design
- clarify separation between file backups (always) and SQL dumps (explicit only) - document correct nested backup directory layout - remove legacy script-based usage and outdated sections - add explicit explanation of database definition scope - update usage examples to current baudolo CLI https://chatgpt.com/share/694ef6d2-7584-800f-a32b-27367f234d1d
2025-12-27 11:06:35 +00:00 · 2025-12-27 09:30:38 +01:00 · 2025-12-26 21:57:46 +01:00 · 2025-12-26 21:03:00 +01:00 · 2025-12-26 20:50:29 +01:00 · 2025-12-26 20:43:06 +01:00
10 changed files with 360 additions and 46 deletions
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -0,0 +1,91 @@
+name: CI (make tests, stable, publish)
+
+on:
+  push:
+    branches: ["**"]
+    tags: ["v*.*.*"]   # SemVer tags like v1.2.3
+  pull_request:
+
+permissions:
+  contents: write      # push/update 'stable' tag
+  packages: write      # push to GHCR
+
+env:
+  IMAGE_NAME: baudolo
+  REGISTRY: ghcr.io
+  IMAGE_REPO: ${{ github.repository }}
+
+jobs:
+  test:
+    name: make test
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Show docker info
+        run: |
+          docker version
+          docker info
+
+      - name: Run all tests via Makefile
+        run: |
+          make test
+
+      - name: Upload E2E artifacts (always)
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: e2e-artifacts
+          path: artifacts
+          if-no-files-found: ignore
+
+  stable_and_publish:
+    name: Mark stable + publish image (SemVer tags only)
+    needs: [test]
+    runs-on: ubuntu-latest
+    if: startsWith(github.ref, 'refs/tags/v')
+
+    steps:
+      - name: Checkout (full history for tags)
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Derive version from tag
+        id: ver
+        run: |
+          TAG="${GITHUB_REF#refs/tags/}"   # v1.2.3
+          echo "tag=${TAG}" >> "$GITHUB_OUTPUT"
+
+      - name: Mark 'stable' git tag (force update)
+        run: |
+          git config user.name  "github-actions[bot]"
+          git config user.email "github-actions[bot]@users.noreply.github.com"
+          git tag -f stable "${GITHUB_SHA}"
+          git push -f origin stable
+
+      - name: Login to GHCR
+        uses: docker/login-action@v3
+        with:
+          registry: ${{ env.REGISTRY }}
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Build image (Makefile)
+        run: |
+          make build
+
+      - name: Tag image for registry
+        run: |
+          # local image built by Makefile is: baudolo:local
+          docker tag "${IMAGE_NAME}:local" "${REGISTRY}/${IMAGE_REPO}:${{ steps.ver.outputs.tag }}"
+          docker tag "${IMAGE_NAME}:local" "${REGISTRY}/${IMAGE_REPO}:stable"
+          docker tag "${IMAGE_NAME}:local" "${REGISTRY}/${IMAGE_REPO}:sha-${GITHUB_SHA::12}"
+
+      - name: Push image
+        run: |
+          docker push "${REGISTRY}/${IMAGE_REPO}:${{ steps.ver.outputs.tag }}"
+          docker push "${REGISTRY}/${IMAGE_REPO}:stable"
+          docker push "${REGISTRY}/${IMAGE_REPO}:sha-${GITHUB_SHA::12}"
--- a/.travis.yml
+++ b/.travis.yml
@@ -1,2 +0,0 @@
-language: shell
-script: shellcheck $(find . -type f -name '*.sh')
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -0,0 +1,4 @@
+## [1.0.0] - 2025-12-27
+
+* Official Release 🥳
+
--- a/4
+++ b/4
@@ -0,0 +1,4 @@
+git@github.com:kevinveenbirkenbach/backup-docker-to-local.git
+ssh://git@git.veen.world:2201/kevinveenbirkenbach/backup-docker-to-local.git
+ssh://git@code.infinito.nexus:2201/kevinveenbirkenbach/backup-docker-to-local.git
+https://pypi.org/project/baudolo/
--- a/17
+++ b/17
@@ -1,4 +1,5 @@
-.PHONY: install build test-e2e
+.PHONY: install build \
+		test-e2e test test-unit test-integration

 # Default python if no venv is active
 PY_DEFAULT ?= python3
@@ -41,4 +42,16 @@ clean:
 # - loads the freshly built image into DinD
 # - runs the unittest suite inside a container that talks to DinD via DOCKER_HOST
 test-e2e: clean build
-	@bash scripts/test-e2e.sh
+	@bash scripts/test-e2e.sh
+
+test: test-unit test-integration test-e2e
+
+test-unit: clean build
+	@echo ">> Running unit tests"
+	@docker run --rm -t $(IMAGE) \
+	  sh -lc 'python -m unittest discover -t . -s tests/unit -p "test_*.py" -v'
+
+test-integration: clean build
+	@echo ">> Running integration tests"
+	@docker run --rm -t $(IMAGE) \
+	  sh -lc 'python -m unittest discover -t . -s tests/integration -p "test_*.py" -v'
--- a/README.md
+++ b/README.md
@@ -1,80 +1,196 @@
-# Backup Docker Volumes to Local (baudolo) 📦🔄
-[![GitHub Sponsors](https://img.shields.io/badge/Sponsor-GitHub%20Sponsors-blue?logo=github)](https://github.com/sponsors/kevinveenbirkenbach) [![Patreon](https://img.shields.io/badge/Support-Patreon-orange?logo=patreon)](https://www.patreon.com/c/kevinveenbirkenbach) [![Buy Me a Coffee](https://img.shields.io/badge/Buy%20me%20a%20Coffee-Funding-yellow?logo=buymeacoffee)](https://buymeacoffee.com/kevinveenbirkenbach) [![PayPal](https://img.shields.io/badge/Donate-PayPal-blue?logo=paypal)](https://s.veen.world/paypaldonate)
+# baudolo – Deterministic Backup & Restore for Docker Volumes 📦🔄
+[![GitHub Sponsors](https://img.shields.io/badge/Sponsor-GitHub%20Sponsors-blue?logo=github)](https://github.com/sponsors/kevinveenbirkenbach) [![Patreon](https://img.shields.io/badge/Support-Patreon-orange?logo=patreon)](https://www.patreon.com/c/kevinveenbirkenbach) [![Buy Me a Coffee](https://img.shields.io/badge/Buy%20me%20a%20Coffee-Funding-yellow?logo=buymeacoffee)](https://buymeacoffee.com/kevinveenbirkenbach) [![PayPal](https://img.shields.io/badge/Donate-PayPal-blue?logo=paypal)](https://s.veen.world/paypaldonate) [![License: AGPL v3](https://img.shields.io/badge/License-AGPL%20v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0) [![Docker Version](https://img.shields.io/badge/Docker-Yes-blue.svg)](https://www.docker.com) [![Python Version](https://img.shields.io/badge/Python-3.x-blue.svg)](https://www.python.org) [![GitHub stars](https://img.shields.io/github/stars/kevinveenbirkenbach/backup-docker-to-local.svg?style=social)](https://github.com/kevinveenbirkenbach/backup-docker-to-local/stargazers)


-**Backup Docker Volumes to Local** is a set of Python and shell scripts that enable you to perform incremental backups of all your Docker volumes using rsync. It is designed to integrate seamlessly with [Kevin's Package Manager](https://github.com/kevinveenbirkenbach/package-manager) under the alias **baudolo**, making it easy to install and manage. The tool supports both file and database recoveries with a clear, automated backup scheme.
+`baudolo` is a backup and restore system for Docker volumes with
+**mandatory file backups** and **explicit, deterministic database dumps**.
+It is designed for environments with many Docker services where:
+- file-level backups must always exist
+- database dumps must be intentional, predictable, and auditable

-[![License: AGPL v3](https://img.shields.io/badge/License-AGPL%20v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0) [![Docker Version](https://img.shields.io/badge/Docker-Yes-blue.svg)](https://www.docker.com) [![Python Version](https://img.shields.io/badge/Python-3.x-blue.svg)](https://www.python.org) [![GitHub stars](https://img.shields.io/github/stars/kevinveenbirkenbach/backup-docker-to-local.svg?style=social)](https://github.com/kevinveenbirkenbach/backup-docker-to-local/stargazers)
+## ✨ Key Features

-## 🎯 Goal
+- 📦 Incremental Docker volume backups using `rsync --link-dest`
+- 🗄 Optional SQL dumps for:
+  - PostgreSQL
+  - MariaDB / MySQL
+- 🌱 Explicit database definition for SQL backups (no auto-discovery)
+- 🧾 Backup integrity stamping via `dirval` (Python API)
+- ⏸ Automatic container stop/start when required for consistency
+- 🚫 Whitelisting of containers that do not require stopping
+- ♻️ Modular, maintainable Python architecture

-This project automates the backup of Docker volumes using incremental backups (rsync) and supports recovering both files and database dumps (MariaDB/PostgreSQL). A robust directory stamping mechanism ensures data integrity, and the tool also handles restarting Docker Compose services when necessary.

-## 🚀 Features
+## 🧠 Core Concept (Important!)

- **Incremental Backups:** Uses rsync with `--link-dest` for efficient, versioned backups.
- **Database Backup Support:** Backs up MariaDB and PostgreSQL databases from running containers.
- **Volume Recovery:** Provides scripts to recover volumes and databases from backups.
- **Docker Compose Integration:** Option to automatically restart Docker Compose services after backup.
- **Flexible Configuration:** Easily integrated with your Docker environment with minimal setup.
- **Comprehensive Logging:** Detailed command output and error handling for safe operations.
+`baudolo` **separates file backups from database dumps**.

-## 🛠 Requirements
+- **Docker volumes are always backed up at file level**
+- **SQL dumps are created only for explicitly defined databases**

- **Linux Operating System** (with Docker installed) 🐧
- **Python 3.x** 🐍
- **Docker & Docker Compose** 🔧
- **rsync** installed on your system
+This results in the following behavior:

-## 📥 Installation
+| Database defined | File backup | SQL dump |
+|------------------|-------------|----------|
+| No               | ✔ yes       | ✘ no     |
+| Yes              | ✔ yes       | ✔ yes    |

-You can install **Backup Docker Volumes to Local** easily via [Kevin's Package Manager](https://github.com/kevinveenbirkenbach/package-manager) using the alias **baudolo**:
+## 📁 Backup Layout

-```bash
-pkgmgr install baudolo
+Backups are stored in a deterministic, fully nested structure:
+
+```text
+<backups-dir>/
+└── <machine-hash>/
+    └── <repo-name>/
+        └── <timestamp>/
+            └── <volume-name>/
+                ├── files/
+                └── sql/
+                    └── <database>.backup.sql
 ```

-Alternatively, clone the repository directly:
+### Meaning of each level
+
+* `<machine-hash>`
+  SHA256 hash of `/etc/machine-id` (host separation)
+
+* `<repo-name>`
+  Logical backup namespace (project / stack)
+
+* `<timestamp>`
+  Backup generation (`YYYYMMDDHHMMSS`)
+
+* `<volume-name>`
+  Docker volume name
+
+* `files/`
+  Incremental file backup (rsync)
+
+* `sql/`
+  Optional SQL dumps (only for defined databases)
+
+## 🚀 Installation
+
+### Local (editable install)

 ```bash
-git clone https://github.com/kevinveenbirkenbach/backup-docker-to-local.git
-cd backup-docker-to-local
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -e .
 ```

-## 🚀 Usage
+## 🌱 Database Definition (SQL Backup Scope)

-### Backup All Volumes
+### How SQL backups are defined

-To backup all Docker volumes, simply run:
+`baudolo` creates SQL dumps **only** for databases that are **explicitly defined**
+via configuration (e.g. a databases definition file or seeding step).
+
+If a database is **not defined**:
+
+* its Docker volume is still backed up (files)
+* **no SQL dump is created**
+
+> No database definition → file backup only
+> Database definition present → file backup + SQL dump
+
+### Why explicit definition?
+
+`baudolo` does **not** inspect running containers to guess databases.
+
+Databases must be explicitly defined to guarantee:
+
+* deterministic backups
+* predictable restore behavior
+* reproducible environments
+* zero accidental production data exposure
+
+### Required database metadata
+
+Each database definition provides:
+
+* database instance (container or logical instance)
+* database name
+* database user
+* database password
+
+This information is used by `baudolo` to execute
+`pg_dump`, `pg_dumpall`, or `mariadb-dump`.
+
+## 💾 Running a Backup

 ```bash
-./backup-docker-to-local.sh
+baudolo \
+  --compose-dir /srv/docker \
+  --databases-csv /etc/baudolo/databases.csv \
+  --database-containers central-postgres central-mariadb \
+  --images-no-stop-required alpine postgres mariadb mysql \
+  --images-no-backup-required redis busybox
 ```

-### Recovery
+### Common Backup Flags

-#### Recover Volume Files
+| Flag            | Description                                 |
+| --------------- | ------------------------------------------- |
+| `--everything`  | Always stop containers and re-run rsync     |
+| `--dump-only`   | Only create SQL dumps, skip file backups    |
+| `--shutdown`    | Do not restart containers after backup      |
+| `--backups-dir` | Backup root directory (default: `/Backups`) |
+| `--repo-name`   | Backup namespace under machine hash         |
+
+## ♻️ Restore Operations
+
+### Restore Volume Files

 ```bash
-bash ./recover-docker-from-local.sh "{{volume_name}}" "$(sha256sum /etc/machine-id | head -c 64)" "{{version_to_recover}}"
+baudolo-restore files \
+  my-volume \
+  <machine-hash> \
+  <version> \
+  --backups-dir /Backups \
+  --repo-name my-repo
 ```

-#### Recover Database
-
-For example, to recover a MySQL/MariaDB database:
+Restore into a **different target volume**:

 ```bash
-docker exec -i mysql_container mysql -uroot -psecret database < db.sql
+baudolo-restore files \
+  target-volume \
+  <machine-hash> \
+  <version> \
+  --source-volume source-volume
 ```

-#### Debug Mode
-
-To inspect what’s happening inside a container:
+### Restore PostgreSQL

 ```bash
-docker run -it --entrypoint /bin/sh --rm --volumes-from {{container_name}} -v /Backups/:/Backups/ kevinveenbirkenbach/alpine-rsync
+baudolo-restore postgres \
+  my-volume \
+  <machine-hash> \
+  <version> \
+  --container postgres \
+  --db-name appdb \
+  --db-password secret \
+  --empty
 ```

+### Restore MariaDB / MySQL
+
+```bash
+baudolo-restore mariadb \
+  my-volume \
+  <machine-hash> \
+  <version> \
+  --container mariadb \
+  --db-name shopdb \
+  --db-password secret \
+  --empty
+```
+
+> `baudolo` automatically detects whether `mariadb` or `mysql`
+> is available inside the container
+
 ## 🔍 Backup Scheme

 The backup mechanism uses incremental backups with rsync and stamps directories with a unique hash. For more details on the backup scheme, check out [this blog post](https://blog.veen.world/blog/2020/12/26/how-i-backup-dedicated-root-servers/).  
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"

 [project]
 name = "backup-docker-to-local"
-version = "0.1.0"
+version = "1.0.0"
 description = "Backup Docker volumes to local with rsync and optional DB dumps."
 readme = "README.md"
 requires-python = ">=3.9"
@@ -19,7 +19,7 @@ dependencies = [
 [project.scripts]
 baudolo = "baudolo.backup.__main__:main"
 baudolo-restore = "baudolo.restore.__main__:main"
-baudolo-configure = "baudolo.configure.__main__:main"
+baudolo-seed = "baudolo.seed.__main__:main"

 [tool.setuptools]
 package-dir = { "" = "src" }
--- a/src/baudolo/configure/main.py
+++ b/src/baudolo/configure/main.py
--- a/tests/integration/init.py
+++ b/tests/integration/init.py
--- a/tests/integration/test_seed_integration.py
+++ b/tests/integration/test_seed_integration.py
@@ -0,0 +1,88 @@
+import csv
+import subprocess
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+
+def run_seed(csv_path: Path, instance: str, database: str, username: str, password: str = "") -> subprocess.CompletedProcess:
+    # Run the real CLI module (integration-style).
+    return subprocess.run(
+        [
+            sys.executable,
+            "-m",
+            "baudolo.seed",
+            str(csv_path),
+            instance,
+            database,
+            username,
+            password,
+        ],
+        text=True,
+        capture_output=True,
+        check=True,
+    )
+
+
+def read_csv_semicolon(path: Path) -> list[dict]:
+    with path.open("r", encoding="utf-8", newline="") as f:
+        reader = csv.DictReader(f, delimiter=";")
+        return list(reader)
+
+
+class TestSeedIntegration(unittest.TestCase):
+    def test_creates_file_and_adds_entry_when_missing(self) -> None:
+        with tempfile.TemporaryDirectory() as td:
+            p = Path(td) / "databases.csv"
+            self.assertFalse(p.exists())
+
+            cp = run_seed(p, "docker.test", "appdb", "alice", "secret")
+
+            self.assertEqual(cp.returncode, 0, cp.stderr)
+            self.assertTrue(p.exists())
+
+            rows = read_csv_semicolon(p)
+            self.assertEqual(len(rows), 1)
+            self.assertEqual(rows[0]["instance"], "docker.test")
+            self.assertEqual(rows[0]["database"], "appdb")
+            self.assertEqual(rows[0]["username"], "alice")
+            self.assertEqual(rows[0]["password"], "secret")
+
+    def test_replaces_existing_entry_same_keys(self) -> None:
+        with tempfile.TemporaryDirectory() as td:
+            p = Path(td) / "databases.csv"
+
+            # First add
+            run_seed(p, "docker.test", "appdb", "alice", "oldpw")
+            rows = read_csv_semicolon(p)
+            self.assertEqual(len(rows), 1)
+            self.assertEqual(rows[0]["password"], "oldpw")
+
+            # Replace (same instance+database+username)
+            run_seed(p, "docker.test", "appdb", "alice", "newpw")
+            rows = read_csv_semicolon(p)
+
+            self.assertEqual(len(rows), 1, "Expected replacement, not a duplicate row")
+            self.assertEqual(rows[0]["instance"], "docker.test")
+            self.assertEqual(rows[0]["database"], "appdb")
+            self.assertEqual(rows[0]["username"], "alice")
+            self.assertEqual(rows[0]["password"], "newpw")
+
+    def test_database_empty_string_matches_existing_empty_database(self) -> None:
+        with tempfile.TemporaryDirectory() as td:
+            p = Path(td) / "databases.csv"
+
+            # Add with empty database
+            run_seed(p, "docker.test", "", "alice", "pw1")
+            rows = read_csv_semicolon(p)
+            self.assertEqual(len(rows), 1)
+            self.assertEqual(rows[0]["database"], "")
+
+            # Replace with empty database again
+            run_seed(p, "docker.test", "", "alice", "pw2")
+            rows = read_csv_semicolon(p)
+
+            self.assertEqual(len(rows), 1)
+            self.assertEqual(rows[0]["database"], "")
+            self.assertEqual(rows[0]["password"], "pw2")
Author	SHA1	Message	Date
Kevin Veen-Birkenbach	3b39a6ef02	Release version 1.0.0	2025-12-27 09:30:38 +01:00
Kevin Veen-Birkenbach	e0b2e8934e	docs(readme): rewrite README to reflect deterministic backup design - clarify separation between file backups (always) and SQL dumps (explicit only) - document correct nested backup directory layout - remove legacy script-based usage and outdated sections - add explicit explanation of database definition scope - update usage examples to current baudolo CLI https://chatgpt.com/share/694ef6d2-7584-800f-a32b-27367f234d1d	2025-12-26 21:57:46 +01:00
Kevin Veen-Birkenbach	bbb2dd1732	Removed .travis	2025-12-26 21:03:00 +01:00
Kevin Veen-Birkenbach	159502af5e	Added mirros	2025-12-26 20:50:29 +01:00
Kevin Veen-Birkenbach	698d1e7a9e	ci: add Makefile-driven CI with unit, integration and e2e tests - add GitHub Actions CI workflow using Makefile targets exclusively - run unit, integration and e2e tests via `make test` - publish Docker image to GHCR on SemVer tags - force-update `stable` git tag after successful release - add integration test for seed CLI (CSV upsert behavior) - extend Makefile with test-unit and test-integration targets https://chatgpt.com/share/694ee54f-b814-800f-a714-e87563e538b7	2025-12-26 20:43:06 +01:00
Kevin Veen-Birkenbach	f8420c8bea	renamed configure to seed	2025-12-26 19:58:39 +01:00