9 Commits

Author SHA1 Message Date
e3f28098bd Release version 1.5.0 2026-01-31 22:06:05 +01:00
babadcb038 fix(backup,ci): make databases.csv optional and upgrade Docker CLI in image
- Handle missing or empty databases.csv gracefully with warnings and empty DataFrame
- Add unit tests for robust databases.csv loading behavior
- Adjust seed tests to assert warnings across multiple print calls
- Replace Debian docker.io with docker-ce-cli to avoid Docker API version mismatch
- Install required build tools (curl, gnupg) for Docker repo setup

https://chatgpt.com/share/697e6d9d-6458-800f-9d12-1e337509be4e
2026-01-31 22:01:12 +01:00
fbfdb8615f **Commit message:**
Fix Docker CLI install, switch test runners to bash, and stabilize unit tests for compose/seed mocks

https://chatgpt.com/share/697e68cd-d22c-800f-9b2e-47ef231b6502
2026-01-31 21:40:39 +01:00
2f5882f5c1 Release version 1.4.0 2026-01-31 18:28:29 +01:00
522391fdd3 Merge branch 'main' of github.com:kevinveenbirkenbach/backup-docker-to-local 2026-01-31 18:25:55 +01:00
b3c9cf5ce1 backup: restart compose stacks via wrapper-aware command resolution
- Prefer `compose` wrapper (if present) when restarting stacks to ensure
  identical file and env resolution as Infinito.Nexus
- Fallback to `docker compose` with explicit detection of:
  - docker-compose.yml
  - docker-compose.override.yml
  - docker-compose.ca.override.yml
  - .env / .env/env via --env-file
- Replace legacy `docker-compose` usage
- Log exact compose commands before execution
- Add unit tests covering wrapper vs fallback behavior

https://chatgpt.com/share/697e3b0c-85d4-800f-91a7-42324599a63c
2026-01-31 18:25:23 +01:00
2ed3472527 Ignored build/ 2026-01-16 10:45:09 +01:00
54737cefa7 Release version 1.3.0 2026-01-10 18:41:55 +01:00
d976640312 fix(seed): handle empty databases.csv and add unit tests
- Gracefully handle empty databases.csv by creating header columns and emitting a warning
- Add _empty_df() helper for consistent DataFrame initialization
- Add unit tests for baudolo-seed including empty-file regression case
- Apply minor formatting fixes across backup and e2e test files

https://chatgpt.com/share/69628f0b-8744-800f-b08d-2633e05167da
2026-01-10 18:40:22 +01:00
19 changed files with 786 additions and 70 deletions

1
.gitignore vendored
View File

@@ -2,3 +2,4 @@ __pycache__
artifacts/ artifacts/
*.egg-info *.egg-info
dist/ dist/
build/

View File

@@ -1,3 +1,19 @@
## [1.5.0] - 2026-01-31
* * Make `databases.csv` optional: missing or empty files now emit warnings and no longer break backups
* Fix Docker CLI compatibility by switching to `docker-ce-cli` and required build tools
## [1.4.0] - 2026-01-31
* Baudolo now restarts Docker Compose stacks in a wrapper-aware way (with a `docker compose` fallback), ensuring that all Compose overrides and env files are applied identically to the Infinito.Nexus workflow.
## [1.3.0] - 2026-01-10
* Empty databases.csv no longer causes baudolo-seed to fail
## [1.2.0] - 2025-12-29 ## [1.2.0] - 2025-12-29
* * Introduced **`--dump-only-sql`** mode for reliable, SQL-only database backups (replaces `--dump-only`). * * Introduced **`--dump-only-sql`** mode for reliable, SQL-only database backups (replaces `--dump-only`).

View File

@@ -3,32 +3,35 @@ FROM python:3.11-slim
WORKDIR /app WORKDIR /app
# Runtime + build essentials: # Base deps for build/runtime + docker repo key
# - rsync: required for file backup/restore
# - ca-certificates: TLS
# - docker-cli: needed if you want to control the host Docker engine (via /var/run/docker.sock mount)
# - make: to delegate install logic to Makefile
#
# Notes:
# - On Debian slim, the docker client package is typically "docker.io".
# - If you only want restore-without-docker, you can drop docker.io later.
RUN apt-get update && apt-get install -y --no-install-recommends \ RUN apt-get update && apt-get install -y --no-install-recommends \
make \ make \
rsync \ rsync \
ca-certificates \ ca-certificates \
docker-cli \ bash \
curl \
gnupg \
&& rm -rf /var/lib/apt/lists/* && rm -rf /var/lib/apt/lists/*
# Install Docker CLI (docker-ce-cli) from Docker's official apt repo
RUN bash -lc "set -euo pipefail \
&& install -m 0755 -d /etc/apt/keyrings \
&& curl -fsSL https://download.docker.com/linux/debian/gpg \
| gpg --dearmor -o /etc/apt/keyrings/docker.gpg \
&& chmod a+r /etc/apt/keyrings/docker.gpg \
&& . /etc/os-release \
&& echo \"deb [arch=\$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian \${VERSION_CODENAME} stable\" \
> /etc/apt/sources.list.d/docker.list \
&& apt-get update \
&& apt-get install -y --no-install-recommends docker-ce-cli \
&& rm -rf /var/lib/apt/lists/*"
# Fail fast if docker client is missing # Fail fast if docker client is missing
RUN docker version || true
RUN command -v docker RUN command -v docker
COPY . . COPY . .
# All install decisions are handled by the Makefile.
RUN make install RUN make install
# Sensible defaults (can be overridden at runtime)
ENV PYTHONUNBUFFERED=1 ENV PYTHONUNBUFFERED=1
# Default: show CLI help
CMD ["baudolo", "--help"] CMD ["baudolo", "--help"]

View File

@@ -49,9 +49,9 @@ test: test-unit test-integration test-e2e
test-unit: clean build test-unit: clean build
@echo ">> Running unit tests" @echo ">> Running unit tests"
@docker run --rm -t $(IMAGE) \ @docker run --rm -t $(IMAGE) \
sh -lc 'python -m unittest discover -t . -s tests/unit -p "test_*.py" -v' bash -lc 'python -m unittest discover -t . -s tests/unit -p "test_*.py" -v'
test-integration: clean build test-integration: clean build
@echo ">> Running integration tests" @echo ">> Running integration tests"
@docker run --rm -t $(IMAGE) \ @docker run --rm -t $(IMAGE) \
sh -lc 'python -m unittest discover -t . -s tests/integration -p "test_*.py" -v' bash -lc 'python -m unittest discover -t . -s tests/integration -p "test_*.py" -v'

View File

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
[project] [project]
name = "backup-docker-to-local" name = "backup-docker-to-local"
version = "1.2.0" version = "1.5.0"
description = "Backup Docker volumes to local with rsync and optional DB dumps." description = "Backup Docker volumes to local with rsync and optional DB dumps."
readme = "README.md" readme = "README.md"
requires-python = ">=3.9" requires-python = ">=3.9"

View File

@@ -83,7 +83,7 @@ dump_debug() {
docker -H "${DIND_HOST}" run --rm \ docker -H "${DIND_HOST}" run --rm \
-v "${E2E_TMP_VOL}:/tmp" \ -v "${E2E_TMP_VOL}:/tmp" \
alpine:3.20 \ alpine:3.20 \
sh -lc 'cd /tmp && tar -czf /out.tar.gz . || true' \ bash -lc 'cd /tmp && tar -czf /out.tar.gz . || true' \
>/dev/null 2>&1 || true >/dev/null 2>&1 || true
# The above writes inside the container FS, not to host. So do it properly: # The above writes inside the container FS, not to host. So do it properly:
@@ -91,7 +91,7 @@ dump_debug() {
local tmpc="baudolo-e2e-tmpdump-${TS}" local tmpc="baudolo-e2e-tmpdump-${TS}"
docker -H "${DIND_HOST}" rm -f "${tmpc}" >/dev/null 2>&1 || true docker -H "${DIND_HOST}" rm -f "${tmpc}" >/dev/null 2>&1 || true
docker -H "${DIND_HOST}" create --name "${tmpc}" -v "${E2E_TMP_VOL}:/tmp" alpine:3.20 \ docker -H "${DIND_HOST}" create --name "${tmpc}" -v "${E2E_TMP_VOL}:/tmp" alpine:3.20 \
sh -lc 'cd /tmp && tar -czf /tmpdump.tar.gz . || true' >/dev/null bash -lc 'cd /tmp && tar -czf /tmpdump.tar.gz . || true' >/dev/null
docker -H "${DIND_HOST}" start -a "${tmpc}" >/dev/null 2>&1 || true docker -H "${DIND_HOST}" start -a "${tmpc}" >/dev/null 2>&1 || true
docker -H "${DIND_HOST}" cp "${tmpc}:/tmpdump.tar.gz" "${ARTIFACTS_DIR}/e2e-tmp-${TS}.tar.gz" >/dev/null 2>&1 || true docker -H "${DIND_HOST}" cp "${tmpc}:/tmpdump.tar.gz" "${ARTIFACTS_DIR}/e2e-tmp-${TS}.tar.gz" >/dev/null 2>&1 || true
docker -H "${DIND_HOST}" rm -f "${tmpc}" >/dev/null 2>&1 || true docker -H "${DIND_HOST}" rm -f "${tmpc}" >/dev/null 2>&1 || true
@@ -187,7 +187,7 @@ if [ "${DEBUG_SHELL}" = "1" ]; then
-v "${DIND_VOL}:/var/lib/docker:ro" \ -v "${DIND_VOL}:/var/lib/docker:ro" \
-v "${E2E_TMP_VOL}:/tmp" \ -v "${E2E_TMP_VOL}:/tmp" \
"${IMG}" \ "${IMG}" \
sh -lc ' bash -lc '
set -e set -e
if [ ! -f /etc/machine-id ]; then if [ ! -f /etc/machine-id ]; then
mkdir -p /etc mkdir -p /etc
@@ -195,7 +195,7 @@ if [ "${DEBUG_SHELL}" = "1" ]; then
fi fi
echo ">> DOCKER_HOST=${DOCKER_HOST}" echo ">> DOCKER_HOST=${DOCKER_HOST}"
docker ps -a || true docker ps -a || true
exec sh exec bash
' '
rc=$? rc=$?
else else
@@ -206,7 +206,7 @@ else
-v "${DIND_VOL}:/var/lib/docker:ro" \ -v "${DIND_VOL}:/var/lib/docker:ro" \
-v "${E2E_TMP_VOL}:/tmp" \ -v "${E2E_TMP_VOL}:/tmp" \
"${IMG}" \ "${IMG}" \
sh -lc ' bash -lc '
set -euo pipefail set -euo pipefail
set -x set -x
export PYTHONUNBUFFERED=1 export PYTHONUNBUFFERED=1

View File

@@ -2,10 +2,12 @@ from __future__ import annotations
import os import os
import pathlib import pathlib
import sys
from datetime import datetime from datetime import datetime
import pandas import pandas
from dirval import create_stamp_file from dirval import create_stamp_file
from pandas.errors import EmptyDataError
from .cli import parse_args from .cli import parse_args
from .compose import handle_docker_compose_services from .compose import handle_docker_compose_services
@@ -72,6 +74,7 @@ def requires_stop(containers: list[str], images_no_stop_required: list[str]) ->
return True return True
return False return False
def backup_mariadb_or_postgres( def backup_mariadb_or_postgres(
*, *,
container: str, container: str,
@@ -95,6 +98,42 @@ def backup_mariadb_or_postgres(
return False, False return False, False
def _empty_databases_df() -> "pandas.DataFrame":
"""
Create an empty DataFrame with the expected schema for databases.csv.
This allows the backup to continue without DB dumps when the CSV is missing
or empty (pandas EmptyDataError).
"""
return pandas.DataFrame(columns=["instance", "database", "username", "password"])
def _load_databases_df(csv_path: str) -> "pandas.DataFrame":
"""
Load databases.csv robustly.
- Missing file -> warn, continue with empty df
- Empty file -> warn, continue with empty df
- Valid CSV -> return dataframe
"""
try:
return pandas.read_csv(csv_path, sep=";", keep_default_na=False, dtype=str)
except FileNotFoundError:
print(
f"WARNING: databases.csv not found: {csv_path}. Continuing without database dumps.",
file=sys.stderr,
flush=True,
)
return _empty_databases_df()
except EmptyDataError:
print(
f"WARNING: databases.csv exists but is empty: {csv_path}. Continuing without database dumps.",
file=sys.stderr,
flush=True,
)
return _empty_databases_df()
def _backup_dumps_for_volume( def _backup_dumps_for_volume(
*, *,
containers: list[str], containers: list[str],
@@ -135,9 +174,10 @@ def main() -> int:
# IMPORTANT: # IMPORTANT:
# - keep_default_na=False prevents empty fields from turning into NaN # - keep_default_na=False prevents empty fields from turning into NaN
# - dtype=str keeps all columns stable for comparisons/validation # - dtype=str keeps all columns stable for comparisons/validation
databases_df = pandas.read_csv( #
args.databases_csv, sep=";", keep_default_na=False, dtype=str # Robust behavior:
) # - if the file is missing or empty, we continue without DB dumps.
databases_df = _load_databases_df(args.databases_csv)
print("💾 Start volume backups...", flush=True) print("💾 Start volume backups...", flush=True)
@@ -167,7 +207,8 @@ def main() -> int:
if found_db: if found_db:
if not dumped_any: if not dumped_any:
print( print(
f"WARNING: dump-only-sql requested but no DB dump was produced for DB volume '{volume_name}'. Falling back to file backup.", f"WARNING: dump-only-sql requested but no DB dump was produced for DB volume '{volume_name}'. "
"Falling back to file backup.",
flush=True, flush=True,
) )
# fall through to file backup below # fall through to file backup below

View File

@@ -1,13 +1,104 @@
from __future__ import annotations from __future__ import annotations
import os import os
import shutil
import subprocess import subprocess
from pathlib import Path
from typing import List, Optional
def _detect_env_file(project_dir: Path) -> Optional[Path]:
"""
Detect Compose env file in a directory.
Preference (same as Infinito.Nexus wrapper):
1) <dir>/.env (file)
2) <dir>/.env/env (file) (legacy layout)
"""
c1 = project_dir / ".env"
if c1.is_file():
return c1
c2 = project_dir / ".env" / "env"
if c2.is_file():
return c2
return None
def _detect_compose_files(project_dir: Path) -> List[Path]:
"""
Detect Compose file stack in a directory (same as Infinito.Nexus wrapper).
Always requires docker-compose.yml.
Optionals:
- docker-compose.override.yml
- docker-compose.ca.override.yml
"""
base = project_dir / "docker-compose.yml"
if not base.is_file():
raise FileNotFoundError(f"Missing docker-compose.yml in: {project_dir}")
files = [base]
override = project_dir / "docker-compose.override.yml"
if override.is_file():
files.append(override)
ca_override = project_dir / "docker-compose.ca.override.yml"
if ca_override.is_file():
files.append(ca_override)
return files
def _compose_wrapper_path() -> Optional[str]:
"""
Prefer the Infinito.Nexus compose wrapper if present.
Equivalent to: `which compose`
"""
return shutil.which("compose")
def _build_compose_cmd(project_dir: str, passthrough: List[str]) -> List[str]:
"""
Build the compose command for this project directory.
Behavior:
- If `compose` wrapper exists: use it with --chdir (so it resolves -f/--env-file itself)
- Else: use `docker compose` and replicate wrapper's file/env detection.
"""
pdir = Path(project_dir).resolve()
wrapper = _compose_wrapper_path()
if wrapper:
# Wrapper defaults project name to basename of --chdir.
# "--" ensures wrapper stops parsing its own args.
return [wrapper, "--chdir", str(pdir), "--", *passthrough]
# Fallback: pure docker compose, but mirror wrapper behavior.
files = _detect_compose_files(pdir)
env_file = _detect_env_file(pdir)
cmd: List[str] = ["docker", "compose"]
for f in files:
cmd += ["-f", str(f)]
if env_file:
cmd += ["--env-file", str(env_file)]
cmd += passthrough
return cmd
def hard_restart_docker_services(dir_path: str) -> None: def hard_restart_docker_services(dir_path: str) -> None:
print(f"Hard restart docker-compose services in: {dir_path}", flush=True) print(f"Hard restart compose services in: {dir_path}", flush=True)
subprocess.run(["docker-compose", "down"], cwd=dir_path, check=True)
subprocess.run(["docker-compose", "up", "-d"], cwd=dir_path, check=True) down_cmd = _build_compose_cmd(dir_path, ["down"])
up_cmd = _build_compose_cmd(dir_path, ["up", "-d"])
print(">>> " + " ".join(down_cmd), flush=True)
subprocess.run(down_cmd, check=True)
print(">>> " + " ".join(up_cmd), flush=True)
subprocess.run(up_cmd, check=True)
def handle_docker_compose_services( def handle_docker_compose_services(

View File

@@ -52,7 +52,9 @@ def _atomic_write_cmd(cmd: str, out_file: str) -> None:
execute_shell_command(f"mv {tmp} {out_file}") execute_shell_command(f"mv {tmp} {out_file}")
def fallback_pg_dumpall(container: str, username: str, password: str, out_file: str) -> None: def fallback_pg_dumpall(
container: str, username: str, password: str, out_file: str
) -> None:
""" """
Perform a full Postgres cluster dump using pg_dumpall. Perform a full Postgres cluster dump using pg_dumpall.
""" """
@@ -103,9 +105,7 @@ def backup_database(
"'*' is currently only supported for Postgres." "'*' is currently only supported for Postgres."
) )
cluster_file = os.path.join( cluster_file = os.path.join(out_dir, f"{instance_name}.cluster.backup.sql")
out_dir, f"{instance_name}.cluster.backup.sql"
)
fallback_pg_dumpall(container, user, password, cluster_file) fallback_pg_dumpall(container, user, password, cluster_file)
produced = True produced = True
continue continue

View File

@@ -7,10 +7,11 @@ import re
import sys import sys
import pandas as pd import pandas as pd
from typing import Optional from typing import Optional
from pandas.errors import EmptyDataError
DB_NAME_RE = re.compile(r"^[a-zA-Z0-9_][a-zA-Z0-9_-]*$") DB_NAME_RE = re.compile(r"^[a-zA-Z0-9_][a-zA-Z0-9_-]*$")
def _validate_database_value(value: Optional[str], *, instance: str) -> str: def _validate_database_value(value: Optional[str], *, instance: str) -> str:
v = (value or "").strip() v = (value or "").strip()
if v == "": if v == "":
@@ -31,6 +32,11 @@ def _validate_database_value(value: Optional[str], *, instance: str) -> str:
) )
return v return v
def _empty_df() -> pd.DataFrame:
return pd.DataFrame(columns=["instance", "database", "username", "password"])
def check_and_add_entry( def check_and_add_entry(
file_path: str, file_path: str,
instance: str, instance: str,
@@ -48,17 +54,21 @@ def check_and_add_entry(
database = _validate_database_value(database, instance=instance) database = _validate_database_value(database, instance=instance)
if os.path.exists(file_path): if os.path.exists(file_path):
try:
df = pd.read_csv( df = pd.read_csv(
file_path, file_path,
sep=";", sep=";",
dtype=str, dtype=str,
keep_default_na=False, keep_default_na=False,
) )
else: except EmptyDataError:
df = pd.DataFrame( print(
columns=["instance", "database", "username", "password"] f"WARNING: databases.csv exists but is empty: {file_path}. Creating header columns.",
file=sys.stderr,
) )
df = _empty_df()
else:
df = _empty_df()
mask = (df["instance"] == instance) & (df["database"] == database) mask = (df["instance"] == instance) & (df["database"] == database)
if mask.any(): if mask.any():

View File

@@ -133,22 +133,28 @@ class TestE2EDumpOnlyFallbackToFiles(unittest.TestCase):
) )
def test_files_backup_exists_due_to_fallback(self) -> None: def test_files_backup_exists_due_to_fallback(self) -> None:
p = backup_path( p = (
backup_path(
self.backups_dir, self.backups_dir,
self.repo_name, self.repo_name,
self.version, self.version,
self.pg_volume, self.pg_volume,
) / "files" )
/ "files"
)
self.assertTrue(p.is_dir(), f"Expected files backup dir at: {p}") self.assertTrue(p.is_dir(), f"Expected files backup dir at: {p}")
def test_sql_dump_not_present(self) -> None: def test_sql_dump_not_present(self) -> None:
# There should be no sql dumps because databases.csv had no matching entry. # There should be no sql dumps because databases.csv had no matching entry.
sql_dir = backup_path( sql_dir = (
backup_path(
self.backups_dir, self.backups_dir,
self.repo_name, self.repo_name,
self.version, self.version,
self.pg_volume, self.pg_volume,
) / "sql" )
/ "sql"
)
# Could exist (dir created) in some edge cases, but should contain no *.sql dumps. # Could exist (dir created) in some edge cases, but should contain no *.sql dumps.
if sql_dir.exists(): if sql_dir.exists():
dumps = list(sql_dir.glob("*.sql")) dumps = list(sql_dir.glob("*.sql"))

View File

@@ -96,10 +96,10 @@ class TestE2EDumpOnlySqlMixedRun(unittest.TestCase):
"sh", "sh",
"-lc", "-lc",
( (
f'psql -U postgres -d {cls.pg_db} -c ' f"psql -U postgres -d {cls.pg_db} -c "
'"CREATE TABLE IF NOT EXISTS t (id INT PRIMARY KEY, v TEXT);' '"CREATE TABLE IF NOT EXISTS t (id INT PRIMARY KEY, v TEXT);'
"INSERT INTO t(id,v) VALUES (1,'hello-db') " "INSERT INTO t(id,v) VALUES (1,'hello-db') "
"ON CONFLICT (id) DO UPDATE SET v=EXCLUDED.v;\"" 'ON CONFLICT (id) DO UPDATE SET v=EXCLUDED.v;"'
), ),
], ],
check=True, check=True,
@@ -143,7 +143,9 @@ class TestE2EDumpOnlySqlMixedRun(unittest.TestCase):
cleanup_docker(containers=cls.containers, volumes=cls.volumes) cleanup_docker(containers=cls.containers, volumes=cls.volumes)
def test_db_volume_has_dump_and_no_files_dir(self) -> None: def test_db_volume_has_dump_and_no_files_dir(self) -> None:
base = backup_path(self.backups_dir, self.repo_name, self.version, self.db_volume) base = backup_path(
self.backups_dir, self.repo_name, self.version, self.db_volume
)
dumps = base / "sql" dumps = base / "sql"
files = base / "files" files = base / "files"

View File

@@ -99,10 +99,10 @@ class TestE2ESeedStarAndDbEntriesBackupPostgres(unittest.TestCase):
"sh", "sh",
"-lc", "-lc",
( (
f'psql -U {cls.pg_user} -d {cls.pg_db1} -c ' f"psql -U {cls.pg_user} -d {cls.pg_db1} -c "
'"CREATE TABLE IF NOT EXISTS t (id INT PRIMARY KEY, v TEXT);' '"CREATE TABLE IF NOT EXISTS t (id INT PRIMARY KEY, v TEXT);'
"INSERT INTO t(id,v) VALUES (1,'hello-db1') " "INSERT INTO t(id,v) VALUES (1,'hello-db1') "
"ON CONFLICT (id) DO UPDATE SET v=EXCLUDED.v;\"" 'ON CONFLICT (id) DO UPDATE SET v=EXCLUDED.v;"'
), ),
], ],
check=True, check=True,
@@ -115,10 +115,10 @@ class TestE2ESeedStarAndDbEntriesBackupPostgres(unittest.TestCase):
"sh", "sh",
"-lc", "-lc",
( (
f'psql -U {cls.pg_user} -d {cls.pg_db2} -c ' f"psql -U {cls.pg_user} -d {cls.pg_db2} -c "
'"CREATE TABLE IF NOT EXISTS t (id INT PRIMARY KEY, v TEXT);' '"CREATE TABLE IF NOT EXISTS t (id INT PRIMARY KEY, v TEXT);'
"INSERT INTO t(id,v) VALUES (1,'hello-db2') " "INSERT INTO t(id,v) VALUES (1,'hello-db2') "
"ON CONFLICT (id) DO UPDATE SET v=EXCLUDED.v;\"" 'ON CONFLICT (id) DO UPDATE SET v=EXCLUDED.v;"'
), ),
], ],
check=True, check=True,
@@ -132,7 +132,16 @@ class TestE2ESeedStarAndDbEntriesBackupPostgres(unittest.TestCase):
instance = cls.pg_container instance = cls.pg_container
# Seed star entry (pg_dumpall) # Seed star entry (pg_dumpall)
run(["baudolo-seed", cls.databases_csv, instance, "*", cls.pg_user, cls.pg_password]) run(
[
"baudolo-seed",
cls.databases_csv,
instance,
"*",
cls.pg_user,
cls.pg_password,
]
)
# Seed concrete DB entry (pg_dump) # Seed concrete DB entry (pg_dump)
run( run(
@@ -177,7 +186,9 @@ class TestE2ESeedStarAndDbEntriesBackupPostgres(unittest.TestCase):
cleanup_docker(containers=cls.containers, volumes=cls.volumes) cleanup_docker(containers=cls.containers, volumes=cls.volumes)
def test_db_volume_has_cluster_dump_and_concrete_db_dump_and_no_files(self) -> None: def test_db_volume_has_cluster_dump_and_concrete_db_dump_and_no_files(self) -> None:
base = backup_path(self.backups_dir, self.repo_name, self.version, self.db_volume) base = backup_path(
self.backups_dir, self.repo_name, self.version, self.db_volume
)
sql_dir = base / "sql" sql_dir = base / "sql"
files_dir = base / "files" files_dir = base / "files"
@@ -204,10 +215,14 @@ class TestE2ESeedStarAndDbEntriesBackupPostgres(unittest.TestCase):
) )
def test_non_db_volume_still_has_files_backup(self) -> None: def test_non_db_volume_still_has_files_backup(self) -> None:
base = backup_path(self.backups_dir, self.repo_name, self.version, self.files_volume) base = backup_path(
self.backups_dir, self.repo_name, self.version, self.files_volume
)
files_dir = base / "files" files_dir = base / "files"
self.assertTrue(files_dir.exists(), f"Expected files dir for non-DB volume at: {files_dir}") self.assertTrue(
files_dir.exists(), f"Expected files dir for non-DB volume at: {files_dir}"
)
marker = files_dir / "hello.txt" marker = files_dir / "hello.txt"
self.assertTrue(marker.is_file(), f"Expected marker file at: {marker}") self.assertTrue(marker.is_file(), f"Expected marker file at: {marker}")

View File

View File

@@ -0,0 +1,77 @@
import io
import os
import tempfile
import unittest
from contextlib import redirect_stderr
import pandas as pd
# Adjust if your package name/import path differs.
from baudolo.backup.app import _load_databases_df
EXPECTED_COLUMNS = ["instance", "database", "username", "password"]
class TestLoadDatabasesDf(unittest.TestCase):
def test_missing_csv_is_handled_with_warning_and_empty_df(self) -> None:
with tempfile.TemporaryDirectory() as td:
missing_path = os.path.join(td, "does-not-exist.csv")
buf = io.StringIO()
with redirect_stderr(buf):
df = _load_databases_df(missing_path)
stderr = buf.getvalue()
self.assertIn("WARNING:", stderr)
self.assertIn("databases.csv not found", stderr)
self.assertIsInstance(df, pd.DataFrame)
self.assertListEqual(list(df.columns), EXPECTED_COLUMNS)
self.assertTrue(df.empty)
def test_empty_csv_is_handled_with_warning_and_empty_df(self) -> None:
with tempfile.TemporaryDirectory() as td:
empty_path = os.path.join(td, "databases.csv")
# Create an empty file (0 bytes)
with open(empty_path, "w", encoding="utf-8") as f:
f.write("")
buf = io.StringIO()
with redirect_stderr(buf):
df = _load_databases_df(empty_path)
stderr = buf.getvalue()
self.assertIn("WARNING:", stderr)
self.assertIn("exists but is empty", stderr)
self.assertIsInstance(df, pd.DataFrame)
self.assertListEqual(list(df.columns), EXPECTED_COLUMNS)
self.assertTrue(df.empty)
def test_valid_csv_loads_without_warning(self) -> None:
with tempfile.TemporaryDirectory() as td:
csv_path = os.path.join(td, "databases.csv")
content = "instance;database;username;password\nmyapp;*;dbuser;secret\n"
with open(csv_path, "w", encoding="utf-8") as f:
f.write(content)
buf = io.StringIO()
with redirect_stderr(buf):
df = _load_databases_df(csv_path)
stderr = buf.getvalue()
self.assertEqual(stderr, "") # no warning expected
self.assertIsInstance(df, pd.DataFrame)
self.assertListEqual(list(df.columns), EXPECTED_COLUMNS)
self.assertEqual(len(df), 1)
self.assertEqual(df.loc[0, "instance"], "myapp")
self.assertEqual(df.loc[0, "database"], "*")
self.assertEqual(df.loc[0, "username"], "dbuser")
self.assertEqual(df.loc[0, "password"], "secret")
if __name__ == "__main__":
unittest.main()

View File

@@ -0,0 +1,239 @@
from __future__ import annotations
import shutil
import tempfile
import unittest
from pathlib import Path
from typing import List
from unittest.mock import patch
def _touch(p: Path) -> None:
p.parent.mkdir(parents=True, exist_ok=True)
# If the path already exists as a directory (e.g. ".env" created by ".env/env"),
# remove it so we can create a file with the same name.
if p.exists() and p.is_dir():
shutil.rmtree(p)
p.write_text("x", encoding="utf-8")
def _setup_compose_dir(
tmp_path: Path,
name: str = "mailu",
*,
with_override: bool = False,
with_ca_override: bool = False,
env_layout: str | None = None, # None | ".env" | ".env/env"
) -> Path:
d = tmp_path / name
d.mkdir(parents=True, exist_ok=True)
_touch(d / "docker-compose.yml")
if with_override:
_touch(d / "docker-compose.override.yml")
if with_ca_override:
_touch(d / "docker-compose.ca.override.yml")
if env_layout == ".env":
_touch(d / ".env")
elif env_layout == ".env/env":
_touch(d / ".env" / "env")
return d
class TestCompose(unittest.TestCase):
@classmethod
def setUpClass(cls) -> None:
from baudolo.backup import compose as mod
cls.compose_mod = mod
def test_detect_env_file_prefers_dotenv_over_legacy(self) -> None:
with tempfile.TemporaryDirectory() as td:
tmp_path = Path(td)
d = _setup_compose_dir(tmp_path, env_layout=".env/env")
# Also create .env file -> should be preferred
_touch(d / ".env")
env_file = self.compose_mod._detect_env_file(d)
self.assertEqual(env_file, d / ".env")
def test_detect_env_file_uses_legacy_if_no_dotenv(self) -> None:
with tempfile.TemporaryDirectory() as td:
tmp_path = Path(td)
d = _setup_compose_dir(tmp_path, env_layout=".env/env")
env_file = self.compose_mod._detect_env_file(d)
self.assertEqual(env_file, d / ".env" / "env")
def test_detect_compose_files_requires_base(self) -> None:
with tempfile.TemporaryDirectory() as td:
tmp_path = Path(td)
d = tmp_path / "stack"
d.mkdir()
with self.assertRaises(FileNotFoundError):
self.compose_mod._detect_compose_files(d)
def test_detect_compose_files_includes_optional_overrides(self) -> None:
with tempfile.TemporaryDirectory() as td:
tmp_path = Path(td)
d = _setup_compose_dir(
tmp_path,
with_override=True,
with_ca_override=True,
)
files = self.compose_mod._detect_compose_files(d)
self.assertEqual(
files,
[
d / "docker-compose.yml",
d / "docker-compose.override.yml",
d / "docker-compose.ca.override.yml",
],
)
def test_build_cmd_uses_wrapper_when_present(self) -> None:
with tempfile.TemporaryDirectory() as td:
tmp_path = Path(td)
d = _setup_compose_dir(
tmp_path, with_override=True, with_ca_override=True, env_layout=".env"
)
with patch.object(
self.compose_mod.shutil, "which", lambda name: "/usr/local/bin/compose"
):
cmd = self.compose_mod._build_compose_cmd(str(d), ["up", "-d"])
self.assertEqual(
cmd,
[
"/usr/local/bin/compose",
"--chdir",
str(d.resolve()),
"--",
"up",
"-d",
],
)
def test_build_cmd_fallback_docker_compose_with_all_files_and_env(self) -> None:
with tempfile.TemporaryDirectory() as td:
tmp_path = Path(td)
d = _setup_compose_dir(
tmp_path,
with_override=True,
with_ca_override=True,
env_layout=".env",
)
with patch.object(self.compose_mod.shutil, "which", lambda name: None):
cmd = self.compose_mod._build_compose_cmd(
str(d), ["up", "-d", "--force-recreate"]
)
expected: List[str] = [
"docker",
"compose",
"-f",
str((d / "docker-compose.yml").resolve()),
"-f",
str((d / "docker-compose.override.yml").resolve()),
"-f",
str((d / "docker-compose.ca.override.yml").resolve()),
"--env-file",
str((d / ".env").resolve()),
"up",
"-d",
"--force-recreate",
]
self.assertEqual(cmd, expected)
def test_hard_restart_calls_run_twice_with_correct_cmds_wrapper(self) -> None:
with tempfile.TemporaryDirectory() as td:
tmp_path = Path(td)
d = _setup_compose_dir(tmp_path, name="mailu", env_layout=".env")
with patch.object(
self.compose_mod.shutil, "which", lambda name: "/usr/local/bin/compose"
):
calls = []
def fake_run(cmd, check: bool):
calls.append((cmd, check))
return 0
with patch.object(self.compose_mod.subprocess, "run", fake_run):
self.compose_mod.hard_restart_docker_services(str(d))
self.assertEqual(
calls,
[
(
[
"/usr/local/bin/compose",
"--chdir",
str(d.resolve()),
"--",
"down",
],
True,
),
(
[
"/usr/local/bin/compose",
"--chdir",
str(d.resolve()),
"--",
"up",
"-d",
],
True,
),
],
)
def test_hard_restart_calls_run_twice_with_correct_cmds_fallback(self) -> None:
with tempfile.TemporaryDirectory() as td:
tmp_path = Path(td)
d = _setup_compose_dir(
tmp_path,
name="mailu",
with_override=True,
with_ca_override=True,
env_layout=".env/env",
)
with patch.object(self.compose_mod.shutil, "which", lambda name: None):
calls = []
def fake_run(cmd, check: bool):
calls.append((cmd, check))
return 0
with patch.object(self.compose_mod.subprocess, "run", fake_run):
self.compose_mod.hard_restart_docker_services(str(d))
down_cmd = calls[0][0]
up_cmd = calls[1][0]
self.assertTrue(calls[0][1] is True)
self.assertTrue(calls[1][1] is True)
self.assertEqual(down_cmd[0:2], ["docker", "compose"])
self.assertEqual(down_cmd[-1], "down")
self.assertIn("--env-file", down_cmd)
self.assertEqual(up_cmd[0:2], ["docker", "compose"])
self.assertTrue(up_cmd[-2:] == ["up", "-d"] or up_cmd[-3:] == ["up", "-d"])
self.assertIn("--env-file", up_cmd)
if __name__ == "__main__":
unittest.main(verbosity=2)

View File

View File

@@ -0,0 +1,215 @@
from __future__ import annotations
import unittest
from unittest.mock import MagicMock, patch
import pandas as pd
from pandas.errors import EmptyDataError
import baudolo.seed.__main__ as seed_main
class TestSeedMain(unittest.TestCase):
@patch("baudolo.seed.__main__.pd.DataFrame")
def test_empty_df_creates_expected_columns(self, df_ctor: MagicMock) -> None:
seed_main._empty_df()
df_ctor.assert_called_once_with(
columns=["instance", "database", "username", "password"]
)
def test_validate_database_value_rejects_empty(self) -> None:
with self.assertRaises(ValueError):
seed_main._validate_database_value("", instance="x")
def test_validate_database_value_accepts_star(self) -> None:
self.assertEqual(seed_main._validate_database_value("*", instance="x"), "*")
def test_validate_database_value_rejects_nan(self) -> None:
with self.assertRaises(ValueError):
seed_main._validate_database_value("nan", instance="x")
def test_validate_database_value_rejects_invalid_name(self) -> None:
with self.assertRaises(ValueError):
seed_main._validate_database_value("bad name", instance="x")
def _mock_df_mask_any(self, *, any_value: bool) -> MagicMock:
"""
Build a DataFrame-like mock such that:
mask = (df["instance"] == instance) & (df["database"] == database)
mask.any() returns any_value
"""
df = MagicMock(spec=pd.DataFrame)
left = MagicMock()
right = MagicMock()
mask = MagicMock()
mask.any.return_value = any_value
# (left & right) => mask
left.__and__.return_value = mask
# df["instance"] / df["database"] => return objects whose == produces left/right
col = MagicMock()
col.__eq__.side_effect = [left, right]
df.__getitem__.return_value = col
return df
@patch("baudolo.seed.__main__.os.path.exists", return_value=False)
@patch("baudolo.seed.__main__.pd.read_csv")
@patch("baudolo.seed.__main__._empty_df")
@patch("baudolo.seed.__main__.pd.concat")
def test_check_and_add_entry_file_missing_adds_entry(
self,
concat: MagicMock,
empty_df: MagicMock,
read_csv: MagicMock,
exists: MagicMock,
) -> None:
df_existing = self._mock_df_mask_any(any_value=False)
empty_df.return_value = df_existing
df_out = MagicMock(spec=pd.DataFrame)
concat.return_value = df_out
seed_main.check_and_add_entry(
file_path="/tmp/databases.csv",
instance="inst",
database="db",
username="user",
password="pass",
)
read_csv.assert_not_called()
empty_df.assert_called_once()
concat.assert_called_once()
df_out.to_csv.assert_called_once_with(
"/tmp/databases.csv", sep=";", index=False
)
@patch("baudolo.seed.__main__.os.path.exists", return_value=True)
@patch("baudolo.seed.__main__.pd.read_csv", side_effect=EmptyDataError("empty"))
@patch("baudolo.seed.__main__._empty_df")
@patch("baudolo.seed.__main__.pd.concat")
@patch("baudolo.seed.__main__.print")
def test_check_and_add_entry_empty_file_warns_and_creates_columns_and_adds(
self,
print_: MagicMock,
concat: MagicMock,
empty_df: MagicMock,
read_csv: MagicMock,
exists: MagicMock,
) -> None:
df_existing = self._mock_df_mask_any(any_value=False)
empty_df.return_value = df_existing
df_out = MagicMock(spec=pd.DataFrame)
concat.return_value = df_out
seed_main.check_and_add_entry(
file_path="/tmp/databases.csv",
instance="inst",
database="db",
username="user",
password="pass",
)
exists.assert_called_once_with("/tmp/databases.csv")
read_csv.assert_called_once()
empty_df.assert_called_once()
concat.assert_called_once()
# Assert: at least one print call contains the WARNING and prints to stderr
warning_calls = []
for call in print_.call_args_list:
args, kwargs = call
if args and "WARNING: databases.csv exists but is empty" in str(args[0]):
warning_calls.append((args, kwargs))
self.assertTrue(
warning_calls,
"Expected a WARNING print when databases.csv is empty, but none was found.",
)
# Ensure the warning goes to stderr
_, warn_kwargs = warning_calls[0]
self.assertEqual(warn_kwargs.get("file"), seed_main.sys.stderr)
df_out.to_csv.assert_called_once_with(
"/tmp/databases.csv", sep=";", index=False
)
@patch("baudolo.seed.__main__.os.path.exists", return_value=True)
@patch("baudolo.seed.__main__.pd.read_csv")
def test_check_and_add_entry_updates_existing_row(
self,
read_csv: MagicMock,
exists: MagicMock,
) -> None:
df = self._mock_df_mask_any(any_value=True)
read_csv.return_value = df
seed_main.check_and_add_entry(
file_path="/tmp/databases.csv",
instance="inst",
database="db",
username="user",
password="pass",
)
df.to_csv.assert_called_once_with("/tmp/databases.csv", sep=";", index=False)
@patch("baudolo.seed.__main__.check_and_add_entry")
@patch("baudolo.seed.__main__.argparse.ArgumentParser.parse_args")
def test_main_calls_check_and_add_entry(
self, parse_args: MagicMock, cae: MagicMock
) -> None:
ns = MagicMock()
ns.file = "/tmp/databases.csv"
ns.instance = "inst"
ns.database = "db"
ns.username = "user"
ns.password = "pass"
parse_args.return_value = ns
seed_main.main()
cae.assert_called_once_with(
file_path="/tmp/databases.csv",
instance="inst",
database="db",
username="user",
password="pass",
)
@patch("baudolo.seed.__main__.sys.exit")
@patch("baudolo.seed.__main__.print")
@patch(
"baudolo.seed.__main__.check_and_add_entry", side_effect=RuntimeError("boom")
)
@patch("baudolo.seed.__main__.argparse.ArgumentParser.parse_args")
def test_main_exits_nonzero_on_error(
self,
parse_args: MagicMock,
cae: MagicMock,
print_: MagicMock,
exit_: MagicMock,
) -> None:
ns = MagicMock()
ns.file = "/tmp/databases.csv"
ns.instance = "inst"
ns.database = "db"
ns.username = "user"
ns.password = "pass"
parse_args.return_value = ns
seed_main.main()
self.assertTrue(print_.called)
_, kwargs = print_.call_args
self.assertEqual(kwargs.get("file"), seed_main.sys.stderr)
exit_.assert_called_once_with(1)
if __name__ == "__main__":
unittest.main(verbosity=2)