Release version 7.0.1

The --empty pre-clean picked its candidates by owner, on the stated
assumption that extension members are superuser-owned and would never be
selected. That holds only when a superuser installed the extension. A role
that installs one itself owns its functions, so they were listed for a
one-by-one DROP that postgres refuses:

    cannot drop function vector_in(cstring,oid,integer)
    because extension vector requires it

Under ON_ERROR_STOP that ends the whole restore, which is how a discourse
generation - it declares the vector extension - became unreplayable.

Membership now comes from pg_depend rather than from ownership. Each branch
carries its oid and classid so one NOT EXISTS covers all seven instead of
seven separate predicates, and the schema branch gets the same guard because
an extension can own a schema too. Skipping the members is enough: the dump's
CREATE EXTENSION IF NOT EXISTS finds the surviving extension either way.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

CHANGELOG.md

@@ -1,5 +1,102 @@
 # Changelog
 
+## [7.0.1] - 2026-08-18
+
+- Restore: *--empty* no longer aborts on a database that carries an extension.
+  The pre-clean picked its candidates by owner, on the stated assumption that
+  extension members are superuser-owned and would therefore never be selected.
+  That holds only when a superuser installed the extension: a role that
+  installs one itself owns its functions, so they were listed for a one-by-one
+  *DROP* that postgres refuses — *cannot drop function
+  vector_in(cstring,oid,integer) because extension vector requires it*. Under
+  *ON_ERROR_STOP* that ends the whole restore, which is how a generation of an
+  application declaring the *vector* extension became unreplayable. Membership
+  now comes from *pg_depend* rather than from ownership; each branch carries
+  its oid and classid so one *NOT EXISTS* covers all seven instead of seven
+  separate predicates, and the schema branch is guarded too because an
+  extension can own a schema. Skipping the members suffices — the dump's
+  *CREATE EXTENSION IF NOT EXISTS* finds the surviving extension either way.
+
+## [7.0.0] - 2026-08-18
+
+Breaking:
+- Backup: *mariadb* and *mysql* join the suffix tokens, so a container named
+  *<app>-mariadb* or *<app>-mysql* now resolves to the instance *<app>* instead
+  of to its own name. A *databases.csv* keyed on the full container name has to
+  move to the application name, or name the container in
+  *--database-containers*. This narrows what 6.0.0 broke rather than widening
+  it: a container named exactly *postgres*, *mariadb*, *mysql*, *db* or
+  *database* resolves again without any declaration, which is the shape a
+  compose file writes as *container_name: postgres* and the most common
+  configuration there is.
+
+Fixed:
+- Backup: a container named exactly after its engine is dumped again. The
+  suffix match needs a hyphen or underscore in front of the token, which a bare
+  name does not carry, so 6.0.0 resolved *container_name: postgres* to nothing
+  and stopped dumping it without saying so. *ENGINE_NAMES* now states the set
+  once and serves both readings — carried as a suffix it makes the rest the
+  instance, being one outright makes the container its own instance.
+- Backup: a central MariaDB under swarm is dumped for the first time. Swarm
+  names its task *mariadb_mariadb.1.<id>*, which matches neither the static
+  *mariadb* passed through *--database-containers* nor any token the suffix
+  match knew, since *_mariadb* is not *_db*. The database was silently absent
+  from every swarm backup this tool has ever written, before 6.0.0 as well.
+- Backup: an application container is no longer recorded as a database.
+  *container_engine* recognises an engine by its client tools, which an
+  application image frequently ships, so refusing its dump alone would have
+  written the volume to the manifest as *database: true, dumped: false* — the
+  exact shape a restore drill reads as a database that was missed. Without an
+  instance there is no database to record, and the volume is a file backup like
+  any other.
+
+## [6.0.0] - 2026-08-18
+
+Breaking:
+- Backup: a database container whose name carries no *database*, *db* or
+  *postgres* token — preceded by a hyphen or underscore — must now be named in
+  *--database-containers*. Without that declaration its *databases.csv* rows no
+  longer match and no dump is written, silently, because nothing fails. The
+  shape this hits hardest is a container named exactly *postgres* or *mariadb*:
+  the token needs a separator in front of it, which a bare name does not have.
+  *app-database*, *app_database.1.<task>* from swarm and *app-postgres-1* are
+  unaffected, as is any container already declared.
+
+Fixed:
+- Backup: *docker exec* now forwards *PGPASSWORD* into the container.
+  *execute_to_file* set the variable on baudolo's own process, but nothing
+  carried it across the container boundary, so an engine whose *pg_hba* demands
+  a password on TCP loopback refused every dump — which is every dedicated
+  Postgres instance on a real host. The name travels as a bare *-e NAME* so
+  docker copies the value out of this process's environment; spelling
+  *-e NAME=value* instead would publish the secret in the host's process list.
+- Backup: *get_instance* no longer claims an instance it never derived. It
+  returned the container name unchanged when that name carried no database
+  token, so an application container answered the same *databases.csv* row as
+  its own dedicated engine. Application images frequently ship the engine's
+  client tools, so the dump command started and wrote a file that looked like a
+  backup and held none of the data: measured against Discourse, 1,680 bytes
+  from the application where the engine produced 10,469,439. The regex stays a
+  normaliser — *<app>-database* from compose and *<app>_database.1.<task>* from
+  swarm still resolve to one instance. Only the fallthrough changed.
+
+New:
+- Tests: *get_instance* has unit coverage for the first time. Eleven cases pin
+  the container names that compose, swarm and explicitly-named engines produce,
+  so a future change to the regex has to state which shape it gives up.
+- Tests: two e2e modules cover shapes the suite structurally could not see.
+  Every fixture passed its container in *--database-containers*, which left the
+  regex branch — the only one a dedicated database ever takes — dead code under
+  test, and no scenario made a password mandatory, because stock
+  *postgres:alpine* grants trust on loopback.
+  *test_e2e_postgres_password_required* starts an engine with
+  *--auth-host=scram-sha-256* and carries a negative control asserting the
+  server refuses an unauthenticated dump; without it the module would pass
+  whether or not the password is forwarded at all.
+  *test_e2e_app_container_ships_client_tools* places an application container
+  beside its engine with neither declared, and requires the engine dumped, the
+  application volume copied as files, and no dump written from the application.
+
 ## [5.0.0] - 2026-08-18
 
 **[5.0.0] - 2026-08-18**

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "backup-docker-to-local"
-version = "5.0.0"
+version = "7.0.1"
 description = "Backup Docker volumes to local with rsync and optional DB dumps."
 readme = "README.md"
 requires-python = ">=3.9"

src/baudolo/backup/db.py

@@ -16,14 +16,35 @@ if TYPE_CHECKING:
 
 log = logging.getLogger(__name__)
 
+ENGINE_NAMES = ("database", "postgres", "mariadb", "mysql", "db")
+_SUFFIX_RE = re.compile(rf"(_|-)({'|'.join(ENGINE_NAMES)})")
 
-def get_instance(container: str, database_containers: list[str]) -> str:
-    """
-    Derive a stable instance name from the container name.
+
+def get_instance(container: str, database_containers: list[str]) -> str | None:
+    """The databases.csv instance a container serves, or None for no database.
+
+    A declared container is its own instance. Every other name is read against
+    ENGINE_NAMES: carrying one as a suffix makes the rest the instance, which
+    maps `<app>-database` from compose and `<app>_database.1.<task>` from swarm
+    onto the same one; being one outright makes the container its own instance,
+    the shape a compose file writes as `container_name: postgres`.
+
+    Args:
+        container: the running container's name.
+        database_containers: names passed via --database-containers, taken as
+            declared engines whatever they are called.
+
+    Returns:
+        The instance name, or None when the name neither carries nor is an
+        engine name: an application container is not an engine, even when it
+        ships the client tools that would let a dump command start.
     """
     if container in database_containers:
         return container
-    return re.split(r"(_|-)(database|db|postgres)", container)[0]
+    parts = _SUFFIX_RE.split(container)
+    if len(parts) > 1:
+        return parts[0]
+    return container if container in ENGINE_NAMES else None
 
 
 def fallback_pg_dumpall(
@@ -37,6 +58,7 @@ def fallback_pg_dumpall(
             container,
             ["pg_dumpall", "-U", username, "-h", "localhost"],
             interactive=True,
+            forward_env=["PGPASSWORD"],
         ),
         out_file,
         env={"PGPASSWORD": password},
@@ -62,6 +84,9 @@ def backup_database(
     Returns True if at least one dump was produced.
     """
     instance_name = get_instance(container, database_containers)
+    if instance_name is None:
+        log.debug("Container '%s' carries no database token", container)
+        return False
 
     entries = databases_df[databases_df["instance"] == instance_name]
     if entries.empty:
@@ -133,6 +158,7 @@ def backup_database(
                             "--no-privileges",
                         ],
                         interactive=True,
+                        forward_env=["PGPASSWORD"],
                     ),
                     dump_file,
                     env={"PGPASSWORD": password},

src/baudolo/backup/docker.py

@@ -9,10 +9,35 @@ if TYPE_CHECKING:
 
 
 def docker_exec_argv(
-    container: str, argv: Sequence[str], *, interactive: bool = False
+    container: str,
+    argv: Sequence[str],
+    *,
+    interactive: bool = False,
+    forward_env: Sequence[str] = (),
 ) -> list[str]:
-    """The argv that runs *argv* inside *container*."""
-    return ["docker", "exec", *(["-i"] if interactive else []), container, *argv]
+    """The argv that runs *argv* inside *container*.
+
+    Args:
+        container: the container to run in.
+        argv: the command, already split.
+        interactive: keep stdin open, for a command that is fed a dump.
+        forward_env: names of environment variables to hand to the container.
+            Passed as bare ``-e NAME``, so docker copies the value out of this
+            process's own environment; spelling ``-e NAME=value`` instead would
+            publish a secret in the host's process list.
+
+    Returns:
+        The argv list.
+    """
+    forwarded = [arg for name in forward_env for arg in ("-e", name)]
+    return [
+        "docker",
+        "exec",
+        *(["-i"] if interactive else []),
+        *forwarded,
+        container,
+        *argv,
+    ]
 
 
 def get_image_info(container: str) -> str:

src/baudolo/backup/dumps.py

@@ -10,7 +10,7 @@ from pandas.errors import EmptyDataError
 
 from baudolo.databases import COLUMNS, DELIMITER
 
-from .db import backup_database
+from .db import backup_database, get_instance
 from .docker import has_tool, image_id
 
 DUMP_TOOLS: tuple[tuple[str, str], ...] = (
@@ -76,6 +76,8 @@ def backup_mariadb_or_postgres(
     engine = container_engine(container)
     if engine is None:
         return VolumeOutcome(database=False, dumped=False)
+    if get_instance(container, database_containers) is None:
+        return VolumeOutcome(database=False, dumped=False)
     db_type, dump_tool = engine
     dumped = backup_database(
         container=container,

src/baudolo/restore/db/empty_preclean.sql

@@ -3,8 +3,12 @@
 -- would run every DROP in one transaction and exhaust max_locks_per_transaction on
 -- large schemas (e.g. gitlab). Also drops user-owned non-public schemas so a dump
 -- that CREATE SCHEMAs (e.g. discourse's discourse_functions) does not fail on an
--- already-existing schema. Extension members (pg_trgm's set_limit) are
--- superuser-owned; IF EXISTS absorbs the CASCADE fallout.
+-- already-existing schema. Objects belonging to an extension are skipped: postgres
+-- refuses to drop them one by one ("cannot drop function vector_in(...) because
+-- extension vector requires it"), and the dump's CREATE EXTENSION IF NOT EXISTS
+-- finds the surviving extension either way. Owning them is not enough to make them
+-- droppable - an extension a role installed itself is owned by that role, so the
+-- owner filter alone lets pgvector's members through.
 SELECT format('DROP %s IF EXISTS public.%s CASCADE', obj.type, obj.name)
   FROM (
     SELECT format('%I', c.relname) AS name,
@@ -13,7 +17,8 @@ SELECT format('DROP %s IF EXISTS public.%s CASCADE', obj.type, obj.name)
              WHEN 'm' THEN 'MATERIALIZED VIEW'
              WHEN 'f' THEN 'FOREIGN TABLE'
              ELSE 'TABLE'
-           END AS type
+           END AS type,
+           c.oid AS objid, 'pg_class'::regclass AS classid
       FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
      WHERE n.nspname = 'public' AND c.relkind IN ('r', 'p', 'v', 'm', 'f')
        AND pg_get_userbyid(c.relowner) = current_user
@@ -21,17 +26,20 @@ SELECT format('DROP %s IF EXISTS public.%s CASCADE', obj.type, obj.name)
     -- Overloaded functions share a proname; DROP needs the identity
     -- signature or psql aborts with "function name is not unique".
     SELECT format('%I(%s)', p.proname, pg_get_function_identity_arguments(p.oid)) AS name,
-           CASE p.prokind WHEN 'p' THEN 'PROCEDURE' ELSE 'FUNCTION' END AS type
+           CASE p.prokind WHEN 'p' THEN 'PROCEDURE' ELSE 'FUNCTION' END AS type,
+           p.oid AS objid, 'pg_proc'::regclass AS classid
       FROM pg_proc p JOIN pg_namespace n ON n.oid = p.pronamespace
      WHERE n.nspname = 'public' AND p.prokind IN ('f', 'p', 'w')
        AND pg_get_userbyid(p.proowner) = current_user
     UNION ALL
-    SELECT format('%I', c.relname) AS name, 'SEQUENCE' AS type
+    SELECT format('%I', c.relname) AS name, 'SEQUENCE' AS type,
+           c.oid AS objid, 'pg_class'::regclass AS classid
       FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
      WHERE n.nspname = 'public' AND c.relkind = 'S'
        AND pg_get_userbyid(c.relowner) = current_user
     UNION ALL
-    SELECT format('%I', t.typname) AS name, 'TYPE' AS type
+    SELECT format('%I', t.typname) AS name, 'TYPE' AS type,
+           t.oid AS objid, 'pg_type'::regclass AS classid
       FROM pg_type t JOIN pg_namespace n ON n.oid = t.typnamespace
      WHERE n.nspname = 'public'
        AND pg_get_userbyid(t.typowner) = current_user
@@ -40,25 +48,36 @@ SELECT format('DROP %s IF EXISTS public.%s CASCADE', obj.type, obj.name)
                   SELECT 1 FROM pg_class c2
                    WHERE c2.oid = t.typrelid AND c2.relkind = 'c')))
     UNION ALL
-    SELECT format('%I', col.collname) AS name, 'COLLATION' AS type
+    SELECT format('%I', col.collname) AS name, 'COLLATION' AS type,
+           col.oid AS objid, 'pg_collation'::regclass AS classid
       FROM pg_collation col JOIN pg_namespace n ON n.oid = col.collnamespace
      WHERE n.nspname = 'public'
        AND pg_get_userbyid(col.collowner) = current_user
     UNION ALL
-    SELECT format('%I', ts.cfgname) AS name, 'TEXT SEARCH CONFIGURATION' AS type
+    SELECT format('%I', ts.cfgname) AS name, 'TEXT SEARCH CONFIGURATION' AS type,
+           ts.oid AS objid, 'pg_ts_config'::regclass AS classid
       FROM pg_ts_config ts JOIN pg_namespace n ON n.oid = ts.cfgnamespace
      WHERE n.nspname = 'public'
        AND pg_get_userbyid(ts.cfgowner) = current_user
     UNION ALL
-    SELECT format('%I', d.dictname) AS name, 'TEXT SEARCH DICTIONARY' AS type
+    SELECT format('%I', d.dictname) AS name, 'TEXT SEARCH DICTIONARY' AS type,
+           d.oid AS objid, 'pg_ts_dict'::regclass AS classid
       FROM pg_ts_dict d JOIN pg_namespace n ON n.oid = d.dictnamespace
      WHERE n.nspname = 'public'
        AND pg_get_userbyid(d.dictowner) = current_user
   ) obj
+ WHERE NOT EXISTS (
+         SELECT 1 FROM pg_depend dep
+          WHERE dep.classid = obj.classid AND dep.objid = obj.objid
+            AND dep.deptype = 'e')
 UNION ALL
 SELECT format('DROP SCHEMA IF EXISTS %I CASCADE', n.nspname)
   FROM pg_namespace n
  WHERE NOT starts_with(n.nspname, 'pg_')
    AND n.nspname NOT IN ('public', 'information_schema')
    AND pg_get_userbyid(n.nspowner) = current_user
+   AND NOT EXISTS (
+         SELECT 1 FROM pg_depend dep
+          WHERE dep.classid = 'pg_namespace'::regclass AND dep.objid = n.oid
+            AND dep.deptype = 'e')
 \gexec

tests/e2e/test_e2e_app_container_ships_client_tools.py

@@ -0,0 +1,184 @@
+"""An application container that ships the engine's client tools.
+
+This is the shape a dedicated database deploys in: the engine runs as
+`<app>-database` while the application itself runs as `<app>`, and neither is
+declared through --database-containers, so both names go through the instance
+regex. `<app>-database` loses its suffix and lands on the instance `<app>` -
+and `<app>` carries no database token at all, so a fallback that returns the
+name unchanged lands on that same instance and offers the application container
+as a second engine for the same row.
+
+Discourse is the live example: its application container is named `discourse`
+by its own launcher and ships pg_dumpall, so a dump command starts there and
+writes a file that looks like a backup and holds none of the data.
+"""
+
+import json
+import unittest
+from pathlib import Path
+
+from baudolo.generation import DUMP_SUFFIX, FILES_DIR, MANIFEST_FILE, SQL_DIR
+
+from .helpers import (
+    POSTGRES_DATA_DIR,
+    POSTGRES_IMAGE,
+    backup_path,
+    backup_run,
+    cleanup_docker,
+    create_minimal_compose_dir,
+    ensure_empty_dir,
+    latest_version_dir,
+    require_docker,
+    run,
+    unique,
+    wait_for_postgres,
+    write_databases_csv,
+)
+
+MARKER = "the-application-volume-holds-files"
+PAYLOAD = "shop-payload"
+
+
+class TestE2EAppContainerShipsClientTools(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls) -> None:
+        require_docker()
+        # uuid4 hex may begin with "db", which the instance regex would split
+        # on and turn the application container into a different instance,
+        # hiding exactly the collision this module is about.
+        cls.prefix = unique("baudolo-e2e-app-tools").replace("-db", "-xb")
+        cls.backups_dir = f"/tmp/{cls.prefix}/Backups"
+        ensure_empty_dir(cls.backups_dir)
+        cls.compose_dir = create_minimal_compose_dir(f"/tmp/{cls.prefix}")
+        cls.repo_name = cls.prefix
+
+        cls.engine = f"{cls.prefix}-shop-database"
+        cls.app = f"{cls.prefix}-shop"
+        cls.engine_volume = f"{cls.prefix}-shop-database-vol"
+        cls.app_volume = f"{cls.prefix}-shop-app-vol"
+        cls.containers = [cls.engine, cls.app]
+        cls.volumes = [cls.engine_volume, cls.app_volume]
+
+        run(["docker", "volume", "create", cls.engine_volume])
+        run(["docker", "volume", "create", cls.app_volume])
+
+        run(
+            [
+                "docker",
+                "run",
+                "-d",
+                "--name",
+                cls.engine,
+                "-e",
+                "POSTGRES_PASSWORD=shoppw",
+                "-e",
+                "POSTGRES_DB=shopdb",
+                "-e",
+                "POSTGRES_USER=postgres",
+                "-v",
+                f"{cls.engine_volume}:{POSTGRES_DATA_DIR}",
+                POSTGRES_IMAGE,
+            ]
+        )
+
+        run(
+            [
+                "docker",
+                "run",
+                "-d",
+                "--name",
+                cls.app,
+                "--entrypoint",
+                "sh",
+                "-v",
+                f"{cls.app_volume}:/data",
+                POSTGRES_IMAGE,
+                "-c",
+                f"echo '{MARKER}' > /data/marker.txt && sleep 3600",
+            ]
+        )
+
+        wait_for_postgres(cls.engine, user="postgres", timeout_s=90)
+        run(
+            [
+                "docker",
+                "exec",
+                cls.engine,
+                "sh",
+                "-lc",
+                (
+                    'psql -U postgres -d shopdb -c "CREATE TABLE orders (id int, '
+                    f"note text); INSERT INTO orders VALUES (1,'{PAYLOAD}');\""
+                ),
+            ],
+            check=True,
+        )
+
+        cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv"
+        write_databases_csv(
+            cls.databases_csv,
+            [(cls.app, "shopdb", "postgres", "shoppw")],
+        )
+
+        backup_run(
+            backups_dir=cls.backups_dir,
+            repo_name=cls.repo_name,
+            compose_dir=cls.compose_dir,
+            databases_csv=cls.databases_csv,
+            database_containers=["dummy-db"],
+            images_no_stop_required=[POSTGRES_IMAGE],
+        )
+        cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)
+
+    @classmethod
+    def tearDownClass(cls) -> None:
+        cleanup_docker(containers=cls.containers, volumes=cls.volumes)
+
+    def volume_dir(self, volume: str) -> Path:
+        return backup_path(self.backups_dir, self.repo_name, self.version, volume)
+
+    def test_the_engine_volume_was_dumped(self) -> None:
+        dump = self.volume_dir(self.engine_volume) / SQL_DIR / f"shopdb{DUMP_SUFFIX}"
+        self.assertTrue(dump.is_file(), f"expected a dump at {dump}")
+        self.assertIn(PAYLOAD, dump.read_text(encoding="utf-8"))
+
+    def test_the_application_volume_produced_no_dump(self) -> None:
+        """The collision this module exists for: the application container
+        answers the same instance as the engine and starts a dump of its own."""
+        sql_dir = self.volume_dir(self.app_volume) / SQL_DIR
+        self.assertFalse(
+            sql_dir.exists(),
+            f"the application container was dumped: {sorted(sql_dir.iterdir())}"
+            if sql_dir.exists()
+            else "",
+        )
+
+    def test_the_application_volume_was_backed_up_as_files(self) -> None:
+        """Refusing the dump must not cost the volume its backup."""
+        marker = self.volume_dir(self.app_volume) / FILES_DIR / "marker.txt"
+        self.assertTrue(marker.is_file(), f"expected a file backup at {marker}")
+        self.assertIn(MARKER, marker.read_text(encoding="utf-8"))
+
+    def test_the_manifest_does_not_call_the_application_volume_a_database(self) -> None:
+        manifest = json.loads(
+            (self.volume_dir(self.app_volume).parent / MANIFEST_FILE).read_text(
+                encoding="utf-8"
+            )
+        )
+        entry = manifest["volumes"][self.app_volume]
+        self.assertFalse(entry["database"], entry)
+        self.assertFalse(entry["dumped"], entry)
+
+    def test_the_manifest_records_the_engine_volume_as_dumped(self) -> None:
+        manifest = json.loads(
+            (self.volume_dir(self.engine_volume).parent / MANIFEST_FILE).read_text(
+                encoding="utf-8"
+            )
+        )
+        entry = manifest["volumes"][self.engine_volume]
+        self.assertTrue(entry["database"], entry)
+        self.assertTrue(entry["dumped"], entry)
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/e2e/test_e2e_postgres_empty_drop_hard.py

@@ -24,6 +24,7 @@ from .helpers import (
 # is not unique") and english_stem_nostop reproduces taiga's text search
 # dictionary abort (duplicate pg_ts_dict_dictname_index).
 SCENARIO_SQL = (
+    "CREATE EXTENSION pg_trgm;"
     "CREATE SCHEMA discourse_functions;"
     "CREATE TABLE discourse_functions.helper (id int);"
     "INSERT INTO discourse_functions.helper VALUES (1);"
@@ -166,6 +167,13 @@ class TestE2EPostgresEmptyDropHard(unittest.TestCase):
         )
         self.assertEqual(self._scalar("SELECT public.f(41) + public.f();"), "42")
 
+    def test_the_extension_survived_the_preclean(self) -> None:
+        self.assertEqual(
+            self._scalar("SELECT count(*) FROM pg_extension WHERE extname='pg_trgm';"),
+            "1",
+        )
+        self.assertEqual(self._scalar("SELECT similarity('abc','abc')::int;"), "1")
+
     def test_text_search_dictionary_restored_once(self) -> None:
         self.assertEqual(
             self._scalar(

tests/e2e/test_e2e_postgres_password_required.py

@@ -0,0 +1,159 @@
+"""An engine whose loopback auth really demands a password.
+
+Every other Postgres scenario runs stock postgres:alpine, whose generated
+pg_hba grants trust on 127.0.0.1 and ::1 - so `pg_dump -h localhost` never
+needs the password and a dump succeeds whether or not baudolo hands one to the
+container. This module makes the password mandatory, which is what a dedicated
+engine on a real host does.
+"""
+
+import unittest
+from pathlib import Path
+
+from baudolo.generation import CLUSTER_SUFFIX, DUMP_SUFFIX, FILES_DIR, SQL_DIR
+
+from .helpers import (
+    POSTGRES_DATA_DIR,
+    POSTGRES_IMAGE,
+    backup_path,
+    backup_run,
+    cleanup_docker,
+    create_minimal_compose_dir,
+    ensure_empty_dir,
+    latest_version_dir,
+    require_docker,
+    run,
+    unique,
+    wait_for_postgres,
+    write_databases_csv,
+)
+
+
+class TestE2EPostgresPasswordRequired(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls) -> None:
+        require_docker()
+        cls.prefix = unique("baudolo-e2e-pg-password-required")
+        cls.backups_dir = f"/tmp/{cls.prefix}/Backups"
+        ensure_empty_dir(cls.backups_dir)
+        cls.compose_dir = create_minimal_compose_dir(f"/tmp/{cls.prefix}")
+        cls.repo_name = cls.prefix
+
+        cls.pg_container = f"{cls.prefix}-pg"
+        cls.pg_volume = f"{cls.prefix}-pg-vol"
+        cls.containers = [cls.pg_container]
+        cls.volumes = [cls.pg_volume]
+
+        run(["docker", "volume", "create", cls.pg_volume])
+        run(
+            [
+                "docker",
+                "run",
+                "-d",
+                "--name",
+                cls.pg_container,
+                "-e",
+                "POSTGRES_PASSWORD=pgpw",
+                "-e",
+                "POSTGRES_DB=appdb",
+                "-e",
+                "POSTGRES_USER=postgres",
+                # The entrypoint evals this into its initdb call, so the host
+                # lines of pg_hba demand scram while the local socket stays
+                # trust - the entrypoint's own init and the seeding below keep
+                # working, and only a TCP connection needs the password.
+                "-e",
+                "POSTGRES_INITDB_ARGS=--auth-host=scram-sha-256",
+                "-v",
+                f"{cls.pg_volume}:{POSTGRES_DATA_DIR}",
+                POSTGRES_IMAGE,
+            ]
+        )
+        wait_for_postgres(cls.pg_container, user="postgres", timeout_s=90)
+
+        run(
+            [
+                "docker",
+                "exec",
+                cls.pg_container,
+                "sh",
+                "-lc",
+                (
+                    'psql -U postgres -d appdb -c "CREATE TABLE t (id int primary '
+                    "key, v text); INSERT INTO t VALUES (1,'ok');\""
+                ),
+            ],
+            check=True,
+        )
+
+        cls.unauthenticated = run(
+            [
+                "docker",
+                "exec",
+                cls.pg_container,
+                "sh",
+                "-lc",
+                "pg_dump -U postgres -d appdb -h localhost",
+            ],
+            capture=True,
+            check=False,
+        )
+
+        cls.databases_csv = f"/tmp/{cls.prefix}/databases.csv"
+        write_databases_csv(
+            cls.databases_csv,
+            [
+                (cls.pg_container, "appdb", "postgres", "pgpw"),
+                (cls.pg_container, "*", "postgres", "pgpw"),
+            ],
+        )
+
+        backup_run(
+            backups_dir=cls.backups_dir,
+            repo_name=cls.repo_name,
+            compose_dir=cls.compose_dir,
+            databases_csv=cls.databases_csv,
+            database_containers=[cls.pg_container],
+            images_no_stop_required=[POSTGRES_IMAGE],
+            only_sql=True,
+        )
+        cls.hash, cls.version = latest_version_dir(cls.backups_dir, cls.repo_name)
+
+    @classmethod
+    def tearDownClass(cls) -> None:
+        cleanup_docker(containers=cls.containers, volumes=cls.volumes)
+
+    def volume_dir(self) -> Path:
+        return backup_path(
+            self.backups_dir, self.repo_name, self.version, self.pg_volume
+        )
+
+    def test_a_dump_without_the_password_is_refused_by_the_server(self) -> None:
+        """Without this the module is vacuous: a pg_hba still saying trust would
+        let a baudolo that forwards nothing pass just as well."""
+        self.assertNotEqual(self.unauthenticated.returncode, 0)
+        self.assertIn("no password supplied", self.unauthenticated.stderr or "")
+
+    def test_the_configured_database_was_dumped(self) -> None:
+        dump = self.volume_dir() / SQL_DIR / f"appdb{DUMP_SUFFIX}"
+        self.assertTrue(dump.is_file(), f"expected a dump at {dump}")
+        self.assertIn("Dumped by pg_dump", dump.read_text(encoding="utf-8"))
+
+    def test_the_dump_carries_the_payload(self) -> None:
+        """pg_dump emits table data as COPY ... FROM stdin, so the row reads as
+        tab-separated values rather than as an INSERT literal."""
+        dump = self.volume_dir() / SQL_DIR / f"appdb{DUMP_SUFFIX}"
+        self.assertIn("COPY public.t (id, v) FROM stdin;", dump.read_text("utf-8"))
+        self.assertIn("1\tok", dump.read_text(encoding="utf-8"))
+
+    def test_the_cluster_row_was_dumped_too(self) -> None:
+        cluster = self.volume_dir() / SQL_DIR / f"{self.pg_container}{CLUSTER_SUFFIX}"
+        self.assertTrue(cluster.is_file(), f"expected a cluster dump at {cluster}")
+        self.assertIn("CREATE DATABASE", cluster.read_text(encoding="utf-8"))
+
+    def test_only_sql_left_no_file_copy_behind(self) -> None:
+        self.assertFalse((self.volume_dir() / FILES_DIR).exists())
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/unit/backup/test_docker_exec_env.py

@@ -0,0 +1,61 @@
+"""How a secret reaches the command running inside the container."""
+
+from __future__ import annotations
+
+import unittest
+
+from baudolo.backup.db import fallback_pg_dumpall
+from baudolo.backup.docker import docker_exec_argv
+
+
+class TestForwardEnv(unittest.TestCase):
+    def test_nothing_is_added_when_no_variable_is_named(self) -> None:
+        self.assertEqual(
+            docker_exec_argv("c1", ["true"]),
+            ["docker", "exec", "c1", "true"],
+        )
+
+    def test_a_named_variable_is_forwarded_without_its_value(self) -> None:
+        """-e NAME=value would publish the secret in the host's process list."""
+        argv = docker_exec_argv("c1", ["true"], forward_env=["PGPASSWORD"])
+        self.assertEqual(argv, ["docker", "exec", "-e", "PGPASSWORD", "c1", "true"])
+
+    def test_the_flag_precedes_the_container(self) -> None:
+        """docker reads options before the container name, arguments after it."""
+        argv = docker_exec_argv(
+            "c1", ["pg_dump", "-U", "u"], interactive=True, forward_env=["PGPASSWORD"]
+        )
+        self.assertLess(argv.index("-e"), argv.index("c1"))
+        self.assertLess(argv.index("-i"), argv.index("c1"))
+        self.assertGreater(argv.index("pg_dump"), argv.index("c1"))
+
+    def test_several_variables_each_get_their_own_flag(self) -> None:
+        argv = docker_exec_argv("c1", ["true"], forward_env=["A", "B"])
+        self.assertEqual(argv[:6], ["docker", "exec", "-e", "A", "-e", "B"])
+
+
+class TestPostgresDumpCarriesThePassword(unittest.TestCase):
+    def test_the_cluster_dump_forwards_pgpassword(self) -> None:
+        seen: dict = {}
+
+        def fake(command, out_file, *, env=None):
+            seen["command"] = command
+            seen["env"] = env
+
+        import baudolo.backup.db as db
+
+        original = db.execute_to_file
+        db.execute_to_file = fake
+        try:
+            fallback_pg_dumpall("pg", "user", "secret", "/tmp/out.sql")
+        finally:
+            db.execute_to_file = original
+
+        self.assertIn("-e", seen["command"])
+        self.assertEqual(seen["command"][seen["command"].index("-e") + 1], "PGPASSWORD")
+        self.assertEqual(seen["env"], {"PGPASSWORD": "secret"})
+        self.assertNotIn("secret", seen["command"])
+
+
+if __name__ == "__main__":
+    unittest.main()

tests/unit/backup/test_get_instance.py

@@ -0,0 +1,80 @@
+"""Which databases.csv instance a container name resolves to.
+
+The cases are the container names real deployments produce, in both compose
+and swarm, so a change to the regex has to state which shape it gives up.
+"""
+
+from __future__ import annotations
+
+import unittest
+
+from baudolo.backup.db import get_instance
+
+
+class TestDeclaredContainers(unittest.TestCase):
+    def test_a_declared_container_is_its_own_instance(self) -> None:
+        self.assertEqual(
+            get_instance("postgres-central", ["postgres-central"]), "postgres-central"
+        )
+
+    def test_a_declaration_beats_the_regex(self) -> None:
+        """A declared name is taken whole even when it carries a token the
+        fallback would otherwise strip."""
+        self.assertEqual(
+            get_instance("shop-database", ["shop-database"]), "shop-database"
+        )
+
+    def test_a_qualified_central_name_still_has_to_be_declared(self) -> None:
+        self.assertIsNone(get_instance("postgres-central", []))
+
+
+class TestContainersNamedAfterTheirEngine(unittest.TestCase):
+    def test_a_bare_engine_name_is_its_own_instance(self) -> None:
+        for name in ("postgres", "mariadb", "mysql", "db", "database"):
+            with self.subTest(container=name):
+                self.assertEqual(get_instance(name, []), name)
+
+    def test_a_swarm_task_of_such_a_container_keeps_the_instance(self) -> None:
+        self.assertEqual(get_instance("postgres_postgres.1.k3f9x2", []), "postgres")
+        self.assertEqual(get_instance("mariadb_mariadb.1.k3f9x2", []), "mariadb")
+
+
+class TestDedicatedEngines(unittest.TestCase):
+    def test_compose_names_the_container_with_a_hyphen(self) -> None:
+        self.assertEqual(get_instance("discourse-database", []), "discourse")
+
+    def test_swarm_names_the_task_with_an_underscore_and_a_slot(self) -> None:
+        """Swarm suppresses container_name and names the task
+        <stack>_<service>.<slot>.<id>, which must land on the same instance as
+        the compose name so one databases.csv serves both modes."""
+        self.assertEqual(get_instance("discourse_database.1.k3f9x2", []), "discourse")
+
+    def test_an_explicitly_named_engine_keeps_its_entity(self) -> None:
+        self.assertEqual(get_instance("bigbluebutton-postgres-1", []), "bigbluebutton")
+
+    def test_the_short_token_is_stripped_too(self) -> None:
+        self.assertEqual(get_instance("matomo-db", []), "matomo")
+
+    def test_mariadb_uses_the_same_suffix(self) -> None:
+        self.assertEqual(get_instance("matomo-database", []), "matomo")
+
+    def test_an_engine_named_suffix_is_stripped_too(self) -> None:
+        self.assertEqual(get_instance("shop-mariadb", []), "shop")
+        self.assertEqual(get_instance("shop-mysql", []), "shop")
+
+
+class TestApplicationContainers(unittest.TestCase):
+    def test_a_bare_application_name_is_not_a_database(self) -> None:
+        """Returning the name unchanged here would offer the application as a
+        second engine for its own dedicated database's instance."""
+        self.assertIsNone(get_instance("discourse", []))
+
+    def test_a_swarm_application_task_is_not_a_database(self) -> None:
+        self.assertIsNone(get_instance("discourse_discourse.1.k3f9x2", []))
+
+    def test_an_application_that_merely_starts_with_a_token_is_not_split(self) -> None:
+        self.assertIsNone(get_instance("dbeaver", []))
+
+
+if __name__ == "__main__":
+    unittest.main()