Used streaming instead of memory to prevent overflow

.github/FUNDING.yml

@@ -0,0 +1,7 @@
+github: kevinveenbirkenbach
+
+patreon: kevinveenbirkenbach
+
+buy_me_a_coffee: kevinveenbirkenbach
+
+custom: https://s.veen.world/paypaldonate

.gitignore

@@ -0,0 +1 @@
+databases.csv

README.md

@@ -1,57 +1,101 @@
-# docker-volume-backup
+# Backup Docker Volumes to Local (baudolo) 📦🔄
-[![License: GPL v3](https://img.shields.io/badge/License-GPL%20v3-blue.svg)](./LICENSE.txt) [![Travis CI](https://travis-ci.org/kevinveenbirkenbach/docker-volume-backup.svg?branch=master)](https://travis-ci.org/kevinveenbirkenbach/docker-volume-backup)
+[![GitHub Sponsors](https://img.shields.io/badge/Sponsor-GitHub%20Sponsors-blue?logo=github)](https://github.com/sponsors/kevinveenbirkenbach) [![Patreon](https://img.shields.io/badge/Support-Patreon-orange?logo=patreon)](https://www.patreon.com/c/kevinveenbirkenbach) [![Buy Me a Coffee](https://img.shields.io/badge/Buy%20me%20a%20Coffee-Funding-yellow?logo=buymeacoffee)](https://buymeacoffee.com/kevinveenbirkenbach) [![PayPal](https://img.shields.io/badge/Donate-PayPal-blue?logo=paypal)](https://s.veen.world/paypaldonate)
 
-## goal
-This script backups all docker-volumes with the help of rsync.
 
-## scheme
+**Backup Docker Volumes to Local** is a set of Python and shell scripts that enable you to perform incremental backups of all your Docker volumes using rsync. It is designed to integrate seamlessly with [Kevin's Package Manager](https://github.com/kevinveenbirkenbach/package-manager) under the alias **baudolo**, making it easy to install and manage. The tool supports both file and database recoveries with a clear, automated backup scheme.
-It is part of the following scheme:
-![backup scheme](https://www.veen.world/wp-content/uploads/2020/12/server-backup-768x567.jpg)
-Further information you will find [in this blog post](https://www.veen.world/2020/12/26/how-i-backup-dedicated-root-servers/).
 
-## Backup
+[![License: AGPL v3](https://img.shields.io/badge/License-AGPL%20v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0) [![Docker Version](https://img.shields.io/badge/Docker-Yes-blue.svg)](https://www.docker.com) [![Python Version](https://img.shields.io/badge/Python-3.x-blue.svg)](https://www.python.org) [![GitHub stars](https://img.shields.io/github/stars/kevinveenbirkenbach/backup-docker-to-local.svg?style=social)](https://github.com/kevinveenbirkenbach/backup-docker-to-local/stargazers)
-Execute:
+
+## 🎯 Goal
+
+This project automates the backup of Docker volumes using incremental backups (rsync) and supports recovering both files and database dumps (MariaDB/PostgreSQL). A robust directory stamping mechanism ensures data integrity, and the tool also handles restarting Docker Compose services when necessary.
+
+## 🚀 Features
+
+- **Incremental Backups:** Uses rsync with `--link-dest` for efficient, versioned backups.
+- **Database Backup Support:** Backs up MariaDB and PostgreSQL databases from running containers.
+- **Volume Recovery:** Provides scripts to recover volumes and databases from backups.
+- **Docker Compose Integration:** Option to automatically restart Docker Compose services after backup.
+- **Flexible Configuration:** Easily integrated with your Docker environment with minimal setup.
+- **Comprehensive Logging:** Detailed command output and error handling for safe operations.
+
+## 🛠 Requirements
+
+- **Linux Operating System** (with Docker installed) 🐧
+- **Python 3.x** 🐍
+- **Docker & Docker Compose** 🔧
+- **rsync** installed on your system
+
+## 📥 Installation
+
+You can install **Backup Docker Volumes to Local** easily via [Kevin's Package Manager](https://github.com/kevinveenbirkenbach/package-manager) using the alias **baudolo**:
 
 ```bash
-./docker-volume-backup.sh
+pkgmgr install baudolo
 ```
 
-## Recover
+Alternatively, clone the repository directly:
-Execute:
 
 ```bash
-./docker-volume-recover.sh {{volume_name}} {{backup_path}}
+git clone https://github.com/kevinveenbirkenbach/backup-docker-to-local.git
+cd backup-docker-to-local
 ```
 
-## Debug
+## 🚀 Usage
-To checkout what's going on in the mount container type in the following command:
+
+### Backup All Volumes
+
+To backup all Docker volumes, simply run:
+
+```bash
+./backup-docker-to-local.sh
+```
+
+### Recovery
+
+#### Recover Volume Files
+
+```bash
+bash ./recover-docker-from-local.sh "{{volume_name}}" "$(sha256sum /etc/machine-id | head -c 64)" "{{version_to_recover}}"
+```
+
+#### Recover Database
+
+For example, to recover a MySQL/MariaDB database:
+
+```bash
+docker exec -i mysql_container mysql -uroot -psecret database < db.sql
+```
+
+#### Debug Mode
+
+To inspect what’s happening inside a container:
 
 ```bash
 docker run -it --entrypoint /bin/sh --rm --volumes-from {{container_name}} -v /Backups/:/Backups/ kevinveenbirkenbach/alpine-rsync
 ```
-## Manual Backup
-rsync -aPvv  '***{{source_path}}***/' ***{{destination_path}}***";
 
-## Test
+## 🔍 Backup Scheme
-Delete the volume.
 
-```bash
+The backup mechanism uses incremental backups with rsync and stamps directories with a unique hash. For more details on the backup scheme, check out [this blog post](https://blog.veen.world/blog/2020/12/26/how-i-backup-dedicated-root-servers/).  
-docker rm -f container-name
+![Backup Scheme](https://blog.veen.world/wp-content/uploads/2020/12/server-backup-1024x755.jpg)
-docker volume rm volume-name
-```
 
-Recover the volume:
+## 👨‍💻 Author
 
-```bash
+**Kevin Veen-Birkenbach**  
-docker volume create volume-name
+- 📧 [kevin@veen.world](mailto:kevin@veen.world)  
-docker run --rm -v volume-name:/recover/ -v ~/backup/:/backup/ "kevinveenbirkenbach/alpine-rsync" sh -c "rsync -avv /backup/ /recover/"
+- 🌐 [https://www.veen.world/](https://www.veen.world/)
-```
 
-Restart the container.
+## 📜 License
 
-## Optimation
+This project is licensed under the **GNU Affero General Public License v3.0**. See the [LICENSE](./LICENSE) file for details.
-This setup script is not optimized yet for performance. Please optimized this script for performance if you want to use it in a professional environment.
 
-## More information
+## 🔗 More Information
-- https://blog.ssdnodes.com/blog/docker-backup-volumes/
+
-- https://www.baculasystems.com/blog/docker-backup-containers/
+- [Docker Volumes Documentation](https://docs.docker.com/storage/volumes/)
+- [Docker Backup Volumes Blog](https://blog.ssdnodes.com/blog/docker-backup-volumes/)
+- [Backup Strategies](https://en.wikipedia.org/wiki/Incremental_backup#Incremental)
+
+---
+
+Happy Backing Up! 🚀🔐

Todo.md

@@ -0,0 +1,2 @@
+# Todo
+- Verify that restore backup is correct implemented

backup-docker-to-local.py

@@ -0,0 +1,363 @@
+#!/bin/python
+# Backups volumes of running containers
+
+import subprocess
+import os
+import re
+import pathlib
+import pandas
+from datetime import datetime
+import argparse
+
+class BackupException(Exception):
+    """Generic exception for backup errors."""
+    pass
+
+def execute_shell_command(command):
+    """Execute a shell command and return its output."""
+    print(command)
+    process = subprocess.Popen([command], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
+    out, err = process.communicate()
+    if process.returncode != 0:
+        raise BackupException(f"Error in command: {command}\nOutput: {out}\nError: {err}\nExit code: {process.returncode}")
+    return [line.decode("utf-8") for line in out.splitlines()]
+
+def create_version_directory():
+    """Create necessary directories for backup."""
+    version_dir = os.path.join(VERSIONS_DIR, BACKUP_TIME)
+    pathlib.Path(version_dir).mkdir(parents=True, exist_ok=True)
+    return version_dir
+
+def get_machine_id():
+    """Get the machine identifier."""
+    return execute_shell_command("sha256sum /etc/machine-id")[0][0:64]
+
+### GLOBAL CONFIGURATION ###
+
+DOCKER_COMPOSE_HARD_RESTART_REQUIRED = ['mailu']
+
+IMAGES_NO_STOP_REQUIRED = [
+    'akaunting',
+    'baserow',
+    'discourse',
+    'element',
+    'gitea',
+    'listmonk',
+    'mastodon',
+    'matomo',
+    'nextcloud',
+    'openproject',
+    'peertube',
+    'pixelfed',
+    'wordpress' 
+]
+
+IMAGES_NO_BACKUP_REQUIRED = [
+    'redis', 
+    'memcached'
+    ]
+
+# DEFINE CONSTANTS
+DIRNAME             = os.path.dirname(__file__)
+SCRIPTS_DIRECTORY   = pathlib.Path(os.path.realpath(__file__)).parent.parent
+DATABASES           = pandas.read_csv(os.path.join(DIRNAME, "databases.csv"), sep=";")
+REPOSITORY_NAME     = os.path.basename(DIRNAME)
+MACHINE_ID          = get_machine_id()
+BACKUPS_DIR         = '/Backups/'
+VERSIONS_DIR        = os.path.join(BACKUPS_DIR, MACHINE_ID, REPOSITORY_NAME)
+BACKUP_TIME         = datetime.now().strftime("%Y%m%d%H%M%S")
+VERSION_DIR         = create_version_directory()
+
+def get_instance(container):
+    # The function is defined to take one parameter, 'container', 
+    # which is expected to be a string.
+
+    # This line uses regular expressions to split the 'container' string.
+    # 're.split' is a method that divides a string into a list, based on the occurrences of a pattern.
+    if container in ['central-mariadb', 'central-postgres']:
+        instance_name = container
+    else:
+        instance_name = re.split("(_|-)(database|db|postgres)", container)[0]
+            
+    # The pattern "(_|-)(database|db|postgres)" is explained as follows:
+    #    - "(_|-)": Matches an underscore '_' or a hyphen '-'.
+    #    - "(database|db|postgres)": Matches one of the strings "database", "db", or "postgres".
+    # So, this pattern will match segments like "_database", "-db", "_postgres", etc.
+    # For example, in "central-db", it matches "-db".
+
+    # After splitting, [0] is used to select the first element of the list resulting from the split.
+    # This element is the string portion before the matched pattern.
+    # For "central-db", the split results in ["central", "db"], and [0] selects "central".
+
+    print(f"Extracted instance name: {instance_name}")
+    return instance_name
+
+def stamp_directory():
+    """Stamp a directory using directory-validator."""
+    stamp_command = f"python {SCRIPTS_DIRECTORY}/directory-validator/directory-validator.py --stamp {VERSION_DIR}"
+    try:
+        execute_shell_command(stamp_command)
+        print(f"Successfully stamped directory: {VERSION_DIR}")
+    except BackupException as e:
+        print(f"Error stamping directory {VERSION_DIR}: {e}")
+        exit(1)
+
+def backup_database(container, volume_dir, db_type):
+    """Backup database (MariaDB or PostgreSQL) if applicable."""
+    print(f"Starting database backup for {container} using {db_type}...")
+    instance_name = get_instance(container)
+
+    # Filter the DataFrame for the given instance_name
+    database_entries = DATABASES.loc[DATABASES['instance'] == instance_name]
+
+    # Check if there is no entry
+    if database_entries.empty:
+        raise BackupException(f"No entry found for instance '{instance_name}'")
+
+    # Get the first (and only) entry
+    for database_entry in database_entries.iloc:
+        database_name     = database_entry['database']
+        database_username = database_entry['username']
+        database_password = database_entry['password']
+        backup_destination_dir = os.path.join(volume_dir, "sql")
+        pathlib.Path(backup_destination_dir).mkdir(parents=True, exist_ok=True)
+        backup_destination_file = os.path.join(backup_destination_dir, f"{database_name}.backup.sql")
+        if db_type == 'mariadb':
+            backup_command = f"docker exec {container} /usr/bin/mariadb-dump -u {database_username} -p{database_password} {database_name} > {backup_destination_file}"
+            execute_shell_command(backup_command)
+        if db_type == 'postgres':
+            cluster_file = os.path.join(backup_destination_dir, f"{instance_name}.cluster.backup.sql")
+
+            if not database_name:
+                fallback_pg_dumpall(container, database_username, database_password, cluster_file)
+                return
+
+            try:
+                if database_password:
+                    backup_command = (
+                        f"PGPASSWORD={database_password} docker exec -i {container} "
+                        f"pg_dump -U {database_username} -d {database_name} "
+                        f"-h localhost > {backup_destination_file}"
+                    )
+                else:
+                    backup_command = (
+                        f"docker exec -i {container} pg_dump -U {database_username} "
+                        f"-d {database_name} -h localhost --no-password "
+                        f"> {backup_destination_file}"
+                    )
+                execute_shell_command(backup_command)
+            except BackupException as e:
+                print(f"pg_dump failed: {e}")
+                print(f"Falling back to pg_dumpall for instance '{instance_name}'")
+                fallback_pg_dumpall(container, database_username, database_password, cluster_file)
+        print(f"Database backup for database {container} completed.")
+
+def get_last_backup_dir(volume_name, current_backup_dir):
+    """Get the most recent backup directory for the specified volume."""
+    versions = sorted(os.listdir(VERSIONS_DIR), reverse=True)
+    for version in versions:
+        backup_dir = os.path.join(VERSIONS_DIR, version, volume_name, "files", "")
+        # Ignore current backup dir
+        if backup_dir != current_backup_dir:
+            if os.path.isdir(backup_dir):
+                return backup_dir
+    print(f"No previous backups available for volume: {volume_name}")
+    return None
+
+def getStoragePath(volume_name):
+    path = execute_shell_command(f"docker volume inspect --format '{{{{ .Mountpoint }}}}' {volume_name}")[0] 
+    return f"{path}/"
+
+def getFileRsyncDestinationPath(volume_dir):
+    path = os.path.join(volume_dir, "files")
+    return f"{path}/"
+
+def fallback_pg_dumpall(container, username, password, backup_destination_file):
+    """Fallback function to run pg_dumpall if pg_dump fails or no DB is defined."""
+    print(f"Running pg_dumpall for container '{container}'...")
+    command = (
+        f"PGPASSWORD={password} docker exec -i {container} "
+        f"pg_dumpall -U {username} -h localhost > {backup_destination_file}"
+    )
+    execute_shell_command(command)
+
+def backup_volume(volume_name, volume_dir):
+    try:
+        """Backup files of a volume with incremental backups."""
+        print(f"Starting backup routine for volume: {volume_name}")
+        files_rsync_destination_path = getFileRsyncDestinationPath(volume_dir)
+        pathlib.Path(files_rsync_destination_path).mkdir(parents=True, exist_ok=True)
+
+        last_backup_dir = get_last_backup_dir(volume_name, files_rsync_destination_path)
+        link_dest_option = f"--link-dest='{last_backup_dir}'" if last_backup_dir else ""
+
+        source_dir = getStoragePath(volume_name)
+        rsync_command = f"rsync -abP --delete --delete-excluded {link_dest_option} {source_dir} {files_rsync_destination_path}"
+        execute_shell_command(rsync_command)
+    except BackupException as e:
+        if "file has vanished" in e.args[0]:
+            print("Warning: Some files vanished before transfer. Continuing.")
+        else:
+            raise
+    print(f"Backup routine for volume: {volume_name} completed.")
+
+def get_image_info(container):
+    return execute_shell_command(f"docker inspect --format '{{{{.Config.Image}}}}' {container}")
+
+def has_image(container,image):
+    """Check if the container is using the image"""
+    image_info = get_image_info(container)
+    return image in image_info[0]
+
+def change_containers_status(containers,status):
+    """Stop a list of containers."""
+    if containers:
+        container_list = ' '.join(containers)
+        print(f"{status} containers {container_list}...")
+        execute_shell_command(f"docker {status} {container_list}")
+    else:
+        print(f"No containers to {status}.")    
+
+def get_container_with_image(containers,image):
+    for container in containers:
+        if has_image(container,image):
+            return container
+    return False
+
+def is_image_whitelisted(container, images):
+    """Check if the container's image is one of the whitelisted images."""
+    image_info = get_image_info(container)
+    container_image = image_info[0]
+
+    for image in images:
+        if image in container_image:
+            return True
+    return False
+
+def is_container_stop_required(containers):
+    """Check if any of the containers are using images that are not whitelisted."""
+    return any(not is_image_whitelisted(container, IMAGES_NO_STOP_REQUIRED) for container in containers)
+
+def create_volume_directory(volume_name):
+    """Create necessary directories for backup."""
+    volume_dir = os.path.join(VERSION_DIR, volume_name)
+    pathlib.Path(volume_dir).mkdir(parents=True, exist_ok=True)
+    return volume_dir
+
+def is_image_ignored(container):
+    """Check if the container's image is one of the ignored images."""
+    for image in IMAGES_NO_BACKUP_REQUIRED:
+        if has_image(container, image):
+            return True
+    return False
+
+def backup_with_containers_paused(volume_name, volume_dir, containers, shutdown):
+    change_containers_status(containers,'stop')
+    backup_volume(volume_name, volume_dir)
+    
+    # Just restart containers if shutdown is false
+    if not shutdown:
+        change_containers_status(containers,'start')
+
+def backup_mariadb_or_postgres(container, volume_dir):
+    '''Performs database image specific backup procedures'''
+    for image in ['mariadb','postgres']:
+        if has_image(container, image):
+            backup_database(container, volume_dir, image)
+            return True
+    return False
+
+def default_backup_routine_for_volume(volume_name, containers, shutdown):
+    """Perform backup routine for a given volume."""
+    volume_dir=""
+    for container in containers:
+        
+        # Skip ignored images
+        if is_image_ignored(container):
+            print(f"Ignoring volume '{volume_name}' linked to container '{container}' with ignored image.")
+            continue 
+
+        # Directory which contains files and sqls
+        volume_dir = create_volume_directory(volume_name)
+        
+        # Execute Database backup and exit if successfull
+        if backup_mariadb_or_postgres(container, volume_dir):
+            return
+
+    # Execute backup if image is not ignored
+    if volume_dir:    
+        backup_volume(volume_name, volume_dir)
+        if is_container_stop_required(containers):
+            backup_with_containers_paused(volume_name, volume_dir, containers, shutdown)
+
+def backup_everything(volume_name, containers, shutdown):
+    """Perform file backup routine for a given volume."""
+    volume_dir=create_volume_directory(volume_name)
+    
+    # Execute sql dumps
+    for container in containers:
+        backup_mariadb_or_postgres(container, volume_dir)
+
+    # Execute file backups
+    backup_volume(volume_name, volume_dir)
+    backup_with_containers_paused(volume_name, volume_dir, containers, shutdown)
+    
+def hard_restart_docker_services(dir_path):
+    """Perform a hard restart of docker-compose services in the given directory."""
+    try:
+        print(f"Performing hard restart for docker-compose services in: {dir_path}")
+        subprocess.run(["docker-compose", "down"], cwd=dir_path, check=True)
+        subprocess.run(["docker-compose", "up", "-d"], cwd=dir_path, check=True)
+        print(f"Hard restart completed successfully in: {dir_path}")
+    except subprocess.CalledProcessError as e:
+        print(f"Error during hard restart in {dir_path}: {e}")
+        exit(2)
+
+def handle_docker_compose_services(parent_directory):
+    """Iterate through directories and restart or hard restart services as needed."""
+    for dir_entry in os.scandir(parent_directory):
+        if dir_entry.is_dir():
+            dir_path = dir_entry.path
+            dir_name = os.path.basename(dir_path)
+            print(f"Checking directory: {dir_path}")
+            
+            docker_compose_file = os.path.join(dir_path, "docker-compose.yml")
+            
+            if os.path.isfile(docker_compose_file):
+                print(f"Found docker-compose.yml in {dir_path}.")
+                if dir_name in DOCKER_COMPOSE_HARD_RESTART_REQUIRED:
+                    print(f"Directory {dir_name} detected. Performing hard restart...")
+                    hard_restart_docker_services(dir_path)
+                else:
+                    print(f"No restart required for services in {dir_path}...")
+            else:
+                print(f"No docker-compose.yml found in {dir_path}. Skipping.")
+
+def main():
+    parser = argparse.ArgumentParser(description='Backup Docker volumes.')
+    parser.add_argument('--everything', action='store_true',
+                        help='Force file backup for all volumes and additional execute database dumps')
+    parser.add_argument('--shutdown', action='store_true',
+                        help='Doesn\'t restart containers after backup')
+    parser.add_argument('--compose-dir', type=str, required=True, help='Path to the parent directory containing docker-compose setups')
+    args = parser.parse_args()
+
+    print('Start volume backups...')
+    volume_names = execute_shell_command("docker volume ls --format '{{.Name}}'")
+    
+    for volume_name in volume_names:
+        print(f'Start backup routine for volume: {volume_name}')
+        containers = execute_shell_command(f"docker ps --filter volume=\"{volume_name}\" --format '{{{{.Names}}}}'")
+        if args.everything:
+            backup_everything(volume_name, containers, args.shutdown)
+        else:    
+            default_backup_routine_for_volume(volume_name, containers, args.shutdown)
+    stamp_directory()
+    print('Finished volume backups.')
+    
+    # Handle Docker Compose services
+    print('Handling Docker Compose services...')
+    handle_docker_compose_services(args.compose_dir)
+
+if __name__ == "__main__":
+    main()

database_entry_seeder.py

@@ -0,0 +1,50 @@
+import pandas as pd
+import argparse
+import os
+
+def check_and_add_entry(file_path, instance, database, username, password):
+    # Check if the file exists and is not empty
+    if os.path.exists(file_path) and os.path.getsize(file_path) > 0:
+        # Read the existing CSV file with header
+        df = pd.read_csv(file_path, sep=';')
+    else:
+        # Create a new DataFrame with columns if file does not exist
+        df = pd.DataFrame(columns=['instance', 'database', 'username', 'password'])
+
+    # Check if the entry exists and remove it
+    mask = (
+        (df['instance'] == instance) &
+        ((df['database'] == database) |
+        (((df['database'].isna()) | (df['database'] == '')) & (database == ''))) &
+        (df['username'] == username)
+    )
+
+    if not df[mask].empty:
+        print("Replacing existing entry.")
+        df = df[~mask]
+    else:
+        print("Adding new entry.")
+
+    # Create a new DataFrame for the new entry
+    new_entry = pd.DataFrame([{'instance': instance, 'database': database, 'username': username, 'password': password}])
+
+    # Add (or replace) the entry using concat
+    df = pd.concat([df, new_entry], ignore_index=True)
+
+    # Save the updated CSV file
+    df.to_csv(file_path, sep=';', index=False)
+
+def main():
+    parser = argparse.ArgumentParser(description="Check and replace (or add) a database entry in a CSV file.")
+    parser.add_argument("file_path", help="Path to the CSV file")
+    parser.add_argument("instance", help="Database instance")
+    parser.add_argument("database", help="Database name")
+    parser.add_argument("username", help="Username")
+    parser.add_argument("password", nargs='?', default="", help="Password (optional)")
+
+    args = parser.parse_args()
+
+    check_and_add_entry(args.file_path, args.instance, args.database, args.username, args.password)
+
+if __name__ == "__main__":
+    main()

databases.csv.tpl

@@ -0,0 +1 @@
+database;username;password;container

docker-volume-backup.sh

@@ -1,42 +0,0 @@
-#!/bin/bash
-# Just backups volumes of running containers
-# If rsync stucks consider:
-# @see https://stackoverflow.com/questions/20773118/rsync-suddenly-hanging-indefinitely-during-transfers
-#
-backup_time="$(date '+%Y%m%d%H%M%S')";
-backups_folder="/Backups/";
-repository_name="$(cd "$(dirname "$(readlink -f "${0}")")" && basename `git rev-parse --show-toplevel`)";
-machine_id="$(sha256sum /etc/machine-id | head -c 64)";
-backup_repository_folder="$backups_folder$machine_id/$repository_name/";
-for volume_name in $(docker volume ls --format '{{.Name}}');
-do
-  echo "start backup routine: $volume_name";
-  for container_name in $(docker ps -a --filter volume="$volume_name" --format '{{.Names}}');
-  do
-    echo "stop container: $container_name" && docker stop "$container_name"
-    for source_path in $(docker inspect --format "{{ range .Mounts }}{{ if eq .Type \"volume\"}}{{ if eq .Name \"$volume_name\"}}{{ println .Destination }}{{ end }}{{ end }}{{ end }}" "$container_name");
-    do
-      destination_path="$backup_repository_folder""latest/$volume_name";
-      raw_destination_path="$destination_path/raw"
-      prepared_destination_path="$destination_path/prepared"
-      log_path="$backup_repository_folder""log.txt";
-      backup_dir_path="$backup_repository_folder""diffs/$backup_time/$volume_name";
-      raw_backup_dir_path="$backup_dir_path/raw";
-      prepared_backup_dir_path="$backup_dir_path/prepared";
-      if [ -d "$destination_path" ]
-        then
-          echo "backup volume: $volume_name";
-        else
-          echo "first backup volume: $volume_name"
-          mkdir -vp "$raw_destination_path";
-          mkdir -vp "$raw_backup_dir_path";
-          mkdir -vp "$prepared_destination_path";
-          mkdir -vp "$prepared_backup_dir_path";
-      fi
-      docker run --rm --volumes-from "$container_name" -v "$backups_folder:$backups_folder" "kevinveenbirkenbach/alpine-rsync" sh -c "
-      rsync -abP --delete --delete-excluded --log-file=$log_path --backup-dir=$raw_backup_dir_path '$source_path/' $raw_destination_path";
-    done
-    echo "start container: $container_name" && docker start "$container_name";
-  done
-  echo "end backup routine: $volume_name";
-done

docker-volume-recover.sh

@@ -1,6 +0,0 @@
-#!/bin/bash
-# @param $1 Volume-Name
-volume_name="$1"
-backup_path="$2"
-docker volume create "$volume_name"
-docker run --rm -v "$volume_name:/recover/" -v "$backup_path:/backup/" "kevinveenbirkenbach/alpine-rsync" sh -c "rsync -avv /backup/ /recover/"

recover-docker-from-local.sh

@@ -0,0 +1,85 @@
+#!/bin/bash
+
+# Check minimum number of arguments
+if [ $# -lt 3 ]; then
+  echo "ERROR: Not enough arguments. Please provide at least a volume name, backup hash, and version."
+  exit 1
+fi
+
+volume_name="$1"                # Volume-Name
+backup_hash="$2"                # Hashed Machine ID
+version="$3"                    # version to recover
+
+# DATABASE PARAMETERS
+database_type="$4"              # Valid values; mariadb, postgress
+database_container="$5"         # optional
+database_password="$6"          # optional
+database_name="$7"              # optional
+database_user="$database_name"  
+
+
+backup_folder="Backups/$backup_hash/backup-docker-to-local/$version/$volume_name"
+backup_files="/$backup_folder/files"
+backup_sql="/$backup_folder/sql/$database_name.backup.sql"
+
+# DATABASE RECOVERY
+
+if [ ! -z "$database_type" ]; then
+  if [ "$database_type" = "postgres" ]; then
+    if [ -n "$database_container" ] && [ -n "$database_password" ] && [ -n "$database_name" ]; then
+      echo "Recover PostgreSQL dump"
+      export PGPASSWORD="$database_password"
+      cat "$backup_sql" | docker exec -i "$database_container" psql -v ON_ERROR_STOP=1 -U "$database_user" -d "$database_name"
+      if [ $? -ne 0 ]; then
+          echo "ERROR: Failed to recover PostgreSQL dump"
+          exit 1
+      fi
+      exit 0
+    fi
+  elif [ "$database_type" = "mariadb" ]; then
+    if [ -n "$database_container" ] && [ -n "$database_password" ] && [ -n "$database_name" ]; then
+      echo "recover mysql dump"
+      cat "$backup_sql" | docker exec -i "$database_container" mariadb -u "$database_user" --password="$database_password" "$database_name"
+      if [ $? -ne 0 ]; then
+          echo "ERROR: Failed to recover mysql dump"
+          exit 1
+      fi
+      exit 0
+    fi
+  fi
+  echo "A database backup exists, but a parameter is missing."
+  exit 1
+fi
+
+# FILE RECOVERY
+
+echo "Inspect volume $volume_name"
+docker volume inspect "$volume_name"
+exit_status_volume_inspect=$?
+
+if [ $exit_status_volume_inspect -eq 0 ]; then
+    echo "Volume $volume_name already exists"
+else
+    echo "Create volume $volume_name"
+    docker volume create "$volume_name"
+    if [ $? -ne 0 ]; then
+        echo "ERROR: Failed to create volume $volume_name"
+        exit 1
+    fi
+fi
+
+if [ -d "$backup_files" ]; then    
+  echo "recover files"
+  docker run --rm -v "$volume_name:/recover/" -v "$backup_files:/backup/" "kevinveenbirkenbach/alpine-rsync" sh -c "rsync -avv --delete /backup/ /recover/"
+  if [ $? -ne 0 ]; then
+    echo "ERROR: Failed to recover files"
+    exit 1
+  fi
+  exit 0
+else
+  echo "ERROR: $backup_files doesn't exist"
+  exit 1
+fi
+
+echo "ERROR: Unhandled case"
+exit 1

requirements.yml

@@ -0,0 +1,3 @@
+pacman:
+  - lsof
+  - python-pandas

restore_backup.py

@@ -0,0 +1,170 @@
+#!/usr/bin/env python3
+# @todo Not tested yet. Needs to be tested
+"""
+restore_backup.py
+
+A script to recover Docker volumes and database dumps from local backups.
+Supports an --empty flag to clear the database objects before import (drops all tables/functions etc.).
+"""
+import argparse
+import os
+import sys
+import subprocess
+
+
+def run_command(cmd, capture_output=False, input=None, **kwargs):
+    """Run a subprocess command and handle errors."""
+    try:
+        result = subprocess.run(cmd, check=True, capture_output=capture_output, input=input, **kwargs)
+        return result
+    except subprocess.CalledProcessError as e:
+        print(f"ERROR: Command '{' '.join(cmd)}' failed with exit code {e.returncode}")
+        if e.stdout:
+            print(e.stdout.decode())
+        if e.stderr:
+            print(e.stderr.decode())
+        sys.exit(1)
+
+
+def recover_postgres(container, password, db_name, user, backup_sql, empty=False):
+    print("Recovering PostgreSQL dump...")
+    os.environ['PGPASSWORD'] = password
+    if empty:
+        print("Dropping existing PostgreSQL objects...")
+        # Drop all tables, views, sequences, functions in public schema
+        drop_sql = """
+DO $$ DECLARE r RECORD;
+BEGIN
+  FOR r IN (
+    SELECT table_name AS name, 'TABLE' AS type FROM information_schema.tables WHERE table_schema='public'
+    UNION ALL
+    SELECT routine_name AS name, 'FUNCTION' AS type FROM information_schema.routines WHERE specific_schema='public'
+    UNION ALL
+    SELECT sequence_name AS name, 'SEQUENCE' AS type FROM information_schema.sequences WHERE sequence_schema='public'
+  ) LOOP
+    -- Use %s for type to avoid quoting the SQL keyword
+    EXECUTE format('DROP %s public.%I CASCADE', r.type, r.name);
+  END LOOP;
+END
+$$;
+"""
+        run_command([
+            'docker', 'exec', '-i', container,
+            'psql', '-v', 'ON_ERROR_STOP=1', '-U', user, '-d', db_name
+        ], input=drop_sql.encode())
+        print("Existing objects dropped.")
+    print("Importing the dump...")
+    with open(backup_sql, 'rb') as f:
+        run_command([
+            'docker', 'exec', '-i', container,
+            'psql', '-v', 'ON_ERROR_STOP=1', '-U', user, '-d', db_name
+        ], stdin=f)
+    print("PostgreSQL recovery complete.")
+
+
+def recover_mariadb(container, password, db_name, user, backup_sql, empty=False):
+    print("Recovering MariaDB dump...")
+    if empty:
+        print("Dropping existing MariaDB tables...")
+        # Disable foreign key checks
+        run_command([
+            'docker', 'exec', container,
+            'mysql', '-u', user, f"--password={password}", '-e', 'SET FOREIGN_KEY_CHECKS=0;'
+        ])
+        # Get all table names
+        result = run_command([
+            'docker', 'exec', container,
+            'mysql', '-u', user, f"--password={password}", '-N', '-e',
+            f"SELECT table_name FROM information_schema.tables WHERE table_schema = '{db_name}';"
+        ], capture_output=True)
+        tables = result.stdout.decode().split()
+        for tbl in tables:
+            run_command([
+                'docker', 'exec', container,
+                'mysql', '-u', user, f"--password={password}", '-e',
+                f"DROP TABLE IF EXISTS `{db_name}`.`{tbl}`;"
+            ])
+        # Enable foreign key checks
+        run_command([
+            'docker', 'exec', container,
+            'mysql', '-u', user, f"--password={password}", '-e', 'SET FOREIGN_KEY_CHECKS=1;'
+        ])
+        print("Existing tables dropped.")
+    print("Importing the dump...")
+    with open(backup_sql, 'rb') as f:
+        run_command([
+            'docker', 'exec', '-i', container,
+            'mariadb', '-u', user, f"--password={password}", db_name
+        ], stdin=f)
+    print("MariaDB recovery complete.")
+
+
+def recover_files(volume_name, backup_files):
+    print(f"Inspecting volume {volume_name}...")
+    inspect = subprocess.run(['docker', 'volume', 'inspect', volume_name], stdout=subprocess.DEVNULL)
+    if inspect.returncode != 0:
+        print(f"Volume {volume_name} does not exist. Creating...")
+        run_command(['docker', 'volume', 'create', volume_name])
+    else:
+        print(f"Volume {volume_name} already exists.")
+
+    if not os.path.isdir(backup_files):
+        print(f"ERROR: Backup files folder '{backup_files}' does not exist.")
+        sys.exit(1)
+
+    print("Recovering files...")
+    run_command([
+        'docker', 'run', '--rm',
+        '-v', f"{volume_name}:/recover/",
+        '-v', f"{backup_files}:/backup/",
+        'kevinveenbirkenbach/alpine-rsync',
+        'sh', '-c', 'rsync -avv --delete /backup/ /recover/'
+    ])
+    print("File recovery complete.")
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description='Recover Docker volumes and database dumps from local backups.'
+    )
+    parser.add_argument('volume_name', help='Name of the Docker volume')
+    parser.add_argument('backup_hash', help='Hashed Machine ID')
+    parser.add_argument('version', help='Version to recover')
+
+    parser.add_argument('--db-type', choices=['postgres', 'mariadb'], help='Type of database backup')
+    parser.add_argument('--db-container', help='Docker container name for the database')
+    parser.add_argument('--db-password', help='Password for the database user')
+    parser.add_argument('--db-name', help='Name of the database')
+    parser.add_argument('--empty', action='store_true', help='Drop existing database objects before importing')
+
+    args = parser.parse_args()
+
+    volume = args.volume_name
+    backup_hash = args.backup_hash
+    version = args.version
+
+    backup_folder = os.path.join('Backups', backup_hash, 'backup-docker-to-local', version, volume)
+    backup_files = os.path.join(os.sep, backup_folder, 'files')
+    backup_sql = None
+    if args.db_name:
+        backup_sql = os.path.join(os.sep, backup_folder, 'sql', f"{args.db_name}.backup.sql")
+
+    # Database recovery
+    if args.db_type:
+        if not (args.db_container and args.db_password and args.db_name):
+            print("ERROR: A database backup exists, aber ein Parameter fehlt.")
+            sys.exit(1)
+
+        user = args.db_name
+        if args.db_type == 'postgres':
+            recover_postgres(args.db_container, args.db_password, args.db_name, user, backup_sql, empty=args.empty)
+        else:
+            recover_mariadb(args.db_container, args.db_password, args.db_name, user, backup_sql, empty=args.empty)
+        sys.exit(0)
+
+    # File recovery
+    recover_files(volume, backup_files)
+
+
+if __name__ == '__main__':
+    main()

restore_postgres_databases.py

@@ -0,0 +1,96 @@
+#!/usr/bin/env python3
+"""
+Restore multiple PostgreSQL databases from .backup.sql files via a Docker container.
+
+Usage:
+  ./restore_databases.py /path/to/backup_dir container_name
+"""
+import argparse
+import subprocess
+import sys
+import os
+import glob
+
+def run_command(cmd, stdin=None):
+    """
+    Run a subprocess command and abort immediately on any failure.
+    :param cmd: list of command parts
+    :param stdin: file-like object to use as stdin
+    """
+    subprocess.run(cmd, stdin=stdin, check=True)
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Restore Postgres databases from backup SQL files via Docker container."
+    )
+    parser.add_argument(
+        "backup_dir",
+        help="Path to directory containing .backup.sql files"
+    )
+    parser.add_argument(
+        "container",
+        help="Name of the Postgres Docker container"
+    )
+    args = parser.parse_args()
+
+    backup_dir = args.backup_dir
+    container = args.container
+
+    pattern = os.path.join(backup_dir, "*.backup.sql")
+    sql_files = sorted(glob.glob(pattern))
+    if not sql_files:
+        print(f"No .backup.sql files found in {backup_dir}", file=sys.stderr)
+        sys.exit(1)
+
+    for sqlfile in sql_files:
+        # Extract database name by stripping the full suffix '.backup.sql'
+        filename = os.path.basename(sqlfile)
+        if not filename.endswith('.backup.sql'):
+            continue
+        dbname = filename[:-len('.backup.sql')]
+        print(f"=== Processing {sqlfile} → database: {dbname} ===")
+
+        # Drop the database, forcing disconnect of sessions if necessary
+        run_command([
+            "docker", "exec", "-i", container,
+            "psql", "-U", "postgres", "-c",
+            f"DROP DATABASE IF EXISTS \"{dbname}\" WITH (FORCE);"
+        ])
+
+        # Create a fresh database
+        run_command([
+            "docker", "exec", "-i", container,
+            "psql", "-U", "postgres", "-c",
+            f"CREATE DATABASE \"{dbname}\";"
+        ])
+
+        # Ensure the ownership role exists
+        print(f"Ensuring role '{dbname}' exists...")
+        run_command([
+            "docker", "exec", "-i", container,
+            "psql", "-U", "postgres", "-c",
+            (
+                "DO $$BEGIN "
+                f"IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = '{dbname}') THEN "
+                f"CREATE ROLE \"{dbname}\"; "
+                "END IF; "
+                "END$$;"
+            )
+        ])
+
+        # Restore the dump into the database by streaming file (will abort on first error)
+        print(f"Restoring dump into {dbname} (this may take a while)…")
+        with open(sqlfile, 'rb') as infile:
+            run_command([
+                "docker", "exec", "-i", container,
+                "psql", "-U", "postgres", "-d", dbname
+            ], stdin=infile)
+
+        print(f"✔ {dbname} restored.")
+
+    print("All databases have been restored.")
+
+
+if __name__ == "__main__":
+    main()