Another big round of refactoring and cleaning...

roles/svc-sys-bkp-rmt-2-loc/Administration.md

@@ -0,0 +1,25 @@
+# Administration Tasks
+
+## Debug Instructions
+
+### Live Monitoring
+
+To track what the service is doing, execute one of the following commands:
+
+#### Using systemctl
+
+```bash
+watch -n2 "systemctl status sys-bkp-remote-to-local.cymais.service"
+```
+
+#### Using journalctl
+
+```bash
+journalctl -fu sys-bkp-remote-to-local.cymais.service
+```
+
+### Viewing History
+
+```bash
+sudo journalctl -u sys-bkp-remote-to-local.cymais.service
+```

roles/svc-sys-bkp-rmt-2-loc/README.md

@@ -0,0 +1,31 @@
+# Backup Remote to Local
+
+## Description
+
+This role pulls backups from a remote server and stores them locally using rsync with retry logic. It is designed to retrieve remote backup data and integrate with your overall backup scheme.
+
+## Overview
+
+Optimized for Archlinux, this role is a key component of a comprehensive backup system. It works in conjunction with other roles to ensure that backup data is collected, verified, and maintained. The role uses a Bash script to pull backups, manage remote connections, and handle incremental backup creation.
+
+## Purpose
+
+Backup Remote to Local is a robust solution for retrieving backup data from remote servers. By leveraging rsync, it creates incremental backups that support both file and database recovery. This ensures the integrity and security of your backup data across distributed environments.
+
+## Features
+
+- **Remote Backup Retrieval:** Pulls backups from a remote server using secure SSH connections.
+- **Incremental Backup with rsync:** Uses rsync with options for archive, backup, and hard linking to efficiently manage changes.
+- **Retry Logic:** Implements a retry mechanism to handle transient network issues or remote errors.
+- **Integration with Other Roles:** Works alongside roles like sys-bkp-directory-validator, sys-cln-faild-bkps, sys-timer, sys-bkp-provider, and sys-lock.
+- **Administrative Debugging:** Detailed debug instructions and administrative tasks are provided in a separate file.
+
+## Other Resources
+
+- **Backup Scheme:**  
+  ![backup scheme](https://blog.veen.world/wp-content/uploads/2020/12/srv-backup-768x567.jpg)  
+  More details can be found in [this blog post](https://blog.veen.world/2020/12/26/how-i-backup-dedicated-root-servers/).
+
+## Administration & Debugging
+
+For detailed debug instructions and administrative tasks, please refer to the [Administration Tasks](Administration.md) file.

roles/svc-sys-bkp-rmt-2-loc/files/sys-bkp-remote-to-local.sh

@@ -0,0 +1,85 @@
+#!/bin/bash
+# @param $1 hostname from which backup should be pulled
+
+echo "pulling backups from: $1" &&
+
+# error counter
+errors=0 &&
+
+echo "loading meta data..." &&
+
+remote_host="backup@$1" &&
+echo "host address:         $remote_host" &&
+
+remote_machine_id="$( (ssh "$remote_host" sha256sum /etc/machine-id) | head -c 64 )" &&
+echo "remote machine id:    $remote_machine_id" &&
+
+general_backup_machine_dir="/Backups/$remote_machine_id/" &&
+echo "backup dir:           $general_backup_machine_dir" &&
+
+remote_backup_types="$(ssh "$remote_host" "find $general_backup_machine_dir -maxdepth 1 -type d -execdir basename {} ;")" &&
+echo "backup types:          $remote_backup_types" || exit 1
+
+for backup_type in $remote_backup_types; do
+  if [ "$backup_type" != "$remote_machine_id" ]; then
+    echo "backup type:              $backup_type" &&
+    
+    general_backup_type_dir="$general_backup_machine_dir""$backup_type/" &&
+    general_versions_dir="$general_backup_type_dir" &&
+    local_previous_version_dir="$(ls -d $general_versions_dir* | tail -1)" &&
+    echo "last local backup:      $local_previous_version_dir" &&
+
+    remote_backup_versions="$(ssh "$remote_host" ls -d "$general_backup_type_dir"\*)" &&
+    echo "remote backup versions:   $remote_backup_versions" &&
+
+
+    remote_last_backup_dir=$(echo "$remote_backup_versions" | tail -1) &&
+    echo "last remote backup:       $remote_last_backup_dir" &&
+
+    remote_source_path="$remote_host:$remote_last_backup_dir/" &&
+    echo "source path:              $remote_source_path" &&
+
+    local_backup_destination_path=$remote_last_backup_dir &&
+    echo "backup destination:       $local_backup_destination_path" &&
+
+    echo "creating local backup destination folder..." &&
+    mkdir -vp "$local_backup_destination_path" &&
+
+    echo "starting backup..."
+    rsync_command='rsync -abP --delete --delete-excluded --rsync-path="sudo rsync" --link-dest="'$local_previous_version_dir'" "'$remote_source_path'" "'$local_backup_destination_path'"'
+
+    echo "executing:                $rsync_command"
+
+    retry_count=0
+    max_retries=12
+    retry_delay=300  # Retry delay in seconds (5 minutes)
+    last_retry_start=0
+    max_retry_duration=43200  # Maximum duration for a single retry attempt (12 hours)
+
+    while [[ $retry_count -lt $max_retries ]]; do
+      echo "Retry attempt: $((retry_count + 1))"
+      if [[ $retry_count -gt 0 ]]; then
+        current_time=$(date +%s)
+        last_retry_duration=$((current_time - last_retry_start))
+        if [[ $last_retry_duration -ge $max_retry_duration ]]; then
+          echo "Last retry took more than 12 hours, increasing max retries to 12."
+          max_retries=12
+        fi
+      fi
+      last_retry_start=$(date +%s)
+      eval "$rsync_command"
+      rsync_exit_code=$?
+      if [[ $rsync_exit_code -eq 0 ]]; then
+        break
+      fi
+      retry_count=$((retry_count + 1))
+      sleep $retry_delay
+    done
+
+    if [[ $rsync_exit_code -ne 0 ]]; then
+      echo "Error: rsync failed after $max_retries attempts"
+      ((errors += 1))
+    fi
+  fi
+done
+exit $errors;

roles/svc-sys-bkp-rmt-2-loc/handlers/main.yml

@@ -0,0 +1,4 @@
+- name: "reload svc-sys-bkp-rmt-2-loc service"
+  systemd:
+    name: svc-sys-bkp-rmt-2-loc.cymais.service
+    daemon_reload: yes

roles/svc-sys-bkp-rmt-2-loc/meta/main.yml

@@ -0,0 +1,34 @@
+
+---
+galaxy_info:
+  author: "Kevin Veen-Birkenbach"
+  description: "Pulls backups from a remote server and stores them locally using rsync with retry logic. This role is part of a comprehensive backup scheme and works in conjunction with other roles to ensure reliable backup operations."
+  license: "CyMaIS NonCommercial License (CNCL)"
+  license_url: "https://s.veen.world/cncl"
+  company: |
+    Kevin Veen-Birkenbach
+    Consulting & Coaching Solutions
+    https://www.veen.world
+  min_ansible_version: "2.9"
+  platforms:
+    - name: Archlinux
+      versions:
+        - rolling
+  galaxy_tags:
+    - backup
+    - remote
+    - local
+    - docker
+    - systemd
+    - automation
+  repository: "https://s.veen.world/cymais"
+  issue_tracker_url: "https://s.veen.world/cymaisissues"
+  documentation: "https://s.veen.world/cymais"
+dependencies:
+  - gen-git
+  - sys-alm-compose
+  - sys-cln-bkps-timer
+  - sys-cln-faild-bkps
+  - sys-lock
+  - user-root
+  - sys-rst-daemon

roles/svc-sys-bkp-rmt-2-loc/tasks/main.yml

@@ -0,0 +1,34 @@
+- name: "create {{docker_backup_remote_to_local_folder}}"
+  file:
+    path: "{{docker_backup_remote_to_local_folder}}"
+    state: directory
+    mode: 0755
+
+- name: create svc-sys-bkp-rmt-2-loc.sh
+  copy:
+    src: svc-sys-bkp-rmt-2-loc.sh
+    dest: "{{docker_backup_remote_to_local_folder}}svc-sys-bkp-rmt-2-loc.sh"
+    mode: 0755
+
+- name: create svc-sys-bkp-rmt-2-loc.cymais.service
+  template: 
+    src:  svc-sys-bkp-rmt-2-loc.service.j2 
+    dest: /etc/systemd/system/svc-sys-bkp-rmt-2-loc.cymais.service
+  notify: reload svc-sys-bkp-rmt-2-loc service
+
+- name: create backups-remote-to-local.sh
+  template: 
+    src: backups-remote-to-local.sh.j2 
+    dest: "{{docker_backup_remote_to_local_folder}}backups-remote-to-local.sh"
+    mode: 0755
+
+- name: "set 'service_name' to '{{ role_name }}'"
+  set_fact:
+    service_name: "{{ role_name }}"
+
+- name: "include role for sys-timer for {{service_name}}"
+  include_role:
+    name: sys-timer
+  vars:
+    on_calendar:  "{{on_calendar_backup_remote_to_local}}"
+

roles/svc-sys-bkp-rmt-2-loc/templates/backups-remote-to-local.sh.j2

@@ -0,0 +1,8 @@
+#!/bin/bash
+# Pulls the remote backups from multiple hosts
+hosts="{{ pull_remote_backups | join(' ') }}";
+errors=0
+for host in $hosts; do
+  bash {{ docker_backup_remote_to_local_folder }}svc-sys-bkp-rmt-2-loc.sh $host || ((errors+=1));
+done;
+exit $errors;

roles/svc-sys-bkp-rmt-2-loc/templates/sys-bkp-remote-to-local.service.j2

@@ -0,0 +1,8 @@
+[Unit]
+Description=pull remote backups
+OnFailure=sys-alm-compose.cymais@%n.service sys-cln-faild-bkps.cymais.service
+
+[Service]
+Type=oneshot
+ExecStartPre=/bin/sh -c '/usr/bin/python {{ path_system_lock_script }} {{ system_maintenance_services | join(' ')  }} --ignore {{system_maintenance_backup_services| join(' ') }} --timeout "{{system_maintenance_lock_timeout_backup_services}}"'
+ExecStart=/bin/sh -c '/usr/bin/bash {{docker_backup_remote_to_local_folder}}backups-remote-to-local.sh'

roles/svc-sys-bkp-rmt-2-loc/vars/main.yml

@@ -0,0 +1,2 @@
+docker_backup_remote_to_local_folder: '{{path_administrator_scripts}}svc-sys-bkp-rmt-2-loc/'
+application_id: svc-sys-bkp-rmt-2-loc