Refactor systemctl services and categories due to alarm bugs

This commit restructures systemctl service definitions and category mappings.

Motivation: Alarm-related bugs revealed inconsistencies in service and role handling.

Preparation step: lays the groundwork for fixing the alarm issues by aligning categories, roles, and service templates.

roles/sys-ctl-bkp-docker-2-loc/README.md

@@ -0,0 +1,26 @@
+# Backup Docker to Local
+
+## Description
+
+This Ansible role automates the process of backing up Docker volumes to a local folder. It pulls the [backup-docker-to-local](https://github.com/kevinveenbirkenbach/backup-docker-to-local), installs required software, configures systemd services for both standard and "everything" backup modes, and seeds backup database entries as needed.
+
+## Overview
+
+Optimized for Archlinux, this role ensures that Docker volume backups are performed reliably with minimal manual intervention. It integrates with several dependent roles to verify backup success and manage related tasks, including:
+- [sys-svc-directory-validator](../sys-svc-directory-validator/) – Validates backup directories.
+- [sys-ctl-cln-faild-bkps](../sys-ctl-cln-faild-bkps/) – Cleans up unsuccessful backup attempts.
+- [sys-timer](../sys-timer/) – Schedules recurring backup tasks.
+- [sys-bkp-provider](../sys-bkp-provider/) – Manages backup sources.
+- [sys-lock](../sys-lock/) – Ensures coordinated maintenance operations.
+
+## Purpose
+
+Backup Docker Volumes to Local is a comprehensive solution that leverages rsync to create incremental backups of Docker volumes, providing seamless recovery for both file and database data. Ideal for ensuring the integrity and security of your container data, this role sets up the necessary environment to safeguard your Docker volumes.
+
+## Features
+
+- **Required Software Installation:** Installs necessary packages (e.g., lsof, python-pandas) via pacman.
+- **Git Repository Pull:** Automatically pulls the latest version of the [backup-docker-to-local](https://github.com/kevinveenbirkenbach/backup-docker-to-local).
+- **Systemd Service Configuration:** Deploys and reloads two systemd service templates to manage backup tasks.
+- **Database Seeding:** Includes tasks to seed and manage a backup database (`databases.csv`) for tracking backup details.
+- **Dependency Integration:** Works in conjunction with the dependent roles listed above to verify and manage backups.

roles/sys-ctl-bkp-docker-2-loc/Todo.md

@@ -0,0 +1,2 @@
+# Todos
+- Add to all of the applications the correct backup procedures.

roles/sys-ctl-bkp-docker-2-loc/filter_plugins/dict_to_cli_args.py

@@ -0,0 +1,36 @@
+def dict_to_cli_args(data):
+    """
+    Convert a dictionary into CLI argument string.
+    Example:
+    {
+        "backup-dir": "/mnt/backups",
+        "shutdown": True,
+        "ignore-volumes": ["redis", "memcached"]
+    }
+    becomes:
+    --backup-dir=/mnt/backups --shutdown --ignore-volumes="redis memcached"
+    """
+    if not isinstance(data, dict):
+        raise TypeError("Expected a dictionary for CLI argument conversion")
+
+    args = []
+
+    for key, value in data.items():
+        cli_key = f"--{key}"
+
+        if isinstance(value, bool):
+            if value:
+                args.append(cli_key)
+        elif isinstance(value, list):
+            items = " ".join(map(str, value))
+            args.append(f'{cli_key}="{items}"')
+        elif value is not None:
+            args.append(f'{cli_key}={value}')
+
+    return " ".join(args)
+
+class FilterModule(object):
+    def filters(self):
+        return {
+            'dict_to_cli_args': dict_to_cli_args
+        }

roles/sys-ctl-bkp-docker-2-loc/filter_plugins/find_dock_val_by_bkp_entr.py

@@ -0,0 +1,37 @@
+from ansible.errors import AnsibleFilterError
+
+def find_dock_val_by_bkp_entr(applications, search_string, mapped_entry):
+    """
+    Iterates over all applications and their docker.services, finds services where
+    .backup[search_string] exists (and is truthy), and returns the value of
+    .[mapped_entry] for each.
+
+    :param applications: dict of applications
+    :param search_string: string, the backup subkey to search for (e.g., "enabled")
+    :param mapped_entry: string, the key to return from the service (e.g., "name")
+    :return: list of mapped_entry values
+    """
+    if not isinstance(applications, dict):
+        raise AnsibleFilterError("applications must be a dict")
+
+    results = []
+
+    for app in applications.values():
+        docker = app.get("docker", {})
+        services = docker.get("services", {})
+        if not isinstance(services, dict):
+            continue
+        for svc in services.values():
+            backup = svc.get("backup", {})
+            # only match if .backup[search_string] is set and truthy
+            if isinstance(backup, dict) and backup.get(search_string):
+                mapped_value = svc.get(mapped_entry)
+                if mapped_value is not None:
+                    results.append(mapped_value)
+    return results
+
+class FilterModule(object):
+    def filters(self):
+        return {
+            'find_dock_val_by_bkp_entr': find_dock_val_by_bkp_entr,
+        }

roles/sys-ctl-bkp-docker-2-loc/handlers/main.yml

@@ -0,0 +1,9 @@
+- name: "reload backup docker to local (all) service"
+  systemd:
+    name: "{{ BKP_DOCKER_2_LOC_SERVICE_ALL }}"
+    daemon_reload: yes
+
+- name: "reload backup docker to local service"
+  systemd:
+    name: "{{ BKP_DOCKER_2_LOC_SERVICE }}"
+    daemon_reload: yes

roles/sys-ctl-bkp-docker-2-loc/meta/main.yml

@@ -0,0 +1,25 @@
+galaxy_info:
+  author: "Kevin Veen-Birkenbach"
+  description: "Automates the backup of Docker volumes to a local folder"
+  license: "Infinito.Nexus NonCommercial License"
+  license_url: "https://s.infinito.nexus/license"
+  company: |
+    Kevin Veen-Birkenbach
+    Consulting & Coaching Solutions
+    https://www.veen.world
+  min_ansible_version: "2.9"
+  platforms:
+  - name: Archlinux
+    versions:
+    - rolling
+  galaxy_tags:
+  - backup
+  - docker
+  - local
+  - systemd
+  - automation
+  repository: "https://s.infinito.nexus/code"
+  issue_tracker_url: "https://s.infinito.nexus/issues"
+  documentation: "https://docs.infinito.nexus"
+dependencies:
+- sys-ctl-cln-faild-bkps

roles/sys-ctl-bkp-docker-2-loc/tasks/01_core.yml

@@ -0,0 +1,37 @@
+- name: Include dependencies
+  include_role:
+    name: '{{ item }}'
+  loop:
+  - sys-bkp-provider
+  - sys-ctl-alm-compose
+  - sys-lock
+  - sys-svc-directory-validator
+
+- include_tasks: 02_pkgmgr_routines.yml
+  when: backup_docker_to_local_folder is not defined
+
+- name: "reset (if enabled)"
+  include_tasks: 03_reset.yml
+  when: MODE_RESET | bool
+
+- name: "setup '{{ BKP_DOCKER_2_LOC_SERVICE_ALL }}'"
+  template:
+    src: "{{ role_name }}-everything.service.j2"
+    dest: /etc/systemd/system/{{ BKP_DOCKER_2_LOC_SERVICE_ALL }}
+  notify: reload backup docker to local (all) service
+
+- name: "setup '{{ BKP_DOCKER_2_LOC_SERVICE }}'"
+  template:
+    src: "{{ role_name }}.service.j2"
+    dest: /etc/systemd/system/{{ BKP_DOCKER_2_LOC_SERVICE }}
+  notify: reload backup docker to local service
+
+- name: "set 'service_name' to '{{ role_name }}'"
+  set_fact:
+    service_name: "{{ role_name }}"
+
+- name: "include role for sys-timer for {{ service_name }}"
+  include_role:
+    name: sys-timer
+  vars:
+    on_calendar: "{{SYS_SCHEDULE_BACKUP_DOCKER_TO_LOCAL}}"

roles/sys-ctl-bkp-docker-2-loc/tasks/02_pkgmgr_routines.yml

@@ -0,0 +1,19 @@
+- block:
+    - name: "pkgmgr install {{ BKP_DOCKER_2_LOC_PKG }}"
+      include_role:
+        name: pkgmgr-install
+      vars:
+        package_name: "{{ BKP_DOCKER_2_LOC_PKG }}"
+
+    - name: "Retrieve {{ BKP_DOCKER_2_LOC_PKG }} path from pkgmgr"
+      command: "pkgmgr path {{ BKP_DOCKER_2_LOC_PKG }}"
+      register: pkgmgr_output
+      changed_when: false
+
+    - name: Set fact for backup_docker_to_local_folder
+      set_fact:
+        backup_docker_to_local_folder: "{{ pkgmgr_output.stdout }}/"
+      changed_when: false
+  when: backup_docker_to_local_folder is not defined
+  vars:
+    BKP_DOCKER_2_LOC_PKG: backup-docker-to-local

roles/sys-ctl-bkp-docker-2-loc/tasks/03_reset.yml

@@ -0,0 +1,4 @@
+- name: "reset {{ backup_docker_to_local_folder }}databases.csv"
+  file:
+    path: "{{ backup_docker_to_local_folder }}databases.csv"
+    state: absent

roles/sys-ctl-bkp-docker-2-loc/tasks/04_seed-database-to-backup.yml

@@ -0,0 +1,68 @@
+- include_tasks: 02_pkgmgr_routines.yml
+  when: backup_docker_to_local_folder is not defined
+
+- name: "Display all database variables"
+  debug:
+    msg: |
+      database_application_id: "{{ database_application_id | default('undefined') }}"
+      database_instance: "{{ database_instance | default('undefined') }}"
+      database_name: "{{ database_name | default('undefined') }}"
+      database_type: "{{ database_type | default('undefined') }}"
+      database_host: "{{ database_host | default('undefined') }}"
+      database_username: "{{ database_username | default('undefined') }}"
+      database_password: "{{ database_password | default('undefined') }}"
+  when: MODE_DEBUG | bool
+  no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}"
+
+- name: "fail if not all required database variables are defined"
+  fail:
+    msg: "You must define all of the following variables: database_instance, database_name, database_username, database_password"
+  when: >
+    (database_instance is defined or
+     database_name is defined or
+     database_username is defined or
+     database_password is defined) and not
+    (database_instance is defined and
+     database_name is defined and
+     database_username is defined and
+     database_password is defined)
+  no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}"
+
+- name: "seed database values in directory {{ backup_docker_to_local_folder }}"
+  command: >
+    python database_entry_seeder.py databases.csv
+    "{{ database_instance }}"
+    "{{ database_name }}"
+    "{{ database_username }}"
+    "{{ database_password }}"
+  args:
+    chdir: "{{ backup_docker_to_local_folder }}"
+  when:
+    - database_instance is defined
+    - database_name is defined
+    - database_username is defined
+    - database_password is defined
+  no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}"
+  async: "{{ ASYNC_TIME if ASYNC_ENABLED | bool else omit }}"
+  poll:  "{{ ASYNC_POLL if ASYNC_ENABLED | bool else omit }}"
+
+- name: Set file permissions for databases.csv to be readable, writable, and executable by root only
+  ansible.builtin.file:
+    path: "{{ backup_docker_to_local_folder }}databases.csv"
+    mode: '0700'
+    owner: root
+    group: root
+  when: >
+    (database_instance is defined and
+    database_name is defined and
+    database_username is defined and
+    database_password is defined) and
+    run_once_sys_ctl_bkp_docker_2_loc_file_permission is not defined
+  register: file_permission_result
+  async: "{{ ASYNC_TIME if ASYNC_ENABLED | bool else omit }}"
+  poll:  "{{ ASYNC_POLL if ASYNC_ENABLED | bool else omit }}"
+
+- name: run the backup_docker_to_local_file_permission tasks once
+  set_fact:
+    run_once_sys_ctl_bkp_docker_2_loc_file_permission: true
+  when: run_once_sys_ctl_bkp_docker_2_loc_file_permission is not defined and file_permission_result is defined and file_permission_result.changed

roles/sys-ctl-bkp-docker-2-loc/tasks/main.yml

@@ -0,0 +1,10 @@
+- block:
+    - include_tasks: 01_core.yml
+    - include_tasks: utils/run_once.yml
+  when: 
+    - run_once_sys_ctl_bkp_docker_2_loc is not defined
+
+- name: "include 04_seed-database-to-backup.yml"
+  include_tasks: 04_seed-database-to-backup.yml
+  when:
+    - BKP_DOCKER_2_LOC_DB_ENABLED | bool

roles/sys-ctl-bkp-docker-2-loc/templates/sys-ctl-bkp-docker-2-loc-everything.service.j2

@@ -0,0 +1,9 @@
+[Unit]
+Description=backup all docker volumes to local folder
+OnFailure=sys-ctl-alm-compose.{{ SOFTWARE_NAME }}@%n.service sys-ctl-cln-faild-bkps{{ SYS_SERVICE_SUFFIX }}
+
+[Service]
+Type=oneshot
+ExecStartPre=/bin/sh -c '/usr/bin/python {{ PATH_SYSTEM_LOCK_SCRIPT }} {{ SYS_SERVICE_GROUP_MANIPULATION | join(' ')  }} --ignore {{ SYS_SERVICE_GROUP_BACKUPS | reject('equalto', role_name ) | join(' ') }} --timeout "{{ SYS_TIMEOUT_BACKUP_SERVICES }}"'
+ExecStart=/bin/sh -c '{{ BKP_DOCKER_2_LOC_EXEC }} --everything'
+ExecStartPost=/bin/sh -c '/bin/systemctl start sys-ctl-rpr-docker-soft{{ SYS_SERVICE_SUFFIX }} &'

roles/sys-ctl-bkp-docker-2-loc/templates/sys-ctl-bkp-docker-2-loc.service.j2

@@ -0,0 +1,9 @@
+[Unit]
+Description=backup docker volumes to local folder
+OnFailure=sys-ctl-alm-compose.{{ SOFTWARE_NAME }}@%n.service sys-ctl-cln-faild-bkps{{ SYS_SERVICE_SUFFIX }}
+
+[Service]
+Type=oneshot
+ExecStartPre=/bin/sh -c '/usr/bin/python {{ PATH_SYSTEM_LOCK_SCRIPT }} {{ SYS_SERVICE_GROUP_MANIPULATION | join(' ')  }} --ignore {{ SYS_SERVICE_GROUP_BACKUPS | reject('equalto', role_name ~ '-everything') | join(' ') }} --timeout "{{ SYS_TIMEOUT_BACKUP_SERVICES }}"'
+ExecStart=/bin/sh -c '{{ BKP_DOCKER_2_LOC_EXEC }}'
+ExecStartPost=/bin/sh -c '/bin/systemctl start sys-ctl-rpr-docker-soft{{ SYS_SERVICE_SUFFIX }} &'

roles/sys-ctl-bkp-docker-2-loc/vars/main.yml

@@ -0,0 +1,50 @@
+# Mapping logic for backup-docker-to-local CLI arguments
+#
+# - BKP_DOCKER_2_LOC_DB_ROUTINE:       All service names where backup.database_routine is set (for --database-containers)
+# - BKP_DOCKER_2_LOC_NO_STOP_REQUIRED: All images where backup.no_stop_required is set (for --images-no-stop-required)
+# - BKP_DOCKER_2_LOC_DISABLED:         All images where backup.disabled is set (for --images-no-backup-required)
+# CLI-ready variables render these lists as argument strings.
+
+BKP_DOCKER_2_LOC_SERVICE:     "{{ role_name ~ SYS_SERVICE_SUFFIX }}"
+
+BKP_DOCKER_2_LOC_SERVICE_ALL: "{{ role_name }}-everything{{ SYS_SERVICE_SUFFIX }}"
+
+# Verify if DB is enabled
+BKP_DOCKER_2_LOC_DB_ENABLED: "{{ database_type | default('') | bool }}"
+
+# Gather mapped values as lists
+BKP_DOCKER_2_LOC_DB_ROUTINE: >-
+  {{ applications | find_dock_val_by_bkp_entr('database_routine', 'name') | list }}
+
+BKP_DOCKER_2_LOC_NO_STOP_REQUIRED: >-
+  {{ applications | find_dock_val_by_bkp_entr('no_stop_required', 'image') | list }}
+
+BKP_DOCKER_2_LOC_DISABLED: >-
+  {{ applications | find_dock_val_by_bkp_entr('disabled', 'image') | list }}
+
+# CLI argument strings (only set if list not empty)
+BKP_DOCKER_2_LOC_DB_ROUTINE_CLI: >-
+  {% if BKP_DOCKER_2_LOC_DB_ROUTINE | length > 0 -%}
+    --database-containers {{ BKP_DOCKER_2_LOC_DB_ROUTINE | join(' ') }}
+  {%- endif %}
+
+BKP_DOCKER_2_LOC_NO_STOP_REQUIRED_CLI: >-
+  {% if BKP_DOCKER_2_LOC_NO_STOP_REQUIRED | length > 0 -%}
+    --images-no-stop-required {{ BKP_DOCKER_2_LOC_NO_STOP_REQUIRED | join(' ') }}
+  {%- endif %}
+
+BKP_DOCKER_2_LOC_DISABLED_CLI: >-
+  {% if BKP_DOCKER_2_LOC_DISABLED | length > 0 -%}
+    --images-no-backup-required {{ BKP_DOCKER_2_LOC_DISABLED | join(' ') }}
+  {%- endif %}
+
+# List of CLI args for convenience (e.g. for looping or joining)
+BKP_DOCKER_2_LOC_CLI_ARGS_LIST:
+  - "{{ BKP_DOCKER_2_LOC_DB_ROUTINE_CLI }}"
+  - "{{ BKP_DOCKER_2_LOC_NO_STOP_REQUIRED_CLI }}"
+  - "{{ BKP_DOCKER_2_LOC_DISABLED_CLI }}"
+
+BKP_DOCKER_2_LOC_EXEC: >-
+  /usr/bin/python {{ backup_docker_to_local_folder }}backup-docker-to-local.py
+  --compose-dir {{ PATH_DOCKER_COMPOSE_INSTANCES }}
+  {{ BKP_DOCKER_2_LOC_CLI_ARGS_LIST | select('string') | join(' ') }}