kevinveenbirkenbach/computer-playbook
1
0 Fork 0
mirror of https://github.com/kevinveenbirkenbach/computer-playbook.git synced 2026-11-06 02:54:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(svc-bkp-nfs-2-local): scheduled differential NFS export backup

Browse Source 
Add a systemd-driven rsync --link-dest backup of the NFS export into the
baudolo-compatible local snapshot tree, plus a pre-deploy snapshot hook in
the constructor. A missing export fails the nightly unit loudly, a failed
rsync removes its incomplete generation so it cannot poison the next
--link-dest, and rsync exit 24 from the live export is tolerated.

Both pre-deploy snapshot scripts (container and nfs) now resolve the
previous deploy's unit through a version glob, so the snapshot still runs
after a project version bump. The user-backup ssh wrapper whitelists the
new backup-nfs-to-local type for downstream pulls. The nfs-server role
declares the backup as a dynamic consumer with true/false pinning
variants; svc-bkp-container-2-local drops its variants.yml because
non-subject closure members resolve foreign round indices against it and
leak its companion pins into other rounds.

Unit tests execute the real script against tempdirs and prove hardlink
deduplication by inode, history isolation, delete semantics, loud failure
on a missing source, and cleanup of incomplete generations.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
kevinveenbirkenbachandClaude Opus 4.8 committed 2026-07-10 18:43:35 +02:00
1 parent f08d71534a
commit adb8bcf696
22 files changed
+466 -97
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

group_vars/all/06_daemons.yml
+1 -4
View File
@@ -4,7 +4,6 @@
# # Meta
SYS_SERVICE_SUFFIX: ".{{ SOFTWARE_DOMAIN }}.service"
## Names
SYS_SERVICE_CLEANUP_BACKUPS: "{{ lookup('unit_name', 'sys-ctl-cln-bkps' ) }}"
SYS_SERVICE_CLEANUP_BACKUPS_FAILED: "{{ lookup('unit_name', 'sys-ctl-cln-faild-bkps' ) }}"
SYS_SERVICE_CLEANUP_ANONYMOUS_VOLUMES: "{{ lookup('unit_name', 'sys-ctl-cln-anon-volumes' ) }}"
@@ -13,13 +12,12 @@ SYS_SERVICE_CLEANUP_DOCKER: "{{ lookup('unit_name', 'sys-ctl-cln-docker'
SYS_SERVICE_OPTIMIZE_DRIVE: "{{ lookup('unit_name', 'svc-opt-ssd-hdd' ) }}"
SYS_SERVICE_BACKUP_REMOTE_2_LOCAL: "{{ lookup('unit_name', 'svc-bkp-remote-2-local' ) }}"
SYS_SERVICE_BACKUP_CONTAINER_2_LOCAL: "{{ lookup('unit_name', 'svc-bkp-container-2-local') }}"
SYS_SERVICE_BACKUP_NFS_2_LOCAL: "{{ lookup('unit_name', 'svc-bkp-nfs-2-local' ) }}"
SYS_SERVICE_REPAIR_CONTAINER_SOFT: "{{ lookup('unit_name', 'sys-ctl-rpr-container-soft' ) }}"
SYS_SERVICE_REPAIR_CONTAINER_HARD: "{{ lookup('unit_name', 'sys-ctl-rpr-container-hard' ) }}"
## On Failure
SYS_SERVICE_ON_FAILURE_COMPOSE: "{{ lookup('unit_name','sys-ctl-alm-compose@', suffix=False ) }}%n.service"
## Groups
SYS_SERVICE_GROUP_BACKUPS: >
{{ (('sys-ctl-bkp-' | get_category_entries) + ('svc-bkp-' | get_category_entries))
| map('regex_replace', '$', SYS_SERVICE_SUFFIX) | list | sort }}
@@ -40,7 +38,6 @@ SYS_SERVICE_GROUP_MAINTANANCE: >
{{ ('svc-mtn-' | get_category_entries)
| map('regex_replace', '$', SYS_SERVICE_SUFFIX) | list | sort }}
## Collection of services to manipulate the system
SYS_SERVICE_GROUP_MANIPULATION: >
{{
(
group_vars/all/07_schedule.yml
+1 -11
View File
@@ -4,12 +4,9 @@
# # Meta
SYS_TIMER_ALL_ENABLED: "{{ MODE_DEBUG }}" # Runtime Variables for Process Control - Activates all timers, independend if the handlers had been triggered
## Server Tact Variables
HOURS_SERVER_AWAKE: "6..23" # Ours in which the server is "awake" (100% working). Rest of the time is reserved for maintanance
RANDOMIZED_DELAY_SEC: "5min" # Random delay for systemd timers to avoid peak loads.
## Timeouts for all services
SYS_TIMEOUT_CONTAINER_RPR_HARD: "10min"
SYS_TIMEOUT_CONTAINER_RPR_SOFT: "{{ SYS_TIMEOUT_CONTAINER_RPR_HARD }}"
SYS_TIMEOUT_CLEANUP_SERVICES: "15min"
@@ -17,9 +14,6 @@ SYS_TIMEOUT_DOCKER_UPDATE: "20min"
SYS_TIMEOUT_STORAGE_OPTIMIZER: "{{ SYS_TIMEOUT_DOCKER_UPDATE }}"
SYS_TIMEOUT_BACKUP_SERVICES: "60min"
## On Calendar
### Schedule for health checks
SYS_SCHEDULE_HEALTH_BTRFS: "*-*-* 00:00:00" # Check once per day the btrfs for errors
SYS_SCHEDULE_HEALTH_JOURNALCTL: "*-*-* 00:00:00" # Check once per day the journalctl for errors
SYS_SCHEDULE_HEALTH_DISC_SPACE: "*-*-* 06,12,18,00:00:00" # Check four times per day if there is sufficient disc space
@@ -29,25 +23,21 @@ SYS_SCHEDULE_HEALTH_CSP_CRAWLER: "*-*-* {{ HOURS_SERVER_AWAKE }}:00:00" # Check
SYS_SCHEDULE_HEALTH_NGINX: "*-*-* {{ HOURS_SERVER_AWAKE }}:00:00" # Check once per hour if all webservices are available
SYS_SCHEDULE_HEALTH_MSMTP: "*-*-* 00:00:00" # Check once per day SMTP Server
### Schedule for cleanup tasks
SYS_SCHEDULE_CLEANUP_CERTS: "*-*-* 20:00" # Deletes and revokes unused certs once per day
SYS_SCHEDULE_CLEANUP_BACKUPS: "*-*-* 21:00" # Cleanup backups once per day, MUST be called before disc space cleanup
SYS_SCHEDULE_CLEANUP_DOCKER: "*-*-* 22:00" # Cleanup docker anonymous volumes and prune ones per day
SYS_SCHEDULE_CLEANUP_DISC_SPACE: "*-*-* 23:30" # Cleanup disc space once per day
SYS_SCHEDULE_CLEANUP_FAILED_BACKUPS: "*-*-* 04:00" # Clean up failed docker backups once per day
### Schedule for repair services
SYS_SCHEDULE_REPAIR_BTRFS_AUTO_BALANCER: "Sat *-*-01..07 00:00:00" # Execute btrfs auto balancer every first Saturday of a month
SYS_SCHEDULE_REPAIR_DOCKER_HARD: "Sun *-*-* 00:00:00" # Restart docker instances every Sunday
### Schedule for backup tasks
SYS_SCHEDULE_BACKUP_REMOTE_TO_LOCAL: "*-*-* 00:30:00" # Pull Backup of the previous day
SYS_SCHEDULE_BACKUP_CONTAINER_TO_LOCAL: "*-*-* 01:00:00" # Backup the current day
SYS_SCHEDULE_BACKUP_NFS_TO_LOCAL: "*-*-* 01:30:00" # Differential NFS export snapshot after the container backup
### Schedule for Maintenance Tasks
SYS_SCHEDULE_MAINTANANCE_LETSENCRYPT_RENEW: "*-*-2/2 10:00:00" # Renew Mailu certificates every two days (even days only)
SYS_SCHEDULE_MAINTANANCE_LETSENCRYPT_DEPLOY: "*-*-* 11,23:00:00" # Deploy letsencrypt certificates twice per day to docker containers
SYS_SCHEDULE_MAINTANANCE_NEXTCLOUD: "21" # Do nextcloud maintanace between 21:00 and 01:00
### Animation
SYS_SCHEDULE_ANIMATION_KEYBOARD_COLOR: "*-*-* *:*:00" # Change the keyboard color every minute
roles/svc-bkp-container-2-local/README.md
+34
View File
@@ -10,6 +10,40 @@ File payloads are captured with rsync hard-link snapshots; databases register th
This role installs the `baudolo` CLI, lays out the on-host backup tree, deploys the systemd service that drives the periodic run, and wires the cleanup-of-failed-backups dependency so partial snapshots are not retained.
Database seeding for individual apps is contributed by the consumer roles via `tasks/04_seed-database-to-backup.yml`, which they include conditionally once `svc-bkp-container-2-local` is in `group_names`.
## Schema
```
Nightly path (SYS_SCHEDULE_BACKUP_CONTAINER_TO_LOCAL, 01:00)
systemd timer
└─> svc-bkp-container-2-local.<version>.<domain>.service
├─ ExecStartPre: sys-lock against the manipulation group
└─ ExecStart: baudolo backup
├─ file payloads: per-volume rsync snapshots,
│ --link-dest <previous generation> (unchanged files = hard links)
├─ databases: every row in databases.csv (seeded by consumer
│ roles via tasks/04_seed-database-to-backup.yml) is dumped
│ as a consistent SQL snapshot
└─ containers: `no_stop_required` keep running,
others stop for the dump and resume
Pre-deploy path (MODE_BACKUP, tasks/stages/01_constructor.yml)
pre_deploy_snapshot.sh <unit> <databases-csv>
├─ no unit installed (fresh host, version glob checked) -> SKIP
├─ databases.csv missing/empty -> SKIP
└─ else: systemctl start <previous deploy's unit> -> nightly path above
Snapshot tree
<backups_dir>/<sha256(machine-id)>/backup-docker-to-local/<YYYYmmddHHMMSS>/...
Failure path
unit failure -> OnFailure: alarm + sys-ctl-cln-faild-bkps
-> partial generations are torn down so they cannot poison --link-dest
Downstream pull
remote host (svc-bkp-remote-2-local) --ssh--> user-backup ssh-wrapper
-> whitelisted ls/rsync per backup type -> pulls the newest generation
```
## Features
- **Per-container snapshots:** rsync `--link-dest` snapshots deduplicate unchanged files across runs.
roles/svc-bkp-container-2-local/files/pre_deploy_snapshot.sh
+9 -2
View File
@@ -2,14 +2,21 @@
# Snapshot the pre-deploy state through the backup unit the PREVIOUS deploy
# installed, before this deploy mutates anything. A missing unit or an empty
# databases.csv means a fresh/unseeded host with nothing to save yet.
# Unit names embed the project version, so after a version bump the previous
# deploy's unit differs from the requested name; fall back to a same-base
# glob and start the newest installed version.
set -euo pipefail
UNIT="${1:?usage: pre_deploy_snapshot.sh <backup-unit-name> <databases-csv>}"
DATABASES_CSV="${2:?usage: pre_deploy_snapshot.sh <backup-unit-name> <databases-csv>}"
if ! systemctl cat "${UNIT}" >/dev/null 2>&1; then
echo "SKIP: ${UNIT} not installed yet (fresh host)"
exit 0
BASE="${UNIT%%.*}"
UNIT="$(systemctl list-unit-files --no-legend "${BASE}.*.service" 2>/dev/null | awk '{print $1}' | sort | tail -n1)"
if [[ -z "${UNIT}" ]]; then
echo "SKIP: no ${BASE}.* unit installed yet (fresh host)"
exit 0
fi
fi
if [[ ! -s "${DATABASES_CSV}" ]]; then
roles/svc-bkp-container-2-local/meta/variants.yml
-56
View File
@@ -1,56 +0,0 @@
---
# Matrix-deploy variant overrides for svc-bkp-container-2-local.
# See docs/contributing/artefact/files/role/variants.md for the file format.
#
# Each variant pins a different companion topology via service-key flags
# that the CombinedResolver translates into provider role includes (per
# `utils.roles.applications.services.registry`). The matrix wrapper then
# deploys the resulting closure for each round, so apps outside the
# closure are not in scope (and any group-name-gated flags they carry
# do not fire because they are not in the inventory at all).
# Variant 0: one mariadb-backed app (web-app-gitea, primary entity
# `gitea`) + one postgres-backed app (web-app-keycloak, primary entity
# `keycloak`). Exercises the full per-app database seed + dump path
# against two distinct engines at once.
- services:
gitea: # nocheck: variants-services-match
enabled: true
shared: true
keycloak: # nocheck: variants-services-match
enabled: true
shared: true
# Variant 1: raw relational DB engines only (no app consumer on top).
# Exercises the engine-side dump path without per-app
# `04_seed-database-to-backup` entries, so the role must still produce
# a valid (possibly empty) backup tree.
- services:
postgres: # nocheck: variants-services-match
enabled: true
shared: true
mariadb: # nocheck: variants-services-match
enabled: true
shared: true
# Variant 2: directory engine only (svc-db-openldap, registered under
# service-key `ldap` via its `provides: ldap` declaration). Exercises
# the non-relational backup path: the consumer-seed mechanism is not
# applicable here, so the run must capture the directory's payload
# without any SQL handler being involved.
- services:
ldap: # nocheck: variants-services-match
enabled: true
shared: true
gitea: # nocheck: variants-services-match
enabled: false
shared: false
keycloak: # nocheck: variants-services-match
enabled: false
shared: false
mariadb: # nocheck: variants-services-match
enabled: false
shared: false
postgres: # nocheck: variants-services-match
enabled: false
shared: false
roles/svc-bkp-nfs-2-local/README.md
+52
View File
@@ -0,0 +1,52 @@
# Backup NFS to Local
## Description
A scheduled differential backup of the NFS export onto the local backup directory.
Each run stores a discrete snapshot under `<backups>/<machine-hash>/backup-nfs-to-local/<generation>/files/`, hard-linked against the previous generation via rsync `--link-dest`.
## Overview
This role installs the backup script and the systemd service that drives it on the configured schedule (`SYS_SCHEDULE_BACKUP_NFS_TO_LOCAL`), and serialises the run against the rest of the manipulation group via [sys-lock](../sys-lock/).
The source path is the NFS export base resolved from the [svc-storage-nfs-server](../svc-storage-nfs-server/) services SPOT; deploy the role on the host that serves the export.
Deploy [sys-ctl-cln-bkps](../sys-ctl-cln-bkps/) to keep the snapshot tree bounded and [user-backup](../user-backup/) so downstream hosts can pull the tree.
## Schema
```
Nightly path (SYS_SCHEDULE_BACKUP_NFS_TO_LOCAL, 01:30)
systemd timer
└─> svc-bkp-nfs-2-local.<version>.<domain>.service
├─ ExecStartPre: sys-lock against the manipulation group
└─ ExecStart: script.sh <export_base> <backups_dir> backup-nfs-to-local
├─ export_base missing -> ERROR exit 1 -> OnFailure alarm
├─ rsync -a --delete
│ --link-dest <previous generation> (unchanged files = hard links)
├─ rsync exit 24 (live export churn) -> WARN, snapshot kept
└─ rsync any other failure -> generation removed, exit != 0
Pre-deploy path (MODE_BACKUP, tasks/stages/01_constructor.yml)
pre_deploy_snapshot.sh <unit> <export_base>
├─ no unit installed (fresh host, version glob checked) -> SKIP
├─ export empty/missing -> SKIP
└─ else: systemctl start <previous deploy's unit> -> nightly path above
Snapshot tree (baudolo-compatible)
<backups_dir>/<sha256(machine-id)>/backup-nfs-to-local/<YYYYmmddHHMMSS>/files/...
Downstream pull
remote host (svc-bkp-remote-2-local) --ssh--> user-backup ssh-wrapper
-> whitelisted ls/rsync per backup type -> pulls the newest generation
```
## Features
- **Differential snapshots:** rsync `--link-dest` against the previous generation deduplicates unchanged files.
- **Baudolo-compatible layout:** snapshots land in the same `<machine-hash>/<repo>/<generation>` tree as the container backups, so pull and cleanup tooling applies unchanged.
- **Schedule-coordinated:** the systemd unit is part of the global manipulation group, so it never races backup/cleanup/repair jobs on the same host.
## Credits
Implemented by **[Kevin Veen-Birkenbach](https://www.veen.world)**.
Part of the [Infinito.Nexus Project](https://s.infinito.nexus/code) and maintained by [Kevin Veen-Birkenbach](https://www.veen.world).
Licensed under the [Infinito.Nexus Community License (Non-Commercial)](https://s.infinito.nexus/license).
roles/svc-bkp-nfs-2-local/files/pre_deploy_snapshot.sh
+29
View File
@@ -0,0 +1,29 @@
#!/usr/bin/env bash
# Snapshot the pre-deploy NFS export state through the backup unit the
# PREVIOUS deploy installed, before this deploy mutates anything. A missing
# unit or an empty export means a fresh host with nothing to save yet.
# Unit names embed the project version, so after a version bump the previous
# deploy's unit differs from the requested name; fall back to a same-base
# glob and start the newest installed version.
set -euo pipefail
UNIT="${1:?usage: pre_deploy_snapshot.sh <backup-unit-name> <export-dir>}"
EXPORT_DIR="${2:?usage: pre_deploy_snapshot.sh <backup-unit-name> <export-dir>}"
if ! systemctl cat "${UNIT}" >/dev/null 2>&1; then
BASE="${UNIT%%.*}"
UNIT="$(systemctl list-unit-files --no-legend "${BASE}.*.service" 2>/dev/null | awk '{print $1}' | sort | tail -n1)"
if [[ -z "${UNIT}" ]]; then
echo "SKIP: no ${BASE}.* unit installed yet (fresh host)"
exit 0
fi
fi
if [[ -z "$(find "${EXPORT_DIR}" -mindepth 1 -print -quit 2>/dev/null)" ]]; then
echo "SKIP: ${EXPORT_DIR} missing or empty (nothing exported yet)"
exit 0
fi
echo "Starting ${UNIT} for the pre-deploy snapshot..."
systemctl start "${UNIT}"
echo "OK: pre-deploy snapshot finished"
roles/svc-bkp-nfs-2-local/files/script.sh
+54
View File
@@ -0,0 +1,54 @@
#!/bin/bash
# Stores a hard-linked differential snapshot of the NFS export in the
# local backup directory.
#
# Arguments:
# $1 SOURCE_DIR NFS export base to back up
# $2 BACKUPS_DIR local backup root
# $3 REPO_NAME repository directory name inside the machine hash dir
set -euo pipefail
SOURCE_DIR="${1:?usage: script.sh SOURCE_DIR BACKUPS_DIR REPO_NAME}"
BACKUPS_DIR="${2:?usage: script.sh SOURCE_DIR BACKUPS_DIR REPO_NAME}"
REPO_NAME="${3:?usage: script.sh SOURCE_DIR BACKUPS_DIR REPO_NAME}"
if [[ ! -d "${SOURCE_DIR}" ]]; then
echo "ERROR: ${SOURCE_DIR} missing; this host is expected to serve the NFS export" >&2
exit 1
fi
MACHINE_HASH="$(sha256sum /etc/machine-id | cut -c1-64)"
REPO_DIR="${BACKUPS_DIR%/}/${MACHINE_HASH}/${REPO_NAME}"
GENERATION="$(date +%Y%m%d%H%M%S)"
DEST_DIR="${REPO_DIR}/${GENERATION}/files"
PREVIOUS_FILES=""
if [[ -d "${REPO_DIR}" ]]; then
PREVIOUS_GENERATION="$(find "${REPO_DIR}" -mindepth 1 -maxdepth 1 -type d | sort | tail -n1)"
if [[ -n "${PREVIOUS_GENERATION}" && -d "${PREVIOUS_GENERATION}/files" ]]; then
PREVIOUS_FILES="${PREVIOUS_GENERATION}/files/"
fi
fi
cleanup_failed_generation() {
rm -rf "${REPO_DIR:?}/${GENERATION:?}"
}
trap cleanup_failed_generation ERR
mkdir -p "${DEST_DIR}"
rsync_args=(-a --delete)
if [[ -n "${PREVIOUS_FILES}" ]]; then
rsync_args+=(--link-dest "${PREVIOUS_FILES}")
fi
rc=0
rsync "${rsync_args[@]}" "${SOURCE_DIR%/}/" "${DEST_DIR}/" || rc=$?
if [[ "${rc}" -eq 24 ]]; then
echo "WARN: rsync exit 24 (files vanished from the live export mid-transfer), snapshot kept"
elif [[ "${rc}" -ne 0 ]]; then
cleanup_failed_generation
echo "ERROR: rsync failed with exit ${rc}, incomplete generation removed" >&2
exit "${rc}"
fi
echo "OK: differential backup of ${SOURCE_DIR} stored in ${DEST_DIR}"
roles/svc-bkp-nfs-2-local/meta/main.yml
+34
View File
@@ -0,0 +1,34 @@
---
galaxy_info:
author: Kevin Veen-Birkenbach
description: Scheduled differential rsync backup of the NFS export to the local backup directory, hard-linked against the previous snapshot.
license: Infinito.Nexus Community License (Non-Commercial)
company: |
Kevin Veen-Birkenbach
https://www.veen.world
min_ansible_version: "2.9"
platforms:
- name: ArchLinux
versions:
- all
- name: Debian
versions:
- all
- name: EL
versions:
- all
- name: Fedora
versions:
- all
- name: Ubuntu
versions:
- all
galaxy_tags:
- backup
- nfs
- local
- systemd
- automation
issue_tracker_url: https://s.infinito.nexus/issues
dependencies:
- sys-ctl-cln-faild-bkps
roles/svc-bkp-nfs-2-local/meta/services.yml
+22
View File
@@ -0,0 +1,22 @@
---
nfs-server:
enabled: "{{ 'svc-storage-nfs-server' in group_names }}"
shared: "{{ 'svc-storage-nfs-server' in group_names }}"
bond: 1
nfs-2-local:
bond: 1
enabled: true
shared: true
preload: false
provides: nfs_backup
run_after:
- svc-storage-nfs-server
lifecycle: beta
skip:
- compose
- swarm
min_storage: 0GB
cpus: "0.5"
mem_reservation: "256m"
mem_limit: "512m"
pids_limit: 256
roles/svc-bkp-nfs-2-local/meta/variants.yml
+9
View File
@@ -0,0 +1,9 @@
---
- services:
nfs-server:
enabled: true
shared: true
- services:
nfs-server:
enabled: false
shared: false
roles/svc-bkp-nfs-2-local/tasks/01_core.yml
+21
View File
@@ -0,0 +1,21 @@
---
- include_tasks: utils/once/flag.yml
- name: 📂 Include dependencies
include_role:
name: "{{ item }}"
loop:
- sys-lock
- sys-ctl-alm-compose
- sys-ctl-cln-bkps
- user-backup
- include_role:
name: sys-service
vars:
system_service_timer_enabled: true
system_service_force_linear_sync: true
system_service_force_flush_instant: false
system_service_force_flush_final: "{{ (RUNTIME in ['act','github']) and (MODE_BACKUP | bool) }}"
system_service_suppress_flush: true
system_service_on_calendar: "{{ SYS_SCHEDULE_BACKUP_NFS_TO_LOCAL }}"
roles/svc-bkp-nfs-2-local/tasks/main.yml
+3
View File
@@ -0,0 +1,3 @@
---
- include_tasks: 01_core.yml
when: run_once_svc_bkp_nfs_2_local is not defined
roles/svc-bkp-nfs-2-local/templates/systemctl.service.j2
+8
View File
@@ -0,0 +1,8 @@
[Unit]
Description=Differential NFS export backup to the local backup directory
OnFailure={{ SYS_SERVICE_ON_FAILURE_COMPOSE }} {{ SYS_SERVICE_CLEANUP_BACKUPS_FAILED }}
[Service]
Type=oneshot
ExecStartPre={{ BIN_LOCK }} {{ SYS_SERVICE_GROUP_MANIPULATION | join(' ') }} --ignore {{ SYS_SERVICE_BACKUP_NFS_2_LOCAL }} --timeout "{{ SYS_TIMEOUT_BACKUP_SERVICES }}"
ExecStart={{ system_service_script_exec }} {{ BKP_NFS_2_LOCAL_SOURCE }} {{ DIR_BACKUPS }} {{ BKP_NFS_2_LOCAL_REPO_NAME }}
roles/svc-bkp-nfs-2-local/vars/main.yml
+6
View File
@@ -0,0 +1,6 @@
---
application_id: svc-bkp-nfs-2-local
system_service_id: "{{ application_id }}"
BKP_NFS_2_LOCAL_SOURCE: "{{ lookup('nfs_export_base') }}"
BKP_NFS_2_LOCAL_REPO_NAME: backup-nfs-to-local
roles/svc-storage-nfs-server/meta/services.yml
+5
View File
@@ -16,3 +16,8 @@ nfs-server:
mem_reservation: "64m"
mem_limit: "256m"
pids_limit: 256
nfs_backup:
bond: 1
enabled: "{{ 'svc-bkp-nfs-2-local' in group_names }}"
shared: "{{ 'svc-bkp-nfs-2-local' in group_names }}"
roles/svc-storage-nfs-server/meta/variants.yml
+10
View File
@@ -2,3 +2,13 @@
- services:
nfs-server:
flavor: "{{ 'ganesha' if RUNTIME in ['dev', 'act'] else 'kernel' }}"
nfs_backup:
enabled: true
shared: true
- services:
nfs-server:
flavor: "{{ 'ganesha' if RUNTIME in ['dev', 'act'] else 'kernel' }}"
nfs_backup:
enabled: false
shared: false
roles/user-backup/templates/ssh-wrapper.sh.j2
+16 -12
View File
@@ -6,31 +6,35 @@ then
echo "`/bin/date`: $SSH_ORIGINAL_COMMAND" | systemd-cat -t "ssh-wrapper.sh"
fi
# define executable commands
get_hashed_machine_id="sha256sum /etc/machine-id";
hashed_machine_id="$($get_hashed_machine_id | head -c 64)"
get_backup_types="find {{ DIR_BACKUPS }}/$hashed_machine_id/ -maxdepth 1 -type d -execdir basename {} ;";
# @todo This configuration is not scalable yet. If other backup services then svc-bkp-container-2-local are integrated, this logic needs to be optimized
get_version_directories="ls -d {{ DIR_BACKUPS }}/$hashed_machine_id/backup-docker-to-local/*"
last_version_directory="$($get_version_directories | tail -1)"
rsync_command="sudo rsync --server --sender -blogDtpre.iLsfxCIvu . $last_version_directory/"
backup_types="backup-docker-to-local backup-nfs-to-local"
for backup_type in $backup_types; do
get_version_directories="ls -d {{ DIR_BACKUPS }}/$hashed_machine_id/$backup_type/*"
last_version_directory="$($get_version_directories 2>/dev/null | tail -1)"
rsync_command="sudo rsync --server --sender -blogDtpre.iLsfxCIvu . $last_version_directory/"
case "$SSH_ORIGINAL_COMMAND" in
"$get_version_directories")
$get_version_directories
exit $?
;;
"$rsync_command")
$rsync_command
exit $?
;;
esac
done
# filter commands
case "$SSH_ORIGINAL_COMMAND" in
"$get_hashed_machine_id")
$get_hashed_machine_id
;;
"$get_version_directories")
$get_version_directories
;;
"$get_backup_types")
$get_backup_types
;;
"$rsync_command")
$rsync_command
;;
*)
echo "This command is not supported."
exit 1
tasks/stages/01_constructor.yml
+24 -12
View File
@@ -166,18 +166,30 @@
name: sys-token-store
when: run_once_sys_token_store is not defined
- name: "💾 Snapshot the pre-deploy state via the previous deploy's backup unit"
ansible.builtin.script:
cmd: >-
{{ lookup('path_absolute', 'roles/svc-bkp-container-2-local/files/pre_deploy_snapshot.sh') }}
{{ SYS_SERVICE_BACKUP_CONTAINER_2_LOCAL }}
{{ FILE_DATABASE_SECRETS }}
register: _pre_deploy_snapshot
changed_when: "'SKIP' not in _pre_deploy_snapshot.stdout"
timeout: "{{ lookup('timeout', 3600) | int }}"
when:
- MODE_BACKUP | bool
- IS_STACK_HOST | bool
- name: "💾 Snapshot the pre-deploy state via the previous deploy's backup units"
when: MODE_BACKUP | bool
block:
- name: "💾 Snapshot the pre-deploy container state"
ansible.builtin.script:
cmd: >-
{{ lookup('path_absolute', 'roles/svc-bkp-container-2-local/files/pre_deploy_snapshot.sh') }}
{{ SYS_SERVICE_BACKUP_CONTAINER_2_LOCAL }}
{{ FILE_DATABASE_SECRETS }}
register: _pre_deploy_snapshot
changed_when: "'SKIP' not in _pre_deploy_snapshot.stdout"
timeout: "{{ lookup('timeout', 3600) | int }}"
when: "'svc-bkp-container-2-local' in group_names"
- name: "💾 Snapshot the pre-deploy NFS export"
ansible.builtin.script:
cmd: >-
{{ lookup('path_absolute', 'roles/svc-bkp-nfs-2-local/files/pre_deploy_snapshot.sh') }}
{{ SYS_SERVICE_BACKUP_NFS_2_LOCAL }}
{{ lookup('nfs_export_base') }}
register: _pre_deploy_nfs_snapshot
changed_when: "'SKIP' not in _pre_deploy_nfs_snapshot.stdout"
timeout: "{{ lookup('timeout', 3600) | int }}"
when: "'svc-bkp-nfs-2-local' in group_names"
- name: 🛠️ Preload shared services
ansible.builtin.include_role:
tests/unit/roles/svc-bkp-nfs-2-local/__init__.py
View File
Whitespace-only changes.
tests/unit/roles/svc-bkp-nfs-2-local/files/__init__.py
+3
View File
@@ -0,0 +1,3 @@
from pathlib import Path
PROJECT_ROOT: Path = Path(__file__).resolve().parents[5]
tests/unit/roles/svc-bkp-nfs-2-local/files/test_script.py
+125
View File
@@ -0,0 +1,125 @@
"""Run roles/svc-bkp-nfs-2-local/files/script.sh against real tempdirs and
verify the differential snapshot semantics: a missing source SKIPs without
side effects, unchanged files are hard-linked between generations instead of
copied, changed files get their own inode, and deletions do not reach back
into older generations."""
import os
import subprocess
import tempfile
import time
import unittest
from pathlib import Path
from . import PROJECT_ROOT
SCRIPT = PROJECT_ROOT / "roles" / "svc-bkp-nfs-2-local" / "files" / "script.sh"
REPO_NAME = "backup-nfs-to-local"
class NfsBackupScriptTests(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
base = Path(self.tmp.name)
self.source = base / "export"
self.backups = base / "backups"
self.source.mkdir()
self.backups.mkdir()
def tearDown(self):
self.tmp.cleanup()
def _run(self, source=None):
return subprocess.run(
[
"bash",
str(SCRIPT),
str(source or self.source),
str(self.backups),
REPO_NAME,
],
capture_output=True,
text=True,
check=False,
)
def _generations(self):
machine_dirs = list(self.backups.iterdir())
self.assertEqual(len(machine_dirs), 1)
repo_dir = machine_dirs[0] / REPO_NAME
return sorted(repo_dir.iterdir())
def test_missing_source_fails_loudly_without_side_effects(self):
result = self._run(source=Path(self.tmp.name) / "does-not-exist")
self.assertEqual(result.returncode, 1)
self.assertIn("ERROR", result.stderr)
self.assertEqual(list(self.backups.iterdir()), [])
def test_first_run_creates_full_snapshot(self):
(self.source / "a.txt").write_text("hello")
result = self._run()
self.assertEqual(result.returncode, 0, result.stderr)
generations = self._generations()
self.assertEqual(len(generations), 1)
content = (
generations[0] / "files" / "a.txt"
).read_text() # nocheck: cache-read tempdir fixture
self.assertEqual(content, "hello")
def test_second_run_hardlinks_unchanged_files(self):
(self.source / "a.txt").write_text("hello")
self.assertEqual(self._run().returncode, 0)
time.sleep(1.1)
self.assertEqual(self._run().returncode, 0)
generations = self._generations()
self.assertEqual(len(generations), 2)
first = generations[0] / "files" / "a.txt"
second = generations[1] / "files" / "a.txt"
self.assertEqual(first.stat().st_ino, second.stat().st_ino)
self.assertEqual(second.stat().st_nlink, 2)
def test_changed_file_gets_own_inode_and_history_survives(self):
(self.source / "a.txt").write_text("v1")
self.assertEqual(self._run().returncode, 0)
time.sleep(1.1)
(self.source / "a.txt").write_text("v2")
self.assertEqual(self._run().returncode, 0)
generations = self._generations()
first = generations[0] / "files" / "a.txt"
second = generations[1] / "files" / "a.txt"
self.assertNotEqual(first.stat().st_ino, second.stat().st_ino)
first_content = first.read_text() # nocheck: cache-read tempdir fixture
second_content = second.read_text() # nocheck: cache-read tempdir fixture
self.assertEqual(first_content, "v1")
self.assertEqual(second_content, "v2")
@unittest.skipIf(os.geteuid() == 0, "root reads past chmod 000")
def test_failed_rsync_removes_incomplete_generation(self):
secret = self.source / "unreadable.txt"
secret.write_text("nope")
secret.chmod(0o000)
result = self._run()
self.assertNotEqual(result.returncode, 0)
self.assertIn("incomplete generation removed", result.stderr)
machine_dirs = list(self.backups.iterdir())
if machine_dirs:
self.assertEqual(list((machine_dirs[0] / REPO_NAME).iterdir()), [])
def test_deleted_file_vanishes_only_from_new_generation(self):
(self.source / "keep.txt").write_text("keep")
(self.source / "gone.txt").write_text("gone")
self.assertEqual(self._run().returncode, 0)
time.sleep(1.1)
(self.source / "gone.txt").unlink()
self.assertEqual(self._run().returncode, 0)
generations = self._generations()
self.assertTrue((generations[0] / "files" / "gone.txt").is_file())
self.assertFalse((generations[1] / "files" / "gone.txt").exists())
self.assertTrue((generations[1] / "files" / "keep.txt").is_file())
if __name__ == "__main__":
unittest.main()