Backup Host Secrets

Description

A scheduled, deduplicating backup of the host's generated secret material to the local backup directory: the Infinito.Nexus secrets and tokens, the self-signed root CA, the Let's Encrypt account + certificates, the ACME DNS credentials, and the node identity (ssh host keys + machine-id). None of this lives in a docker volume, the NFS export or the operator inventory, so it is the gap the volume/nfs backup roles do not cover.

Overview

The role installs a systemd unit that snapshots each configured source directory into <backups>/<machine-hash>/backup-secrets-to-local/<generation>/files/<name>, hard-linked against the previous generation. Optional sources are skipped when absent (the CA in Let's Encrypt mode, ACME/certbot in self-signed mode). The generation joins the same backup tree the pull and device roles consume, so it flows to the backup host and the encrypted device automatically.

Cosmos

The diagram places Backup Host Secrets in the Infinito.Nexus cosmos: the components it deploys (capabilities), the central services it consumes (dependencies), and its outward reach (federation and bridged external networks).

flowchart LR
    subgraph deps [Dependencies]
        dep_sys_ctl_cln_faild_bkps["sys-ctl-cln-faild-bkps 💻 ⚙️"]
    end
    subgraph role [svc-bkp-secrets-2-local 💻]
        svc_secrets_2_local["secrets-2-local"]
    end
    subgraph dependents [Dependents]
        dpt_svc_bkp_volume_2_local["svc-bkp-volume-2-local 💻"]
    end
    dep_sys_ctl_cln_faild_bkps -- "1:1" --> svc_secrets_2_local
    svc_secrets_2_local -- "1:1" --> dpt_svc_bkp_volume_2_local

Solid 1:1 edges are fixed relationships; dashed 0..1 edges are conditional (enabled only in matching deployments); red 0..0 edges are turned off in this role. Node markers show the role's deploy modes (💻 host, 🐳 compose, 🐝 swarm); ❌ marks a service that is explicitly turned off, and ⚙️ an Ansible role dependency declared in meta/main.yml.

Schema

flowchart TD
    TIMER["systemd timer<br>SYS_SCHEDULE_BACKUP_SECRETS_TO_LOCAL"] --> UNIT
    PRELOAD["sys-service-loader preload<br>MODE_BACKUP, before the app pass<br>(force_flush_instant + state started)"] --> UNIT
    UNIT["svc-bkp-secrets-2-local.&lt;version&gt;.&lt;domain&gt;.service"] --> LOCK["ExecStartPre: sys-lock against the manipulation group"]
    LOCK --> SCRIPT["ExecStart: script &lt;backups_dir&gt; backup-secrets-to-local<br>secrets= ca= acme= certbot= sources"]
    SCRIPT --> SOURCES["per-source rsync snapshots<br>--link-dest previous generation (hard links)<br>optional sources skipped when absent"]
    SOURCES --> TREE["&lt;backups_dir&gt;/&lt;sha256(machine-id)&gt;/<br>backup-secrets-to-local/&lt;YYYYmmddHHMMSS&gt;/files/&lt;name&gt;"]
    UNIT -->|failure| ALARM["OnFailure: alarm"]
    TREE --> PULL["svc-bkp-remote-2-local via ssh<br>user-backup ssh-wrapper: whitelisted ls/rsync per type<br>pulls the newest generation"]

Features

Complete secret coverage: secrets + tokens, self-signed CA, Let's Encrypt tree, ACME DNS credentials, ssh host keys and machine-id in one differential generation.
Differential snapshots: rsync --link-dest per subtree against the previous generation deduplicates unchanged files.
Mode-aware: absent optional sources are skipped, not failed, so the same role works in both self-signed and Let's Encrypt setups.
Chain-native: the generation lands in the standard backup tree, so svc-bkp-remote-2-local pulls it and svc-bkp-local-2-device mirrors it onto the encrypted device without extra wiring.

Quick Setup

Development

Clone, set up the workstation, and deploy Backup Host Secrets onto the local stack:

git clone https://github.com/infinito-nexus/core.git
cd core
make onboard
make compose-deploy mode=reinstall apps=svc-bkp-secrets-2-local full_cycle=false

Production

Install Backup Host Secrets directly onto the target machine: clone the repository, install the OS prerequisites and the repository toolchain, then deploy against localhost over a local connection (no SSH, no container):

git clone https://github.com/infinito-nexus/core.git
cd core
bash scripts/install/package.sh
make install
source scripts/meta/env/load.sh

APP=svc-bkp-secrets-2-local
DOMAIN=<your-domain>
TLS_MODE=self_signed
SSH_PUBLIC_KEY="<your-ssh-public-key>"
INVENTORY=inventories/production
infinito administration inventory provision "$INVENTORY" \
  --inventory-file "$INVENTORY/devices.yml" \
  --host localhost \
  --include "$APP" \
  --vars "{\"TLS_MODE\": \"$TLS_MODE\", \"DOMAIN_PRIMARY\": \"$DOMAIN\", \"users\": {\"administrator\": {\"authorized_keys\": [\"$SSH_PUBLIC_KEY\"]}}}"
infinito administration deploy dedicated "$INVENTORY/devices.yml" \
  --password-file "$INVENTORY/.password" \
  --diff -vv

Recover

Run files/python/recover.py on the target host:

recover.py <backups>/<machine-hash>/backup-secrets-to-local/<generation>/files [--restore-node-identity]

The script first starts the role's deployed backup unit (a fresh differential generation of the live material), then mirrors each present subtree back to its fixed system path (rsync -a --delete): secrets into /var/lib/infinito/secrets, ca into /etc/<domain>/ca, acme into /etc/letsencrypt, certbot into /etc/certbot. --no-safety-backup skips the pre-recover unit run. The node subtree (ssh host keys + machine-id) is restored only with --restore-node-identity, because overwriting the running machine-id / ssh host keys changes the host's identity mid-flight — do it on a fresh host after total loss.

Credits

Implemented by Kevin Veen-Birkenbach. Part of the Infinito.Nexus Project and maintained by Kevin Veen-Birkenbach. Licensed under the Infinito.Nexus Community License (Non-Commercial).