fix(backup): carry hard links, ACLs and xattrs on every backup and recover hop

baudolo 7.1.x writes volume generations with `rsync -aHAX`, so the
producing host keeps `user.*` extended attributes (OpenCloud keeps its
storage metadata in `user.oc.*`). Every later copy dropped them again,
because none of the other rsync calls passed `-X` (nor `-A`/`-H`):

- NFS export snapshots (svc-bkp-nfs-2-local `script.sh`),
- the backup host's pull (svc-bkp-remote-2-local `pull_specific_host.py`),
- the device sync (svc-bkp-local-2-device `script.py`),
- every recover into a live target (`utils/recovery/base.py`, used by the
  volume, NFS and device recover scripts),
- the `recover full` and `recover remote` tree pulls.

A disaster recovery through the documented chain therefore restored
OpenCloud without `user.oc.*`, silently: the copies succeeded and the
dirval stamp does not cover attributes.

All of these now run `rsync -aHAX` (`-aHAXP` for the device sync). A
target filesystem without xattr support makes rsync exit non-zero as soon
as a source file carries attributes, so the hop fails loudly instead of
dropping them; there is no per-device opt-out. The single-image pull in
`recover remote` keeps `-a`. The provider's ssh-wrapper needs no change:
its glob accepts any server flags and its character guard allows them.

Two tests write a `user.oc.id` attribute and assert it survives the NFS
snapshot and the recover; with the old `-a` both fail with
`OSError: [Errno 61] No data available`.

Closes #1021

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

cli/administration/recover/README.md

@@ -78,7 +78,7 @@ expanding `volume` per docker volume.
 
 ## Safety
 
-Each recover mirrors a snapshot with `rsync -a --delete` and first runs the
+Each recover mirrors a snapshot with `rsync -aHAX --delete` and first runs the
 role's backup unit as a rollback point. `--no-safety-backup` skips that
 (only when the target holds nothing worth saving: fresh host / empty /
 disposable). `--preview` prints the exact commands without running them.

cli/administration/recover/full.py

@@ -72,7 +72,7 @@ def plan(root: Path) -> tuple[list[tuple[str, str]], list[str]]:
 
 def _pull_cmd(source: str) -> list[str]:
     remote = source if ":" in source else f"{source}:{paths.BACKUP_ROOT}"
-    return ["rsync", "-a", f"{remote}/", f"{PULL_STAGE}/"]
+    return ["rsync", "-aHAX", f"{remote}/", f"{PULL_STAGE}/"]
 
 
 def _is_device(source: str) -> bool:

cli/administration/recover/remote.py

@@ -47,7 +47,7 @@ def commands(
         return (
             [
                 ["ssh", host, shlex.join(remote)],
-                ["rsync", "-a", f"{host}:{REMOTE_ROOT}/", f"{LOCAL_ROOT}/"],
+                ["rsync", "-aHAX", f"{host}:{REMOTE_ROOT}/", f"{LOCAL_ROOT}/"],
             ],
             LOCAL_ROOT,
         )

cli/administration/recover/schema.yml

@@ -58,4 +58,4 @@ flags:
     skip the pre-recover safety backup of the current target
     (only when it holds nothing worth saving: fresh host / empty / disposable)
 
-mechanism: "rsync -a --delete, preceded by a pre-recover safety backup by default"
+mechanism: "rsync -aHAX --delete, preceded by a pre-recover safety backup by default"

roles/svc-bkp-local-2-device/README.md

@@ -97,7 +97,7 @@ Run `files/python/recover.py` on the host with the backup device attached:
 recover.py /dev/sdX1 /mnt/usb-recover /var/lib/infinito/backup [--snapshot <timestamp>]
 ```
 
-The script opens the LUKS device (interactive passphrase prompt; `--passphrase-stdin` for automation), mounts it, picks the newest device snapshot across every machine hash (a fresh host after total loss has a new machine id), mirrors it into the target (`rsync -a --delete`) and unmounts/closes the device again. No pre-recover service backup runs: the device itself is the backup this role maintains. Afterwards restore individual applications from the backup root with the matching role's `recover.py` (`svc-bkp-volume-2-local` for docker volumes, `svc-bkp-nfs-2-local` for NFS exports).
+The script opens the LUKS device (interactive passphrase prompt; `--passphrase-stdin` for automation), mounts it, picks the newest device snapshot across every machine hash (a fresh host after total loss has a new machine id), mirrors it into the target (`rsync -aHAX --delete`) and unmounts/closes the device again. No pre-recover service backup runs: the device itself is the backup this role maintains. Afterwards restore individual applications from the backup root with the matching role's `recover.py` (`svc-bkp-volume-2-local` for docker volumes, `svc-bkp-nfs-2-local` for NFS exports).
 
 ## Credits
 

roles/svc-bkp-local-2-device/files/python/recover.py

@@ -8,7 +8,7 @@ Opens the LUKS device (interactive passphrase prompt, or stdin with
 ``--passphrase-stdin``), mounts it, picks the newest device snapshot
 (any machine hash: a fresh host after total loss has a new
 /etc/machine-id) and mirrors it into the target
-(``rsync -a --delete``), then unmounts and closes the device again.
+(``rsync -aHAX --delete``), then unmounts and closes the device again.
 No pre-recover service backup runs: the device itself is the backup
 this role maintains.
 

roles/svc-bkp-local-2-device/files/python/script.py

@@ -53,7 +53,7 @@ def main():
     try:
         rsync_command = [
             "rsync",
-            "-aP",
+            "-aHAXP",
             "--numeric-ids",
             "--delete",
             "--delete-excluded",

roles/svc-bkp-nfs-2-local/README.md

@@ -44,7 +44,7 @@ flowchart TD
     UNIT["svc-bkp-nfs-2-local.<version>.<domain>.service"] --> LOCK["ExecStartPre: sys-lock against the manipulation group"]
     LOCK --> SCRIPT["ExecStart: script.sh &lt;export_base&gt; &lt;backups_dir&gt;<br>backup-nfs-to-local &lt;state&gt;/backup"]
     SCRIPT -->|export_base missing| ALARM["ERROR exit 1<br>OnFailure alarm"]
-    SCRIPT --> RSYNC["rsync -a --delete<br>--exclude /&lt;state&gt;/backup (no backups-in-backups)<br>--link-dest previous generation (hard links)"]
+    SCRIPT --> RSYNC["rsync -aHAX --delete<br>--exclude /&lt;state&gt;/backup (no backups-in-backups)<br>--link-dest previous generation (hard links)"]
     RSYNC -->|exit 24: live export churn| WARN["WARN, snapshot kept"]
     RSYNC -->|any other failure| CLEAN["generation removed, exit != 0"]
     RSYNC --> TREE["&lt;backups_dir&gt;/&lt;sha256(machine-id)&gt;/<br>backup-nfs-to-local/&lt;YYYYmmddHHMMSS&gt;/files/..."]
@@ -105,7 +105,7 @@ recover.py <backups>/<machine-hash>/backup-nfs-to-local/<generation>/files/<stat
 ```
 
 1. Stop every stack consuming the subtree: `docker stack rm <stack>`.
-2. Run the script; it first starts the role's deployed backup unit (a fresh differential `backup-nfs-to-local` generation of the live export), then mirrors the snapshot into the target (`rsync -a --delete`, with the shared `<state>/backup` root protected from deletion and never copied in). `--no-safety-backup` skips the unit run when the target holds nothing worth saving.
+2. Run the script; it first starts the role's deployed backup unit (a fresh differential `backup-nfs-to-local` generation of the live export), then mirrors the snapshot into the target (`rsync -aHAX --delete`, with the shared `<state>/backup` root protected from deletion and never copied in). `--no-safety-backup` skips the unit run when the target holds nothing worth saving.
 3. Redeploy the stack; the NFS clients re-mount and pick up the restored state.
 
 The target subtree must already exist; the script refuses to create export subtrees implicitly.

roles/svc-bkp-nfs-2-local/files/python/recover.py

@@ -6,7 +6,7 @@
 
 Runs the role's deployed backup unit first (the usual differential
 ``backup-nfs-to-local`` generation of the live export), then mirrors the
-snapshot into the target (``rsync -a --delete``). Run on the host
+snapshot into the target (``rsync -aHAX --delete``). Run on the host
 serving the export with every consuming stack stopped; NFS clients
 re-mount on redeploy.
 

roles/svc-bkp-nfs-2-local/files/shell/script.sh

@@ -45,7 +45,7 @@ trap cleanup_failed_generation ERR
 
 mkdir -p "${DEST_DIR}"
 
-rsync_args=(-a --numeric-ids --delete)
+rsync_args=(-aHAX --numeric-ids --delete)
 if [[ -n "${EXCLUDE_REL}" ]]; then
     rsync_args+=(--exclude "/${EXCLUDE_REL#/}")
 fi

roles/svc-bkp-remote-2-local/files/python/pull_specific_host.py

@@ -120,7 +120,7 @@ def pull_backups(hostname: str, backups_dir: str) -> None:
         Path(local_backup_destination_path).mkdir(parents=True, exist_ok=True)
 
         rsync_command = (
-            f"rsync -a --partial --info=stats2 "
+            f"rsync -aHAX --partial --info=stats2 "
             f"--numeric-ids --delete --delete-excluded --timeout=300 "
             f'-e "ssh {SSH_OPTS}" '
             f'--rsync-path="sudo rsync" '

roles/svc-bkp-volume-2-local/README.md

@@ -78,7 +78,7 @@ flowchart TD
     TREE --> RECOVER["files/python/recover.py volume<br>resolves Mountpoint + Options"]
     RECOVER -->|"Options set"| PROBE{"mountpoint -q<br>(ssh for --docker-host)"}
     PROBE -->|unmounted| REFUSE["SystemExit: a restore would land<br>on the node disk and be shadowed<br>on the next mount"]
-    PROBE -->|mounted| RSYNC["rsync -a --delete through the<br>mounted export onto the server"]
+    PROBE -->|mounted| RSYNC["rsync -aHAX --delete through the<br>mounted export onto the server"]
     DRILL["DR drill [9/9]<br>mounts the export at the volume<br>mountpoint with the volume's own opts"] --> RECOVER
     RSYNC --> VERIFY["marker asserted on the<br>NFS server's disk"]
 ```
@@ -304,7 +304,7 @@ recover.py <backups>/<machine-hash>/backup-docker-to-local/<generation>/<volume>
 ```
 
 1. Stop the consuming project (`docker compose down` / `docker stack rm <stack>`).
-2. Run the script; it first starts the role's deployed backup unit (a fresh differential baudolo generation of every volume and database), resolves the volume's mountpoint and mirrors the snapshot into it (`rsync -a --delete`). `--no-safety-backup` skips the unit run when the target holds nothing worth saving.
+2. Run the script; it first starts the role's deployed backup unit (a fresh differential baudolo generation of every volume and database), resolves the volume's mountpoint and mirrors the snapshot into it (`rsync -aHAX --delete`). `--no-safety-backup` skips the unit run when the target holds nothing worth saving.
 3. Restore the databases with the `--databases` mode above **while the project is still down**, then start it again; on swarm, NFS-backed volumes are restored via `svc-bkp-nfs-2-local`'s `recover.py` instead (see below).
 
 ## Credits

roles/svc-bkp-volume-2-local/files/python/recover.py

@@ -9,7 +9,7 @@ Both modes run the role's deployed backup unit first, unless
 ``--no-safety-backup`` says the target holds nothing worth saving.
 
 Volume mode mirrors the snapshot into the volume's mountpoint
-(``rsync -a --delete``); stop the consuming project first. With
+(``rsync -aHAX --delete``); stop the consuming project first. With
 ``--docker-host ssh://user@host`` it works on a remote host.
 
 Database mode (``--databases``) replays every dump with

tests/unit/python/cli/administration/recover/test_full.py

@@ -74,7 +74,7 @@ class PullCmdTest(unittest.TestCase):
             full._pull_cmd("user@srchost"),
             [
                 "rsync",
-                "-a",
+                "-aHAX",
                 f"user@srchost:{paths.BACKUP_ROOT}/",
                 f"{full.PULL_STAGE}/",
             ],
@@ -83,7 +83,7 @@ class PullCmdTest(unittest.TestCase):
     def test_host_with_explicit_path(self):
         self.assertEqual(
             full._pull_cmd("host:/custom/root"),
-            ["rsync", "-a", "host:/custom/root/", f"{full.PULL_STAGE}/"],
+            ["rsync", "-aHAX", "host:/custom/root/", f"{full.PULL_STAGE}/"],
         )
 
 

tests/unit/python/roles/svc-bkp-nfs-2-local/files/test_script.py

@@ -74,6 +74,14 @@ class NfsBackupScriptTests(unittest.TestCase):
         ).read_text()  # nocheck: cache-read tempdir fixture
         self.assertEqual(content, "hello")
 
+    def test_snapshot_keeps_user_xattrs(self):
+        (self.source / "a.txt").write_text("hello")
+        os.setxattr(self.source / "a.txt", "user.oc.id", b"space-1")
+        result = self._run()
+        self.assertEqual(result.returncode, 0, result.stderr)
+        snapshot = self._generations()[0] / "files" / "a.txt"
+        self.assertEqual(os.getxattr(snapshot, "user.oc.id"), b"space-1")
+
     def test_second_run_hardlinks_unchanged_files(self):
         (self.source / "a.txt").write_text("hello")
         self.assertEqual(self._run(generation="20240101000000").returncode, 0)

tests/unit/python/utils/recovery/test_base.py

@@ -1,5 +1,6 @@
 from __future__ import annotations
 
+import os
 import tempfile
 import unittest
 import unittest.mock as mock
@@ -55,6 +56,15 @@ class TestDirectoryRecovery(unittest.TestCase):
             self.assertEqual(text, "from-snapshot")
             self.assertFalse((target / "live.txt").exists())
 
+    def test_restore_keeps_user_xattrs(self) -> None:
+        with tempfile.TemporaryDirectory() as td:
+            source, target = self._mk(td)
+            os.setxattr(source / "restored.txt", "user.oc.id", b"space-1")
+            _Recovery(str(source), str(target)).restore()
+            self.assertEqual(
+                os.getxattr(target / "restored.txt", "user.oc.id"), b"space-1"
+            )
+
     def test_missing_unit_refuses(self) -> None:
         with tempfile.TemporaryDirectory() as td:
             source, target = self._mk(td)

utils/recovery/base.py

@@ -1,7 +1,7 @@
 """Abstract recover flow shared by every ``svc-bkp-*`` role.
 
 A recover mirrors a snapshot into a live target directory
-(``rsync -a --delete``), so the target's current state would be lost
+(``rsync -aHAX --delete``), so the target's current state would be lost
 without a net. The flow therefore first runs the role's deployed backup
 service (``systemctl start`` on the role's unit), which applies the
 usual differential backup logic to the live data before anything is
@@ -87,7 +87,7 @@ class DirectoryRecovery(ABC):
         subprocess.run(
             [
                 "rsync",
-                "-a",
+                "-aHAX",
                 "--numeric-ids",
                 "--delete",
                 *self.rsync_extra_args,