baudolo 7.1.x writes volume generations with `rsync -aHAX`, so the
producing host keeps `user.*` extended attributes (OpenCloud keeps its
storage metadata in `user.oc.*`). Every later copy dropped them again,
because none of the other rsync calls passed `-X` (nor `-A`/`-H`):
- NFS export snapshots (svc-bkp-nfs-2-local `script.sh`),
- the backup host's pull (svc-bkp-remote-2-local `pull_specific_host.py`),
- the device sync (svc-bkp-local-2-device `script.py`),
- every recover into a live target (`utils/recovery/base.py`, used by the
volume, NFS and device recover scripts),
- the `recover full` and `recover remote` tree pulls.
A disaster recovery through the documented chain therefore restored
OpenCloud without `user.oc.*`, silently: the copies succeeded and the
dirval stamp does not cover attributes.
All of these now run `rsync -aHAX` (`-aHAXP` for the device sync). A
target filesystem without xattr support makes rsync exit non-zero as soon
as a source file carries attributes, so the hop fails loudly instead of
dropping them; there is no per-device opt-out. The single-image pull in
`recover remote` keeps `-a`. The provider's ssh-wrapper needs no change:
its glob accepts any server flags and its character guard allows them.
Two tests write a `user.oc.id` attribute and assert it survives the NFS
snapshot and the recover; with the old `-a` both fail with
`OSError: [Errno 61] No data available`.
Closes#1021
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Production block never set one. inventory provision writes only
ansible_connection for a local host, so DOMAIN_PRIMARY stayed at the
group_vars default of infinito.localhost and the deployment served a zone its
own resolver knows nothing about. CoreDNS templates answers for
INFINITO_DOMAIN_RE and nothing else, so every lookup of the deployment's own
name came back NXDOMAIN.
Mastodon is where that surfaced. tootctl accounts create validates the address
through EmailMxValidator, which resolves MX and then, for the domain itself,
A and AAAA. All six queries returned NXDOMAIN in run 33574056480, the
validator recorded email :unreachable, and the admin account was never created.
Only the guide rows were affected: the matrix rows write
DOMAIN_PRIMARY: "{{ lookup('env', 'INFINITO_DOMAIN') }}" into their host_vars
and resolve fine.
The template now carries DOMAIN=<your-domain> in both the host and the
container form and threads it into --vars, with the container form passing it
through the docker run boundary as well. The two guide runners substitute it
the way they already substitute the SSH key, so the replay deploys under
INFINITO_DOMAIN like every other CI deploy. compose_deploy.sh moves its env
load ahead of the substitutions to have the value; host_deploy.sh had no env
load at all.
A production reader is asked for a domain now instead of silently inheriting
infinito.localhost, which no real deployment wants either. The shipped default
keeps its value: sys-svc-nss documents that *.localhost resolves through the
myhostname NSS module without any zone, and host-side tasks lean on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
4.0.0 renamed --dump-only-sql and rejects the old spelling with exit 2, so the timer's assembled exec string would have failed on every run against the version the previous commit pins. The launcher fixtures and the role README carry the same rename; nothing else in the repo still says it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The drill removed each volume and re-created it. Docker refuses to remove a volume any container still references, and a container outside compose is only stopped here, not removed, so discourse's launcher container aborted the run at that point: after the host was already down and part of the volumes were gone. Re-creating a removed volume also drops its driver options, handing an NFS- or bind-backed volume back as a plain local one.
Volumes are emptied in place instead, from inside a throwaway container, and the emptiness is proven in the same call. Nothing has to be re-created, so a launcher container starts again exactly as it was built.
The engine that receives a replay is started by the service name on its own compose label. The previous fixed name existed for no engine on the host: the central Postgres serves project postgres from volume postgres_data, bigbluebutton serves project bigbluebutton from service postgres. Neither is called database, so the drill died at 'up -d database' after the teardown and before any restore.
A cluster dump yields one marker per database it recreates, or a replay that brought back only part of an instance would still pass. The drill runs once, in the sync pass; repeating it in the async pass would only tear the host down twice for the same answer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cosmos generator linked a consumer to its provider only through an
'<role>' in group_names reference or an explicit provides: name. A hard
dependency has neither -- pinning the provider on unconditionally is
exactly what removes the Jinja reference the generator was reading -- so
web-svc-asset's bond to web-svc-file vanished from the diagram the moment
it became binding. Match a service key against the provider's own primary
key from the registry, in both directions.
Services the role turns off drew a solid 1:1 edge, claiming a mandatory
relationship for something that carries nothing at deploy time. They now
render as a red 0..0 edge; edge merging gained a real precedence so a
provider consumed through two keys stays live when only one is off.
Named colour, not hex: a '#' opens an entity code in the mermaid lexer and
turns the whole diagram into a parse error on GitHub.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A backup generation stores databases as sql dumps, not as a file tree
(--dump-only-sql), and the recover chain only ever restored trees:
`git grep sql HEAD -- cli/administration/recover utils/recovery` was
empty. An operator running `recover full` after a real loss got their
files back, their databases untouched, and an exit code of 0.
Adds the missing leg as a fifth recoverer, planned right after the
volume steps so the dumps land once the volumes are back. Three
properties make it a restore rather than an attempt:
- It refuses while a consumer of that database runs. `--empty`
pre-cleans the schema in one psql session and replays in the next; a
booting consumer recreates it in between and the dump's own CREATE
TABLE fails. That is what turned CI red with `relation
"ar_internal_metadata" already exists`.
- The engine is read, not guessed: the service key in the consuming
role's meta/services.yml, the same value that fed the backup seed.
The two previous call sites each sniffed it from the container image
and could disagree.
- The dump's own header states which server it came out of, and that is
compared against the running engine before the pre-clean drops
anything. Forward across a major version is the upgrade path and
stays allowed; backward is refused. The version deliberately does not
travel in databases.csv: that file is written at seed time and would
state the version deployed back then, and a generation copied to
another host carries its header along while a csv row stays behind.
Mind the header formats, they are not symmetric - postgres puts
`Dumped from database version` on line 7, while mariadb's line 2 opens
with mariadb-dump's own version and only the tab-separated `Server
version` line names the server. Both are pinned as test fixtures,
captured from postgres:17-alpine and mariadb:11 themselves.
baudolo is untouched: nothing beyond the credentials it already carries
is read from databases.csv, and no upstream format changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role README claimed the backup machinery "operates transparently against the mount
paths" and demoted the source-of-truth rule to advice in a prose box - both written
before 2efc73d3b made that rule the code's behaviour. It now states what is enforced:
this role skips export-backed volumes, svc-bkp-nfs-2-local is their single capture, and
manager-pinned roles, roles forcing another compose_mode_force and nfs: false opt-outs
stay covered here. It also names the limit the code actually has - the exclusion checks
that some host is in the svc-bkp-nfs-2-local inventory group, not that the role ever ran
successfully - because a README promising more than the code delivers is worse than one
promising less.
Its Schema section carried the role's timer flow; it now carries the mechanism the
change set is about: meta/volumes.yml -> swarm_nfs_backed() as the single predicate ->
its two consumers (the compose rewrite and the backup exclusion) -> the inventory-group
gate -> the export host's single capture -> the restore path with its mountpoint probe
-> the drill's [3b] guard against over-exclusion.
The two swarm READMEs drew the drill with eight steps and pre-773a25ab6 semantics. Both
now draw the nine that exist, including what makes the proof a proof: the pulled tree is
wiped before the device recover, so the device holds the only copy left; the live export
is wiped with the marker proven gone before it is restored; and [9] deletes the marker
server-side before restoring the volume through its bound backing store.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The launcher declined on every btrfs host for a reason no operator could
guess: `/var/lib/docker` is created by mkdir, so it is an ordinary
directory inside the enclosing subvolume, and `btrfs subvolume snapshot`
needs a subvolume root as its source. Documenting that as a manual
prerequisite would have left the feature switched on and never firing.
ensure_subject_subvolume.sh converts the data root on the deploy that
introduces the mode. The copy uses `cp --reflink=always`: both trees live
on the same btrfs, so extents are shared instead of duplicated and the
conversion costs seconds and no extra space no matter how large the data
root is - where an rsync copy would have meant hours of downtime and a
free-space requirement of 110%. The original tree is kept until the daemon
has answered on the converted root, so a failed copy, a mismatched
verification pass or a daemon that will not start all put it back and
start docker on it again.
The mode itself moves from group_vars into meta/services.yml as
`snapshot_mode`, read through `lookup('config', ...)`, so a single host can
override it from the inventory. Falsey spellings normalise to `never` and
anything else fails a deploy-time assert, because YAML reads a bare
off/no/false as a boolean and `--mode False` would kill the unit nightly.
btrfs-progs is installed wherever a btrfs mount exists, with centos opted
out: centos stream ships neither the kernel module nor the package, and
demanding it there would fail the package-availability test for the whole
RedHat family.
The pin moves to 3.2.2, which is not cosmetic: that release carves the
snapshot inside its subject, and the reaper here looks for leftovers in
exactly that place. Against 3.2.1 it would search beside the data root and
never find one.
Two drills run before anything else in test.sh: subvolume_probe.sh proves
the conversion carries every path, payload and symlink across and that the
result can actually be snapshotted; snapshot_probe.sh pins the launcher's
decision, including the refusals that keep a host on the live copy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
baudolo 3.2.1 can capture every volume from one atomic filesystem
snapshot, which removes both the container stops and the torn copy a
live rsync of a database produces. It states the snapshot kind rather
than probing it, so naming a kind the host cannot deliver costs a whole
backup run.
Deciding that at deploy time is not possible: the set of docker volumes
changes between deploys, and a volume created afterwards would sit in the
snapshot as an empty directory, so the copy would succeed while holding
nothing. The decision therefore moves into the run itself. ExecStart now
points at baudolo-snapshot, which probes the host, appends
--snapshot/--snapshot-subject when a snapshot will be faithful, and
becomes the baudolo process either way.
STORAGE_BACKUP_SNAPSHOT_MODE ships as `never`, so no host changes
behaviour until an operator opts in - baudolo's snapshot path has never
been driven end to end on a real deployment. `auto` probes and degrades
to today's live two-pass copy with the reason in the journal; a stated
`btrfs`/`zfs` skips only the fstype lookup and fails the unit when the
host cannot deliver, which is the point of stating it. The token is
`never` rather than `off` because YAML 1.1 reads a bare off/no/false as a
boolean, which would render `--mode False` and kill the unit on every
timer fire; falsey spellings are normalised and a deploy-time assert
rejects anything the launcher's choices would not accept.
The probe refuses unless the data root is a btrfs subvolume root whose
parent is writable and on the same filesystem with no nested subvolume
below it, and unless every docker volume uses the local driver, declares
no Options of its own, sits under the data root and has nothing mounted
inside. The volume rules read the declaration rather than the mount
table, because docker mounts an NFS-backed volume lazily and unmounts it
when the last container stops - such a volume is a separate filesystem
that a snapshot of the data root captures as an empty directory. `auto`
accepts btrfs only; zfs needs to be stated, since none of its premises
could be exercised here.
Before taking a snapshot the launcher removes the `.baudolo-*` leftovers
a killed run leaves beside the data root: baudolo removes its snapshot in
a finally, which a SIGKILL or a reboot skips, and a read-only snapshot of
the data root pins every block the live root later frees.
--hard-restart-projects is stripped when the snapshot flags are appended:
it exists for stacks whose database cannot be backed up hot, which is
what a snapshot removes.
The pin moves 3.1.3 -> 3.2.1. That version is not on PyPI yet; publish it
before deploying, or the pip task fails on every host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
House style bans em and en dashes. Established role READMEs carry none,
but the template had one in its Quick Setup line, so it had been added
after those roles were generated and reappeared in every regenerated
README since. Replacing it with a colon propagates to the 54 roles the
generator refreshed.
The Credits condition switches from an "is defined" test to a truthiness
test because the generator now always sets application_author_url,
passing None for an unknown author. Left as it was, the branch would have
rendered a broken [Name](None) link instead of falling back to the bare
name.
roles/categories.yml loses the same dashes from the svc.virt description.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add tests/lint/repository/test_task_file_numbering.py: numbered task
files use exactly two leading digits, a numbered *_core.yml must be
00_core.yml, per directory the numbers are unique/consecutive and start
at 01 (00 with a core file), numbered includes appear in ascending
order, and shared helpers live in tasks/utils/ instead of the numbered
chain.
Apply it across every role: rename NN_core.yml to 00_core.yml with a
consecutive down-shift of the siblings, move letter-suffix children into
a subdirectory named after their parent task, relocate helpers included
from more than one place into tasks/utils/, and update every reference
(include_tasks, path_absolute, .ansible-lint-ignore, requirement docs).
Legacy narration comments in the touched files are deleted; the upstream
xwiki.properties reference block opts out via nocheck. The unused
MAILU_IP6_PUBLIC var (only ever named in now-removed comments) is dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Standardize the private inventory dir on the fully spelled "production" (matching inventories/development). Update the role README template (SPOT) and regenerate 153 role READMEs plus the admin deploy guide; add inventories/production/ to .gitignore, propagated to .dockerignore via make fix-dockerignore.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The guide deploys to a .localhost target, where letsencrypt cannot issue (Cloudflare DNS-01 needs a real domain). Default TLS_MODE=self_signed in the generated Production block so the guide uses the working self-signed provider. Document every deploy parameter as a top-level shell var (APP, HOST, TLS_MODE, SSH_PUBLIC_KEY) instead of an inline placeholder.
The compose guide now extracts and runs the whole README block, sourcing params from it (no more -e injection); HOST is forced to localhost. Skip checkout/login under ACT so the guide workflow is iterable locally.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The README Production block went provision -> deploy with no admin-key step, so user-administrator failed early (authorized_keys empty) for both the Guide CI jobs and any real user copying the README. Add a visible 'provision --vars' step that sets users.administrator.authorized_keys (merged into host_vars) in both Production variants of the role README template, regenerate all role READMEs, and replace the invisible CI payload splice with a visible substitution of the documented <your-ssh-public-key> placeholder.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A host role installs onto the machine itself, so its README Production block now shows the direct local-connection procedure (clone, scripts/install/package.sh, make install, source the env, then provision + deploy against localhost) instead of the container docker-run aimed at a remote server; stack roles keep the docker-run block. The mode is derived from role_has_stack. make docs now runs readme-generate quick_setup=true so the block stays in sync, and every host role README is regenerated accordingly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The role README Production section now runs provision and deploy in a
single docker call: APP and HOST are shell variables passed via -e,
INVENTORY is the in-container SPOT for the duplicated paths, and the
redundant --vars-file (optional, default None) and --id (the inventory
is already narrowed by --include) flags are gone. All 153 role READMEs
regenerated from the template; test-run.yml executes exactly this block.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rewrite all 254 role descriptions to a uniform noun-phrase schema
(function-first, no leading verb, no trailing period, <=120 chars) and
enforce it via the extended description lint. Rename 12 role README H1
titles for clarity (e.g. Backup Container to Local -> Backup Docker
Volumes, CSS Service -> Corporate Design, NGINX Legal -> Imprint) and
regenerate their Cosmos and Quick Setup sections. Overview table
regenerated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add cli.build.readme.overview (make readme-index / make docs) rendering a
roles overview in the root README: Name, Lifecycle, Homepage, Video,
Description, Integrated with, Install. New info-media lint requires
homepage + video in every invokable role's meta/info.yml (opt out via
'# nocheck: info-media' in meta/main.yml); backfilled 147 roles with
researched, adversarially-verified homepages and official videos.
Retire the 'unsupported' lifecycle value: fold it into 'eol' as a valid
live end-of-life status (shipped but untested/unmaintained) across the
stage lists, symbol glossary, allow-list lint and docs. Retag confluence,
jira, minio, phpldapadmin to eol and magento to pre-alpha.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
make cosmos regenerated the backup-role READMEs against the current meta state: cardinality labels, the symbol legend and gear-marked Ansible dependencies from the newer generator, without the parked full-backup orchestrator edges.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add cli/build/readme: derives each role README's managed sections from
templates/roles/README.md.j2.tmpl (the Cosmos mermaid diagram from
meta/services.yml, an invokable-gated Quick Setup, and Credits).
The Cosmos diagram shows dependencies, every declared service, and
dependents (roles that consume this one, matched by role name and by
provides provision-name in both directions). Role nodes carry deploy-mode
emoji markers (compose/swarm/host); disabled and shared services get a
stop mark; dependents are capped at 12.
test_readme enforces the Quick Setup body is byte-exact with the template
render. `make readme-generate` and `make cosmos` regenerate; `make cosmos`
also runs inside `make quality`.
Refactor: utils/symbol_glossary.py is the single source of truth for the
word/emoji mapping (unique project-wide), consumed by the complexity
matrix and the Cosmos builder. utils/roles/deploy.py holds role_has_stack
and role_deploy_modes, shared by both so the logic never drifts.
Add `make lint-mermaid` and its CI workflow, rendering every diagram via
mermaid-cli; the render step provisions chrome-headless-shell and unzip
on demand so it works in the minimal container.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add cli.administration.recover: one uniform type/source/target command wrapping each svc-bkp-* role files/recover.py (nfs, volume, secrets, device) plus full for whole-tree recovery. Colon-encoded sources are classified by shape; device supports an on-device subpath, a generation, and remote pull over ssh.
Replace the per-host swarm DR-drill scripts (03/06/07/08_recover_*.sh) with the recover CLI in backup/base.sh. Pass the device on-device subpath relative (the CLI reads absolute segments as the restore-root) and load the applications-free .deploy.yml extras twin on the step-10 redeploy so host_vars credentials survive the -e override.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the constructor's pre_deploy_snapshot.sh loop with the loader
itself: volume/nfs/secrets instant-start their oneshot in the preload
pass under MODE_BACKUP (force_flush_instant + state started), so the
live pre-update state is snapshotted before any app mutation; nfs keeps
run_after svc-storage-nfs-server, volume pulls sys-svc-container for a
live dockerd and pre-creates /opt/compose. remote pulls at end of play
via the terminator (force_flush_final) once every provider snapshot is
done. Dropping suppress_flush arms the result assert: a failed
pre-state backup aborts the deploy. Role READMEs carry the new trigger
schema as mermaid diagrams.
Known residual: on a host whose pull identity is not yet provisioned
the terminator-forced remote pull fails and turns the deploy red; the
swarm lab installs the trust only in the DR drill after round 1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add utils/recovery/base.py DirectoryRecovery: a service-backed pre-recover
(start the role's backup unit first) + rsync -a --delete mirror. Each
svc-bkp-* role gets a files/recover.py subclass (volume resolves the docker
mountpoint, nfs mirrors the export, local-2-device opens the newest LUKS
snapshot across machine hashes) or a documented one-way opt-out
(remote-2-local recover.py.nocheck). A lint guard enforces every svc-bkp-*
role ships a recover.py and a ## Recover README section; docs/architecture/
backup.md documents the end-to-end schema. Unit tests cover the base flow,
the per-role class logic and every recover.py main() CLI wiring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The container and NFS pre-deploy snapshots ran two near-identical role
scripts and two constructor tasks. Merge them into one shared script under
scripts/system/backup/ whose content check handles a file (databases.csv)
or a directory (export base), and drive both from a single looped
constructor task keyed by role, unit, and source. Each iteration keeps its
own group_names gate, its own timeout, and reports changed only when it did
not skip.
The shared script also reaches the fresh-host skip loudly: systemctl
list-unit-files exits non-zero when the version glob matches nothing, so the
listing now sits in an if instead of a bare command substitution that
aborted the snapshot under set -euo pipefail (the compose test-development
failure). The volume e2e test starts the unit through an if and dumps the
unit journal on failure so the real start error surfaces.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename the role, its test tree, the primary entity key
(volume-2-local), the derived vars (BKP_VOLUME_2_LOCAL_*,
SYS_SERVICE_BACKUP_VOLUME_2_LOCAL), the run-once fact, and every
group_names consumer flag, include path, doc, and lint reference. The
container_backup service key and the on-disk backup-docker-to-local
tree keep their names: both are contracts with existing consumer
entries, snapshot trees, and the baudolo pull tooling.
Operators must rename the svc-bkp-container-2-local inventory group and
disable the previous versioned units on already-provisioned hosts; the
pre-deploy snapshot glob derives its base from the new role name and
will not find them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
