The drill (backup/base.sh) no longer runs its own step-10 redeploy; it recovers device -> nfs export -> volume -> secrets and writes a sentinel. The matrix update pass (which runs next anyway) boots the recovered stack, and verify_recovered_marker.sh asserts the live marker after it. Saves one full stack redeploy per drilled app; the marker-gone check before recovery stays in 05_wipe_export.sh.
Also derive recover_drill.sh REPO from the script location instead of an INFINITO_REPO_ROOT shell-side default (lint: no INFINITO_* / absolute-path defaults in .sh).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Node image now builds FROM the distro infinito image (python, ansible, CLI baked in) instead of installing a per-node .deb, so there is no per-node bootstrap or timeout; the swarm CI job depends on the distribution build. Move compose/swarm.yml to compose/swarm/compose.yml and relocate the swarm matrix orchestrator from cli.administration.deploy.swarm to utils.tests.swarm (test code out of the shipped CLI).
Back the NFS export root with a disk-backed named volume instead of a RUNNER_TEMP bind: local-act RUNNER_TEMP resolves onto tmpfs and ganesha FSAL_VFS cannot export a tmpfs, so it loaded zero exports and the controller mount got ENOENT. Add optional compose/swarm/cache.override.yml wiring swarm nodes to the package-cache frontend when present, leaving CI without a cache unaffected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add scripts/tests/deploy/ci/recover_drill.sh: seed a backup root (nfs, volume, secrets, each with a marker) plus a docker volume, then run recover full <root> localhost to restore every backup-stored type into its live target and verify the markers. Runs between PASS 1 and PASS 2 of the compose development deploy.
Hook it via _maybe_recover_drill in cli.administration.deploy.development, gated by INFINITO_RECOVER_DRILL (default true).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add cli.administration.recover: one uniform type/source/target command wrapping each svc-bkp-* role files/recover.py (nfs, volume, secrets, device) plus full for whole-tree recovery. Colon-encoded sources are classified by shape; device supports an on-device subpath, a generation, and remote pull over ssh.
Replace the per-host swarm DR-drill scripts (03/06/07/08_recover_*.sh) with the recover CLI in backup/base.sh. Pass the device on-device subpath relative (the CLI reads absolute segments as the restore-root) and load the applications-free .deploy.yml extras twin on the step-10 redeploy so host_vars credentials survive the -e override.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds make swarm-diagnostic (alias i8sadi): read-only probes across a
live swarm-test cluster's nodes via exec_node.sh - backup unit state +
recent journal, NFS mounts, D-state (wedged-NFS) processes, running
rsync/db-dump processes, and disk usage - so a hung backup unit can be
root-caused without hand-writing docker exec pipelines.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a routine/README.md describing what each phase of the swarm lab
run does (bring-up, the variant-matrix round loop, the DR drill's nine
steps, the chaos phase, teardown), with the moments a backup-executing
systemd unit actually starts highlighted, and links to the per-step
scripts below the diagram.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The DR drill's simulated USB was a fixed 512M loop image (~449M usable
after LUKS2 + ext4), but the deployed local-2-device unit syncs the
whole pulled backup root, which after the remote pull holds two
machine-id trees (the manager's docker+secrets repos and the NFS
export repo). For an app the size of akaunting that overflows the
image mid-rsync and the drill fails with ENOSPC at step [5/9]. base.sh
now measures the pulled tree (du -sm) and passes 2x its size + headroom
(floor 2G) to 02_luks_device.sh, which sizes the loop image from that
argument; fallocate keeps the image sparse so the larger floor is free.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A wedged manager (act-swarm containerd stall) made the docker exec tar
block up to 900s; on a cancelled job that ate the runner's grace window
and the always() Cleanup step never ran, leaking the DiD cluster. Probe
the manager first (10s) and cap the copy at 120s with --kill-after so a
hung exec is force-reaped; teardown then always follows. Reports stay
collected before teardown, since they live inside the manager node.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Promote the Firefox onion fixture (secure-context .onion prefs, SOCKS
remote-DNS) from web-app-dashboard to test-e2e-playwright as the single
source of truth; the dashboard copy now re-exports it. Stage the shared
fixture on all three paths (deploy runner, lint-playwright, rerun-spec).
gotoOnion: onion navigations default to waitUntil=domcontentloaded —
heavy chunked SPAs (Element) serialise 30+ bundle requests over Tor, so
the load event regularly exceeds the navigation cap; callers' explicit
onion-scaled selector waits cover app boot instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The provisioner copied the manager host_vars only to the workers, so the
service-loader preload on the nfs and backup nodes merged a defaults-only
applications dict (credentials lookups failed), and the bootstrap play
seeded /etc/hosts only on manager and nfs, so the backup node's msmtp
healthcheck died with EX_NOHOST. Both nodes now get the host_vars copy
and the app-domain hosts entries.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
swarm_stack_ready.sh piped 'docker stack services' through a process
substitution, so the timeout wrapper added in 8516e3167 turned a slow or
wedged manager into an empty listing, an empty not_running set and exit
0: the convergence gate would have gone green with zero running replicas.
Capture the listing, fail on a non-zero rc.
That exposed the second half: post_deploy gated the swarm wait on the
cluster-wide DEPLOYMENT_MODE, so forced-compose roles (matrix ansible
flavor, bigbluebutton) ran it against a stack that does not exist and
only passed because the old code swallowed the error. Gate it on
compose_mode, like the handlers already do.
teardown.sh: the NFS mountpoint and umount calls, the likeliest hang
sites, were the only ones left unwrapped, and the timeout around docker
inspect inverted the D-state warning (a hung inspect returned 124, the
if went false and the script reported a clean teardown). Warn on rc 0
and 124, and shrink the per-node budget so the script fits inside its
step timeout.
Swarm step budget: the bounded steps alone summed to 365 over a 360-min
cap, so cleanup was still starved. Deploy 240, diagnostics 15, playwright
15, rescue 25, cleanup 15.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
27 jobs of run 29147179243 burned their full 360-min budget. Three
systemic layers, all addressed here:
- The CLI e2e task ran the staged test.sh unbounded, and test.sh blocks
in 'systemctl start' on a oneshot backup unit with no JobTimeout; on a
wedged dockerd or a stalled NFS mount both freeze silently for hours.
The task now runs the script under 'timeout TEST_E2E_CLI_TIMEOUT'
(60 min via the timeout lookup, so TIMEOUT_FACTOR scales it) and the
unit starts are bounded by BKP_TEST_HEALTH_TIMEOUT.
- The converge probe re-ran the full per-service diagnostics (service ps,
service logs, inspect, journalctl) on EVERY retry, so 600 nominal
2-second attempts cost 25-33s each and silently outlived the step
timeout. The block is gone (the workflow-level container.py rescue
snapshot already collects it recursively on failure) and every docker
call is timeout-bounded. Same removal in the swarm collect script,
which keeps only the NFS and render evidence container.py does not see.
- Playwright-collect, rescue diagnostics and Cleanup had no
timeout-minutes, and teardown.sh calls docker against D-state nodes,
so a failed job hung there for hours until the job cap killed it
without artifacts. Every step is bounded now, every teardown call is
timeout-wrapped, and the deploy steps (swarm 270, compose 300) leave
room for diagnostics and cleanup below the 360-min cap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
routine/08_cleanup.sh becomes utils/clean/teardown.sh and topology/
becomes utils/topology/ so routine/ holds only the sequenced flow; all
consumers (workflow steps, compose/swarm.yml env guards, relative
sources, README) follow. The shell rescue collector and its lint are
gone: utils/diagnostics/container.py covers compose and swarm
identically. extend_inventory gains the backup-host placements the DR
drill deploys.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
extend_inventory maps svc-bkp-remote-2-local + svc-bkp-local-2-device onto
the backup host and write_extras configures them (backup_providers =
manager + NFS server IPs; device mount/target/source), so the matrix
deploy installs the real systemd units there. The drill now exercises
them: 01_ssh_trust.sh only installs the pull identity, the remote-2-local
unit pulls every provider (replacing both manual pull calls), and after
02_luks_device.sh simulates the USB plug the local-2-device unit performs
the device sync; the device recover passes the configured target subpath
via --device-target. Unit absence on the backup host is now a hard
failure since the roles deploy with the round.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add routine/backup/: base.sh drives a marker seeded into the live NFS
volume and the manager secrets through the whole chain (volume+secrets+nfs
backup units -> remote-2-local pull -> local-2-device LUKS 'USB') and back
via every recover.py (device -> local root -> nfs export, docker volume and
host secrets), asserting each marker survives. Each per-host step is one
in-node routine run over docker exec. extend_inventory maps the volume/
secrets/nfs backup roles onto their hosts so the units install; write_extras
generates the pull keypair; the INFINITO_SWARM_BACKUP_* keys are added to
default.env + the passthrough handler.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the per-node docker-run bring-up with a single node image
(compose/swarm/Dockerfile bakes python3/dnsmasq/openssh/rsync/cryptsetup)
declared in compose/swarm.yml, one compose up, and one ansible play
(compose/swarm/playbook.yml over ansible_connection: docker) that waits for
systemd, captures IPs, wires lab DNS and installs the .deb per node. Reshape
the harness into topology/ (naming SPOT), routine/ (01_bootstrap + the
sequenced 02-08 steps) and utils/ (clean/, collect/, shared helpers); the
workflow, matrix.py and Makefile follow the new paths. swarm init/join was
already ansible-owned (svc-swarm-node), corrected in the README diagram.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A default like ${BACKUP_KEY_PATH:-/tmp/swarm-nfs-backup.key} duplicates
a path contract that belongs in a SPOT (default.env or a role vars
file) and drifts silently. The guard covers every ALL-CAPS variable
whose inline default starts with a slash, allows per-line nocheck
suppression, and its failure message forbids dodging the syntax via
conditional assignments. Legacy offenders carry explicit nocheck
reasons (act tooling runs before default.env exists, POSIX TMPDIR
convention).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New handler reads TEST_E2E_PLAYWRIGHT_STAGE_BASE_DIR from
roles/test-e2e-playwright/vars/main.yml (via ROLE_FILE_VARS_MAIN, plain
read_text line parse because env handlers bootstrap fresh hosts before
PyYAML exists) and publishes it as INFINITO_PLAYWRIGHT_STAGE_BASE_DIR.
rerun-spec.sh consumes the handler value and falls back to parsing the
same SPOT on DiD nodes without a generated .env, dropping its /tmp path
copy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every Ansible rescue: block used to carry ad-hoc per-role failure dumps that
drifted apart and captured different subsets. Replace them all with one
diagnostics script (scripts/system/diagnostics/rescue.sh) that snapshots
containers (full logs plus inspect per container), swarm services (ps plus
logs per service), journal, host resources, runtime info and stats into a
per-run folder and always exits 1 so the deploy still fails. A new lint
enforces the shape: every rescue ends with the script, at most one
nocheck-justified extra task before it (postgres pg_hba dump, xwiki
extension-repo probe), and a nocheck rescue-recovery exemption for the one
block that must genuinely recover: keycloak object updates, where a
cross-node reschedule needs a swarm-wide node re-resolve plus a delegate_to
change that only a rescue can express; its recovery is kept and now also
captures diagnostics before the terminal failure.
The snapshot folder path is a group_vars SPOT: DIR_RESCUE_DIAGNOSTICS and
DIR_VAR_LIB live in group_vars/all/05_paths.yml as plain strings; new env
handlers derive INFINITO_RESCUE_DIAGNOSTICS_DIR and INFINITO_DIR_VAR_LIB
from it for the generated .env, replacing the default.env entries, and the
play-level environment forwards the value to every task host so rescue.sh
needs no second default. CI uploads the folder like the inventory and
playwright artifacts: compose copies it out of the deploy container in
dedicated.sh, swarm captures per node (manager and both workers) into
node-named subfolders.
Deleting the ad-hoc dumps orphaned helper files (postgres inspect_container,
terminator docker_diagnostics, peertube diagnostics, matomo bootstrap debug,
xwiki and openresty rescue includes) and 14 role vars; all removed. Generic
pieces of those dumps (runtime version and info, stats, dmesg oom evidence,
swapon) moved into the SPOT script instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Run the python suites through pytest-xdist when pytest is importable
(id-level collection parity with unittest discover verified per suite;
serial unittest stays the fallback), with -rP so warnings printed by
passing tests, like the update annotations, still reach the log. run.sh
self-provisions the dev extras through scripts/install/dev-extras.sh,
which stamps inside the venv and skips read-only venvs; make onboard
pre-warms the same path in the running container.
Hoist per-test cache resets and roles scans to setUpClass in the lookup
and resolvable suites (a per-test fingerprint-memo reset closes an
id()-reuse cache-poisoning flake the hoist exposed), inject generation
names into the nfs backup script test instead of sleeping, and narrow
ansible-lint scope to real ansible content. ansible-lint additionally
sits behind a content fingerprint gate that skips the run when nothing
it walks has changed; without a usable git (container bind mounts) the
gate disables itself and the full run remains.
Drop the import-linter layer and ignore for the deleted deps module;
the contract now locks compose -> common -> env. Measured on the host:
unit 101s to 20s, integration 90s to 24s, lint tests 43s to 19s,
repeated ansible-lint 53s to 0.4s.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
01_install.sh runs before python3 exists, so load.sh is skipped and
INFINITO_PACKAGE_INSTALL_SCRIPT was unset; grep it from default.env like
scripts/install/package.sh already does.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
storage.nfs shrinks to the deployment-specific server address: export_base moves into svc-storage-nfs-server's services.yml (the server dictates where exports root) and the mount protocol version into svc-storage-nfs-client's services.yml (the client owns how it mounts). Both are read through utils.storage.nfs loaders by the nfs lookups, the env handler and 00_topology, eliminating the unrendered-variables class for these values; the group_vars assert shrinks accordingly and the swarm test extras stop restating role defaults.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
INFINITO_PYTHON_INSTALL_SCRIPT and INFINITO_DOCKER_CLI_INSTALL_SCRIPT join the default.env SPOT: the Dockerfile consumes them as required args (:? in the RUN as last defense), all three image builders (compose.yml, build.sh, push.sh) pass them hard-required, the shell consumers read the SPOT via default.env grep and the Makefile via BASH_ENV. INFINITO_SWARM_NFS_EXPORT_BASE is no longer a static default: a new env handler derives it from the deploy SPOT group_vars/all/15_storage.yml, 00_topology greps that file (and the STATE_SUBDIR constant) directly, and the swarm test extras stop restating the group_vars value. Stale hadolint directives are restored and recognised; leftover narration comments in the touched files are dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
04_bootstrap_nodes.sh consumed INFINITO_PACKAGE_INSTALL_SCRIPT and INFINITO_NODE_SRC_DIR from the ambient environment; local make targets provide them via BASH_ENV, but plain workflow steps do not, so every swarm CI job died at the required-var guard. Source load.sh explicitly like copy_worktree_to_node.sh.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
INFINITO_PACKAGE_INSTALL_SCRIPT in default.env replaces the scripts/install/package.sh literal in the Dockerfile (new required build arg), the environment install step, and both swarm bootstrap call sites. Shell consumers hard-require the variable.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A wedged DinD dockerd hung a single diagnostics docker exec until the 360-minute job timeout cancelled the job together with its artifact uploads (job 86013153099 sat 5h in service logs). Route every diagnostics exec through a timeout wrapper, tail-limit the log dumps, bound showmount, cap the collect step at 30 minutes, and dump the pending services' log tails once on the final converge failure. The NFS state path moves into a single source of truth: STATE_SUBDIR in utils/storage/nfs.py feeds the nfs_state_subdir lookup (deploy side) and the derived INFINITO_SWARM_NFS_STATE_PATH env var (harness side); the diagnostics script requires both path vars instead of defaulting, and the swarm inventory drops its redundant export_base restatement.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The per-role existence asserts and the test script path were controller paths evaluated on the remote swarm node (docker connection, node checkout under /opt/infinito-nexus), so every svc-runner swarm job failed on a stat that could never succeed. Assert both required assets in one localhost-delegated loop, stage roles/<app>/files/test onto the node, run the staged script, and delegate the optional defaults/vars probes to the controller (they silently skipped before). svc-runner's test reaches its sources via RUNNER_SRC_DIR (compose: INFINITO_SRC_DIR dev-container mount; swarm: node checkout). The node checkout path itself moves into a single source of truth: INFINITO_NODE_SRC_DIR in default.env, consumed by the bootstrap/copy scripts, the swarm-playwright target, and the swarm inventory extras.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rename the meta/services.yml key across all roles, the roles_with_default_placement lookup (now roles_with_placement), the get_role_default_placement / iter_roles_with_default_placement helpers, and every consumer (swarm closure, compose_volumes filter, nfs_prep, container deploy template, constructor stage, extend_inventory) plus their tests and docs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the global TIMEOUT_FACTOR and a shared timeouts.js helper: resolveTimeout(base) scales by TIMEOUT_FACTOR and, over a .onion target, an extra multiplier, so Tor's per-request latency no longer trips hard-coded timeouts. Migrate every raw timeout in the specs (timeout:/timeout=/setTimeout/waitForTimeout) to resolveTimeout, stage the helper in all three staging paths, and add test_no_raw_timeouts to forbid raw numeric timeouts. Clearnet behaviour is unchanged (factor 1: resolveTimeout(N) === N).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
act/workflow.sh: build local/act-runner-fixed:latest on demand when that tag is selected but the image is absent, avoiding an image-not-found failure. swarm/_context.sh: probe_app_reachable falls back to a bash /dev/tcp connect and finally the container's own Docker health status, so toolless images (e.g. fider's debian-slim without curl/wget/nc, which exit 127 on every in-container probe) are no longer reported unreachable. Both additions are additive: the curl/wget/nc path and the non-matching act tag are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
test-e2e-cli renders test.env at 0600 in a 0700 stage dir and always removes the stage tree (it holds base64 DB credentials). test-e2e-playwright + the e2e scripts resolve the node SOCKS from the single spot: playwright via lookup('tor_socks', 'host.docker.internal'), onion.sh takes it as a required arg, rerun-spec.sh composes it from INFINITO_TOR_SOCKS_PORT — no 9050 literal or backup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The previous commit read 'producer_rc=${PIPESTATUS[0]}' then
'consumer_rc=${PIPESTATUS[1]}', but the first scalar assignment resets
PIPESTATUS to a one-element array, so the second read hit an unbound
index and 'set -u' aborted the collector with exit 1 - failing every
swarm job whose deploy and tests actually passed (jenkins, pixelfed).
Snapshot the whole array in one statement, then index it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The collector piped 'docker exec tar | tar' with '|| true', so a hung
'docker exec' blocked the run indefinitely and failures were swallowed.
Bound it with 'timeout 900'; a producer exit of 124 (manager hung) now
fails the collector via PIPESTATUS, while other collection errors stay
best-effort so partial reports do not fail the job.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- svc-net-tor: expose SSH (22) + Mailu SMTP/submission/IMAP (25/465/587/143/993)
as HiddenServicePorts on the node onion (TOR_ONION_SSH_ENABLED, 19_tor.yml).
- email lookup: relay plaintext (tls off, port 25) for .onion mail domains —
no public TLS cert for a .onion; onion transport is already authenticated.
- CA root CN: drop the " Root CA" suffix when a 62-char v3 onion + suffix would
overflow the X.509 commonName 64-char cap (02_tls.yml).
- Playwright onion compat: HSTS is HTTPS-only (RFC 6797), absent on the http
onion vhost. New shared helper expectHstsWhenTls() gates the assertion on the
reached scheme; wired into web-svc-cdn spec.
- rerun-spec.sh: route the inner-loop browser through the Tor SOCKS proxy when
the staged env targets a .onion, mirroring the deploy-time runner.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The post-reschedule assert mounted the state export with vers=3, which
ganesha does not serve ('Protocol not supported'), so the marker check
false-negatived and killed the gitea roundtrip; reproduced and the
v4 mount shown to read the marker on the live cluster. Replace the
single mount with a cascade covering every production client_src form:
nfs4 vers=4 against the full path (ganesha), against :/ (kernel fsid=0
root), then the previous vers=3 full-path arm (kernel v3). The block
also returned the umount status instead of the grep result, letting a
missing marker pass silently; propagate the grep rc explicitly.
The in-container reachability probe's last arm used bash /dev/tcp, but
e.g. the ejabberd image ships neither bash nor curl; use busybox
nc -z instead and move the probe chain into probe_app_reachable() in
_context.sh alongside a NFS_CHECK_MOUNTPOINT SPOT.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Turn a NAT/edge node into a full Tor v3 onion node: DOMAIN_PRIMARY
becomes the node's .onion, so the whole stack (web, SSO, LDAP, CA)
resolves onion domains and is reachable over Tor without a public IP,
port forwarding, VPS or DynDNS.
- utils/tor_onion.py: offline v3 key mint (address + HiddenServiceDir
key files), verified against a known key->address test vector.
- cli .../inventory/onion: `init-env` pre-mints the node onion, writes
INFINITO_DOMAIN(+_RE) into .env and authoritative key files into
.onion-identity/hs (idempotent). The onion is a deploy-time input;
no in-deploy random minting (that would not match the pre-set domain).
- roles/svc-net-tor: Tor sidecar (Debian-built, network_mode: host),
installs the minted keys, torrc with one node HiddenService + a SOCKS
proxy restricted to loopback/DiD ranges. files/test.sh probes every
served onion domain over the Tor SOCKS.
- tls: .onion -> TLS off (onion v3 crypto is the transport auth; LE
cannot sign .onion) + app-level flavor override.
- harness/DNS: coredns resolves the onion zone via INFINITO_DOMAIN;
skip the clearnet DNS sanity check for onion; raise nginx
server_names_hash_bucket_size to 128 for long onion server_names.
- test-e2e-playwright: gated browser SOCKS proxy (PLAYWRIGHT_PROXY) +
navigation/action timeouts so specs reach .onion over Tor and fail
fast; the onion node points it at the Tor SOCKS.
- docs/requirements/031: full design.
Verified: dashboard + SSO (Keycloak) + 9 apps reachable over Tor
(HTTP 200 through the node hidden service). make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
container cp needs its source on the node running the docker CLI, so
role_path sources break on swarm workers. The joomla plugin dir now
ships as a controller-side tarball through container exec -i. A new
repository lint forbids docker/container cp in tracked .yml and .sh
files; the six legitimate same-node uses carry justified nochecks.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/var/lib/infinito on the swarm manager is an NFS mount inside the container's own mount namespace. docker cp reads the daemon-side rootfs and sees the empty mountpoint, so it copied zero files and the uploaded Playwright artifact was always empty. Stream the reports out through docker exec (in-namespace), the same access pattern the swarm diagnostics already use.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
moodle mail failed two ways in a swarm cluster. (1) moodle's build_context
pulled the FULL sys-svc-mail-msmtp role early, which rendered /root/.msmtprc on
the manager before Mailu had created the no-reply token and tripped the role's
run-once flag, so the manager kept an empty-password msmtprc and the health
check failed 'authentication method PLAIN needs a password'. Pull only
tasks_from: ca_bundle so the manager re-renders on the normal post-token pass.
(2) msmtprc forced host=127.0.0.1 for any in-container host, but the NFS server
sits outside the swarm mesh and has no local Mailu, so it hit 'connect to
127.0.0.1:465 refused'. Narrow the 127.0.0.1 branch to compose or a real
svc-swarm-node AND to when Mailu is actually deployed (email.external); else use
email.host, and map the NFS node's test DNS to the manager IP. Also import the
shared self-signed Root CA for the relay's TLS trust.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
down_act_outer.sh is a best-effort pre-clean (set +e), but its last statement (docker rm -f) can race act's own --rm finalize and return non-zero, so the script exited non-zero and aborted swarm-zombie before the deploy. Force exit 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The converge-failure diagnostics truncated service logs to --tail 50 (and head -100), but a crashing service prints its exception at the TOP of the trace, so the root cause was cut off and non-converging services could not be debugged from CI. Remove the caps in the converge-failure rescue (wait_converged.yml) and the diagnostics collector (12_collect_diagnostics.sh) so the full log, including the exception, is captured.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
docker rm -f on a node stuck on a dead NFS mount races (did not receive
an exit event) and leaves the container stopped but not removed. Add a
SIGKILL + plain docker rm retry loop before the host-docker-restart
fallback, so make swarm-clean self-recovers without sudo.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A multi-variant app expanded into N swarm matrix entries, and
swarm-zombie ran them all on one host under identical SWARM_NAME
cluster names, so every deploy after the first aborted with
'network ... already exists'. Pin the act matrix to variant 0
(default, overridable via variant=) so a multi-variant app validates
a single cluster, in parity with the compose step's variant=0.
workflow.sh now splits ACT_MATRIX on ';' into multiple --matrix
filters so apps and variant can be pinned together.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Swarm deploy-test harness fixes (run 28439744622); swarm-test-harness only,
compose path untouched:
- NFS_STATE_PATH (<export>/infinito-state/<vol>): per-volume dirs are nested
under the fsid=0 state subdir (STORAGE_NFS_STATE_SUBDIR), but the seed,
assert and diagnostics used the bare export base -> "Directory nonexistent".
00_topology derives it, the workflow exports it, 09/11/12 consume it.
- pull-retry: pre-pull jrei/systemd-ubuntu with bounded retries before
docker run, so a Docker Hub registry blip no longer fails the job (exit 125).
- jitsi: derive the converge/drain SERVICE_NAME from proxy.service_key so
multi-service stacks (jitsi_jitsi-web/-jvb/...) resolve; single-service apps
fall back to the entity name (unchanged).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The test-deploy-compose "Fail on deprecation warnings" gate failed ~10 apps
while the deploys themselves succeeded. Sources cleared:
- svc-db-postgres: community.postgresql `db` -> `login_db` (the `db` param
alias is deprecated, removed in collection 5.0.0)
- assert_no_deprecation_warnings.sh: ignore ansible-core's legacy filter/test
PluginLoader-aliases deprecation (needs a collection migration with
meta/runtime.yml, not fixable in role YAML)
- joomla/wordpress: drop the `{{ }}` delimiters from task conditionals
(when/until/changed_when), per ansible-core 2.23
Also drop `upstream`/`compat`/`security`/`bug` from the comment-policy markers
and delete those comments from the joomla install tasks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
