Commit Graph
14682 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 4.8 030f03eff4 fix(bbb): give the graphql server a realistic cold-start window
The Postgres-dependent Hasura was still inside its 10s start_period
when the terminal hlth-container gate fired on a cold DinD node, so the
whole deploy failed on a container that was merely booting. 10m matches
the kix backend budget; start_period only defers the unhealthy verdict,
a passing probe still flips healthy immediately.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 16:15:28 +02:00
kevinveenbirkenbachandClaude Opus 4.8 b8f1090ed0 fix(svc-runner): give the nested test deploy a 3h exec budget
The sync-pass test runs a full cold nested build plus reinstall deploy
in one container exec; the CI inventory's 60min
CONTAINER_EXEC_TIMEOUT SIGKILLed it mid-deploy (output truncated with
no nested recap). Export a 3h budget for the test - finite rather than
0 so the OS-level kill still fires on a wedged docker socket, bounded
by the outer 4h cli.timeout.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 16:15:28 +02:00
kevinveenbirkenbachandClaude Opus 4.8 4814367451 fix(penpot): cap the per-replica database pool below the postgres budget
penpot's HikariCP pool fills eagerly and upstream defaults to 60
connections; with 3 swarm replicas that exceeds the non-superuser slots
(max_connections 170 minus reserved) and the post-deploy backup's
pg_dump is rejected with 'remaining connection slots are reserved for
roles with the SUPERUSER attribute'. Cap the pool per replica from the
topology group size. Deliberately not using
POSTGRES_ALLOWED_AVG_CONNECTIONS: its consumer count resolves to 1 and
it is replica-blind; repairing that SPOT is tracked in
https://github.com/infinito-nexus/core/issues/416.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 16:15:28 +02:00
kevinveenbirkenbachandClaude Opus 4.8 d205e32545 fix(kix): actually render the 10m healthcheck start period
compose.yml.j2 set container_healthcheck_start_period but no template
in the include chain consumed it, so swarm honored the image's short
default start period and killed the cold-starting backend as unhealthy
before its init finished (not converged on every variant). Emit a
healthcheck block with only start_period so docker keeps the image's
own curl probe and merely widens the cold-start window; the frontend
gets the same override since it converges equally slowly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 16:15:28 +02:00
kevinveenbirkenbachandClaude Opus 4.8 7a874fc266 fix(compose): exclude services reusing locally-built images from compose pull
--ignore-buildable only skips services that declare build:, so a
service that merely reuses an image another service builds (e.g. a cron
service on the app's custom tag) made docker compose pull abort the
whole run with 'pull access denied', taking the legitimate mirror pulls
down with it. Compute the locally-built image set from the compose
config and pass only genuinely pullable services to pull; skip the pull
entirely when every image is built locally.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 16:15:28 +02:00
kevinveenbirkenbachandClaude Opus 4.8 8d5dd59930 fix(seaweedfs): escape sidecar dollars for stack deploy only
docker stack deploy interpolates $ in command strings and needs $$
escapes, but the compose path passes $$ through verbatim, so the
escaped sidecar script reached busybox as PID expansion plus a syntax
error and crash-looped (opentalk compose). Gate the escape on
DEPLOYMENT_MODE == 'swarm'.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 16:15:27 +02:00
kevinveenbirkenbachandClaude Opus 4.8 3a6c181dc8 fix(espocrm): tolerate an existing admin during upstream first-boot install
The upstream v10 entrypoint runs create-admin-user unguarded under
set -euo pipefail; on a persistent database that already holds the
admin (redeploy, DR recovery) it exits non-zero and aborts the install
before isInstalled is persisted, so every boot re-runs the install and
crash-loops. Append '|| true' the same way the adjacent set-password
line is already patched; the grep guard fails the build if upstream
moves the line.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 16:15:27 +02:00
kevinveenbirkenbachandClaude Opus 4.8 6cc9228a10 fix(swarm-nfs): materialize export subdirs per node and honor volume nfs metadata
The export subdir was created run_once on the NFS server only; a worker
that had already readdir'd the export kept a negative dentry and swarm
rejected every replicas>1 task with 'readdirent .../_data: no such file
or directory' for the whole converge window. Materialize the bind-device
dir through each node's own mount, guarded by a per-node mountpoint
assert so an unmounted node fails the deploy loudly instead of receiving
a local mkdir that splits the brain and trips the nfs-client
has-local-data assert on the next full deploy.

compose_volumes popped the volume 'nfs' metadata before nfs_prep could
read it, so declared uid/gid/mode never took effect and every subdir
fell back to root 0755; keep the mapping under the compose-spec
extension key x-infinito-nfs (ignored by docker, verified against stack
config and compose config). KIX declares uid/gid 110010100 mode 0770
through it, fixing the backend_api_token EACCES crash on its shared
volume.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 16:15:27 +02:00
kevinveenbirkenbachandClaude Opus 4.8 f483ee9c9b fix(tls): trust the self-signed CA from PHP in wrapper-less containers
PHP's libcurl and stream layer ignore CURL_CA_BUNDLE/SSL_CERT_FILE, so
swarm containers - which get env-and-mount CA injection only, the
entrypoint wrapper being stack-deploy-incompatible - fail TLS against
self-signed in-cluster services from PHP code (verified live: curl CLI
reaches keycloak, PHP curl_exec returns 'unable to get local issuer
certificate'; nextcloud's /apps/oidc_login/oidc answered 500 on every
swarm variant). Ship a ca-trust-php.ini pointing openssl.cafile and
curl.cainfo at the mounted CA and bind it into the official-image PHP
conf.d path from the compose CA override in both deploy modes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 16:15:27 +02:00
kevinveenbirkenbachandClaude Opus 4.8 f75ea754f3 fix(swarm): make secret/config name rotation content-addressed on the stack host
The controller-side template cannot read file-backed secret sources
that only exist on the stack host, so the content-hash suffix silently
fell back to hashing the path string and never rotated. A regenerated
source (e.g. a self-signed key re-minted on SAN drift between deploy
passes) then collided with the immutable docker secret of the previous
deploy and stack deploy failed with 'only updates to Labels are
allowed'. Recompute the suffix from the actual file bytes on the stack
host right before the deploy; service references stay untouched since
they point at the yaml key, not the name.

Retention follow-up (pruning superseded generations) tracked in
https://github.com/infinito-nexus/core/issues/415.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 16:15:27 +02:00
kevinveenbirkenbachandClaude Opus 4.8 37d1ea16d4 fix(swarm-drill): restore NFS coherence and numeric tar ownership after the DR restore
Step 8 rewrites the export subtree directly on the ganesha server's
backing filesystem while the server keeps running and the nodes keep
their client mounts, so ganesha's mdcache and the client dentry caches
still describe the pre-wipe tree and update-pass tasks are Rejected
with readdirent ENOENT for the whole converge window. Step 8b restarts
ganesha, lazily remounts every swarm node and gates on a real write
probe through the manager's mount, each command timeout-bounded so a
wedged DinD node fails loudly instead of hanging the job.

The drill's tar extracts remapped ownership by name on the receiving
host like the rsync legs did; extract with --numeric-owner. Drill paths
now come from the INFINITO_DIR_* SPOT env vars.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 16:15:27 +02:00
kevinveenbirkenbachandClaude Opus 4.8 ab3accba3a refactor(paths): derive var-lib subpaths everywhere from the paths SPOT
Add DIR_BACKUPS to utils.paths and INFINITO_DIR_BACKUPS /
INFINITO_DIR_SECRETS env handlers, derive the recover-console constants
(including NFS_EXPORT_STATE) from utils.paths and the nfs export SPOT,
and switch every remaining self-built /var/lib/infinito subpath - the
secrets recover targets, the swarm extras backup source, the users
token-store candidate chain, the CI recover drill, the swarm NFS
diagnostics and the playwright reports dir - to those constants. Tests
consume the env constants directly; narration comments in the touched
files are dropped per the comment lint.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 16:15:27 +02:00
kevinveenbirkenbachandClaude Opus 4.8 69321d4135 fix(backup): preserve numeric ownership and NFS coherence across the DR chain
Every cross-host hop in the backup/recover chain remapped file owners
by NAME on the receiving host (rsync -a default; GNU tar extract as
root prefers name lookup), so a restored uid-1000-owned 0600 config
(n8n) came back owned by a foreign uid and the post-recovery redeploy
crash-looped on EACCES. Pass --numeric-ids to the capture, remote-pull,
device-sync and restore rsyncs and --numeric-owner to the drill's tar
extracts.

The drill additionally wiped and rewrote the export subtree directly on
the ganesha server's backing filesystem while the server kept running
and the nodes kept their client mounts: ganesha's mdcache and the
client dentry caches still described the pre-wipe tree, so update-pass
tasks were Rejected with readdirent ENOENT for the whole converge
window (moodle, espocrm multi-replica). Step 8b now restarts ganesha,
remounts every node and probes a real write through the manager's
mount, each bounded by timeout so a wedged DinD node fails loudly.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 16:15:27 +02:00
kevinveenbirkenbachandClaude Opus 4.8 fdc8fb6d4f fix(swarm-cache): hijack only HTTP apt mirrors on DinD nodes
Image builds on the swarm test nodes inherit the node /etc/hosts, and
build containers carry neither the package-cache frontend CA nor any
skip-cert config, so every HTTPS-defaulting hijack broke them with
untrusted-cert errors (openresty apk via dl-cdn.alpinelinux.org,
simpleicons npm install via registry.npmjs.org). Align the node wiring
with the compose-mode build design: hijack only the HTTP apt mirrors
and keep apt.list, dropping the pip/npm hijacks and their configs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 16:15:26 +02:00
kevinveenbirkenbachandClaude Opus 4.8 6b40d63d30 fix(svc-net-tor): portable dnsmasq filter + non-fatal real-Tor probe
filter-rr was added in dnsmasq 2.86, so it aborts dnsmasq startup on distros shipping older builds (CentOS Stream 9 has 2.85). Use filter-AAAA instead, which parses on every CI distro (dnsmasq 2.85 through 2.93, verified per-distro with dnsmasq --test).

Split onion_self_reach.py into internal/external/both modes. Keep the internal loopback fast path as the hard gate (later host-side steps resolve the onion through it), and probe the real Tor path best-effort (retries the full ~450s publish window, failed_when: false, warns): descriptor publication is async, needs an app backend behind the onion, and is blocked on CI runners with no Tor egress, none of which should fail a deploy. Playwright retries the real onion with its own onion-scaled timeouts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 16:05:00 +02:00
kevinveenbirkenbachandClaude Opus 4.8 9319d7ff62 fix(tor): annotate static tor pins in the WebRTC variants
The variants-static-override integration test flags variant overrides that repeat a static meta/services.yml literal. The tor enabled:false pins across the WebRTC roles' variants are intentional hard-invariant documentation, so annotate them with nocheck: variants-static-override like their shared:false siblings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 13:58:17 +02:00
kevinveenbirkenbachandClaude Opus 4.8 8a2e0f6a2e fix(svc-net-tor): portable dnsmasq drop-in + tolerant self-reach gate
CI deploys on Arch runners died at the dnsmasq drop-in render: the pacman dnsmasq package ships neither /etc/dnsmasq.d nor an active conf-dir, so create the drop-in directory and enforce conf-dir in /etc/dnsmasq.conf (restarting dnsmasq when it changes). The self-reach gate then blocked every node without a web server on :80 (standalone svc-net-tor, dev environments): with the loopback shortcut an instant connection refusal proves the fast path works, so treat it as reached; likewise curl rc 52/56 over the Tor SOCKS require a completed rendezvous (descriptor published) and count as reached, while SOCKS failures and timeouts still fail.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 13:58:16 +02:00
kevinveenbirkenbachandClaude Opus 4.8 c596ad0e20 fix(tor): WebRTC roles never ride the node onion
coturn, bigbluebutton, jitsi and opentalk move WebRTC media over UDP (STUN/TURN, SRTP, JVB, LiveKit SFU), which Tor cannot carry. Pin their tor service block to enabled/shared false in meta/services.yml and in every variant, so no deploy scope or variant ever injects onion domains for them and every consumer lookup renders their clearnet URLs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 13:27:20 +02:00
Alejandro Roman b796260c4f Merge remote-tracking branch 'upstream/main' into feature/role-stalwart-mail 2026-07-14 11:37:13 +02:00
kevinveenbirkenbachandClaude Opus 4.8 07c6ad8e7c test(e2e): scale every nav click + timeout literal for onion targets
Over Tor the redirect chain behind a login/logout/OIDC click routinely exceeds Playwright's fixed 30s action timeout, so the click throws before the flow's scaled waits run. Sweep all role Playwright suites and the shared persona helpers: give every navigation/auth-transition click an onion-scaled timeout (.click({ timeout: resolveTimeout(30_000) })) and route the remaining raw millisecond literals (timeout options, waitForTimeout, sleep-promise setTimeout, positional wait args) through resolveTimeout. Local UI clicks (menus, toggles, modal dismissals, DOM clicks inside evaluate) stay untouched.

Two new lints keep it that way: test_no_raw_timeout_literals forbids raw numeric timeout literals in Playwright specs (opt-out: nocheck: raw-timeout), test_persona_clicks_scaled requires every click in the shared persona auth flows to carry a resolveTimeout timeout (opt-out: nocheck: unscaled-click).

Verified: node --check on all 133 changed JS files, all 13 playwright lints green, make autoformat clean, make test green (2745+196+163+4 tests OK).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 11:35:53 +02:00
kevinveenbirkenbachandClaude Opus 4.8 f2825407f6 fix(web-app-nextcloud): disable files_linkeditor on nc33 (removed New-menu API)
files_linkeditor 1.1.25 registers its New-link entries via the legacy fileList.newFileMenu API that Nextcloud removed in the v28 Files rewrite, so the entries never render on nc33 (the menu opens and every other addon registers, this one is absent) - network-independent, not Tor-related. Disable the addon and let its spec skip via skipUnlessAddonEnabled until an upstream version using addNewFileMenuEntry ships.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 10:50:09 +02:00
kevinveenbirkenbachandClaude Opus 4.8 5e194d885d test(web-app-nextcloud): onion-harden the Playwright suite
Scale the login helper's waitForVisibleCandidate call-sites and the Sign In / Log out clicks through resolveTimeout, and wait for network idle after logout so the slow keycloak end-session redirect chain over Tor settles before the logged-out state is probed.

Skip the Talk-admin and SeaweedFS-filer specs on onion targets (WebRTC HPB/STUN/TURN and a clearnet-only filer UI). Assert the XWiki partner host against XWIKI_BASE_URL from the env instead of a hard-coded clearnet host, so the check holds on both clearnet and onion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 10:50:09 +02:00
kevinveenbirkenbachandClaude Opus 4.8 d5aa44715b fix(web-app-nextcloud): route OIDC through Tor SOCKS on an onion node
On an onion node the OIDC issuer (Keycloak) is a .onion; libcurl (oidc_login's Guzzle client) refuses to resolve .onion per RFC 7686, so the server-side OIDC backchannel (discovery/token/jwks) fails. Set Nextcloud's system proxy to socks5h://host.docker.internal:<tor-socks> and expose the host Tor SOCKS to the app container via extra_hosts, both gated on SSO being enabled and the Keycloak domain being an onion. No-op on clearnet nodes.

Hoist the repeated services.sso.enabled and Keycloak-domain lookups into NEXTCLOUD_SSO_ENABLED and NEXTCLOUD_OIDC_HOST constants (mirroring MATRIX_OIDC_HOST) and use them across the role.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 10:50:09 +02:00
kevinveenbirkenbachandClaude Opus 4.8 453ddb9170 fix(web-app-bigbluebutton): do not load BigBlueButton on an onion node
BigBlueButton is WebRTC (SRTP/UDP with ICE/STUN) which Tor cannot carry. Guard the role's core include on the resolved Keycloak domain not being an onion (alongside the existing run_once guard), disable the co-installed cloud_bbb Nextcloud app on onion at runtime, and skip the BBB Playwright specs on onion targets.

Relax the run_once schema test (test_service_core_first_task_run_once) to accept a when-list that contains the run_once guard, so a role can add extra include conditions while staying protected against duplicate execution.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 10:50:09 +02:00
kevinveenbirkenbachandClaude Opus 4.8 dcd7f9861c fix(svc-net-tor): gate deploy on both onion paths + node-onion loopback shortcut
Replace the internal-only reachability one-liner with onion_self_reach.py, which gates the role until the node onion answers both internally (host resolver loopback fast path) AND over the real Tor SOCKS (descriptor published — the path Playwright/users take). Add a dnsmasq address= shortcut mapping the node's own onion suffix to loopback so host-side deploy steps skip the Tor circuit; other .onion still egress via Tor and container/Playwright traffic (Tor SOCKS) is unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 10:50:08 +02:00
kevinveenbirkenbach c107c49fad Raised timeout from 10 to 20 2026-07-14 05:01:33 +02:00
kevinveenbirkenbachandClaude Opus 4.8 9b5d014d72 docs(swarm-test): drill recovery verified after the update pass
Reflect the folded drill: base.sh ends after recovery, the matrix update pass boots the recovered stack, and verify_recovered_marker.sh asserts the live marker.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 04:42:32 +02:00
kevinveenbirkenbachandClaude Opus 4.8 84811a4c41 fix(swarm-test): detach stale loop devices before LUKS format
A prior wedged drill run can leave loop devices bound (backing file deleted, or bound to the image path); a full loop pool makes cryptsetup luksFormat abort with "loop device with autoclear flag is required". Detach the orphaned and stale loops first so cryptsetup can autoclear-attach a fresh one.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 04:42:32 +02:00
kevinveenbirkenbachandClaude Opus 4.8 29c01a55dc fix(espocrm): static patch-config task name via loop_control label
The task name templated item.key, which Ansible renders before the loop binds item (object of type str has no attribute key). Use a static name and surface the per-item key via loop_control.label.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 04:42:32 +02:00
kevinveenbirkenbachandClaude Opus 4.8 55787357ef fix(mail): defer sys-svc-mail past the service-loader preload
sys-ctl-alm-email now routes its sys-svc-mail include through the service-loader list_or_shoot queue. During preload a backup provider pulls the alarm -> email -> sys-svc-mail chain in before Mailu is preloaded, so the "Mailu was preloaded" assert fired and msmtprc would render with an empty no-reply token. Queuing sys-svc-mail to the post-load pass runs it after every service (including Mailu) is preloaded and the token is minted.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 04:42:32 +02:00
kevinveenbirkenbachandClaude Opus 4.8 c44235d792 perf(swarm-test): fold DR-drill redeploy into the matrix update pass
The drill (backup/base.sh) no longer runs its own step-10 redeploy; it recovers device -> nfs export -> volume -> secrets and writes a sentinel. The matrix update pass (which runs next anyway) boots the recovered stack, and verify_recovered_marker.sh asserts the live marker after it. Saves one full stack redeploy per drilled app; the marker-gone check before recovery stays in 05_wipe_export.sh.

Also derive recover_drill.sh REPO from the script location instead of an INFINITO_REPO_ROOT shell-side default (lint: no INFINITO_* / absolute-path defaults in .sh).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 04:42:31 +02:00
kevinveenbirkenbachandClaude Opus 4.8 e493a92a1d fix(svc-bkp-volume): force a post-deploy backup in the E2E test
The pre-state backup runs in service-loader preload before the app volume exists, so the test could verify against an empty generation. Trigger a fresh backup unit run before the verify and restore cycle so a populated generation is present.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 02:47:40 +02:00
kevinveenbirkenbachandClaude Opus 4.8 e77aae2afe perf(service-loader): single applications-load preload
Flatten a single list term in the service lookup so query(service, [ids]) does one applications load instead of one per id. Precompute the required service ids in sys-service-loader and gate the preload by set membership, collapsing about 30 applications loads into one.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 02:47:40 +02:00
kevinveenbirkenbachandClaude Opus 4.8 7ba69c69aa refactor(swarm-test): distro node image, cache wiring, disk-backed NFS export
Node image now builds FROM the distro infinito image (python, ansible, CLI baked in) instead of installing a per-node .deb, so there is no per-node bootstrap or timeout; the swarm CI job depends on the distribution build. Move compose/swarm.yml to compose/swarm/compose.yml and relocate the swarm matrix orchestrator from cli.administration.deploy.swarm to utils.tests.swarm (test code out of the shipped CLI).

Back the NFS export root with a disk-backed named volume instead of a RUNNER_TEMP bind: local-act RUNNER_TEMP resolves onto tmpfs and ganesha FSAL_VFS cannot export a tmpfs, so it loaded zero exports and the controller mount got ENOENT. Add optional compose/swarm/cache.override.yml wiring swarm nodes to the package-cache frontend when present, leaving CI without a cache unaffected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 02:47:40 +02:00
kevinveenbirkenbachandClaude Opus 4.8 ac09f25750 fix(cache): metadata/negative TTL so unknown packages refresh
Split the single cache freshness window into content (immutable artifacts: wheels, debs, rpms) and metadata/negative-cache (indexes, not-found lookups). Content keeps the long 6-day window; metadata and negativeCache.timeToLive default to 0 so Nexus always revalidates indexes, and a newly published version shows at once instead of being masked by a stale cached index.

ensure_proxy_repo now PUT-updates an existing proxy (POST only creates), so the split applies to already-provisioned repos. Covers every ecosystem proxy (apt, pypi, npm, rubygems, packagist, alpine, go), not just python.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 02:47:40 +02:00
kevinveenbirkenbachandClaude Opus 4.8 5541b132d1 feat(backup): recover drill between compose deploy passes
Add scripts/tests/deploy/ci/recover_drill.sh: seed a backup root (nfs, volume, secrets, each with a marker) plus a docker volume, then run recover full <root> localhost to restore every backup-stored type into its live target and verify the markers. Runs between PASS 1 and PASS 2 of the compose development deploy.

Hook it via _maybe_recover_drill in cli.administration.deploy.development, gated by INFINITO_RECOVER_DRILL (default true).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 02:47:40 +02:00
kevinveenbirkenbachandClaude Opus 4.8 75d84af636 feat(recover): unified recover CLI + swarm DR drill via CLI
Add cli.administration.recover: one uniform type/source/target command wrapping each svc-bkp-* role files/recover.py (nfs, volume, secrets, device) plus full for whole-tree recovery. Colon-encoded sources are classified by shape; device supports an on-device subpath, a generation, and remote pull over ssh.

Replace the per-host swarm DR-drill scripts (03/06/07/08_recover_*.sh) with the recover CLI in backup/base.sh. Pass the device on-device subpath relative (the CLI reads absolute segments as the restore-root) and load the applications-free .deploy.yml extras twin on the step-10 redeploy so host_vars credentials survive the -e override.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-14 02:47:39 +02:00
Alejandro Roman bdeab36bd4 next cloud issues with storage vs mailu 2026-07-14 01:16:31 +02:00
Alejandro Roman df71fb9d40 fixes in CI 2026-07-13 18:20:35 +02:00
kevinveenbirkenbachandClaude Opus 4.8 c25872a388 feat(tooling): swarm-diagnostic target for backup/NFS post-mortem
Adds make swarm-diagnostic (alias i8sadi): read-only probes across a
live swarm-test cluster's nodes via exec_node.sh - backup unit state +
recent journal, NFS mounts, D-state (wedged-NFS) processes, running
rsync/db-dump processes, and disk usage - so a hung backup unit can be
root-caused without hand-writing docker exec pipelines.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 10:24:34 +02:00
kevinveenbirkenbachandClaude Opus 4.8 8d2d8f4a7c feat(inventory): disable services by service key or provider app id
The disable env only matched service/entity keys (email, prometheus);
naming the provider role (web-app-mailu) silently did nothing. A shared
expand_service_tokens (registry SPOT) now normalizes either spelling to
the provided service key(s), and every disabler resolver plus the
conflict check normalize at entry, so compose and swarm accept both
forms.

derive_includes additionally prunes the disabled providers from the
swarm include closure. Without it the provision-time disabler removed
the provider role but extend_inventory re-added it via the same dep
walk, so the deploy tripped the 'provider role still active' conflict.
Live-verified on a swarm-dashboard run: disable='web-app-mailu,...'
expands to email->web-app-mailu and deploy round 1/1 proceeds with no
conflict.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-13 10:24:33 +02:00
kevinveenbirkenbachandClaude Opus 4.8 76cc5db491 fix(deploy): tolerate first-install state in matrix reconfigure + cached read
- matrix 02_render_inventory: the mdad-state volume does not exist yet on a
  fresh install, so `container volume inspect` errored the reconfigure task
  (`no such volume`) on the row-1 baseline. Guard the marker drop behind the
  inspect succeeding; the runner bounce still runs if the container is up.
- test_coverage: read services.yml via utils.cache.files.read_text so the
  no-uncached-file-reads lint passes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 02:34:00 +02:00
kevinveenbirkenbachandClaude Opus 4.8 21ae788a92 fix(hlth-webserver): skip canonicals of disabled services
A service entry may declare `domains: [<canonical key>, ...]`; when its
`enabled` resolves falsy those canonicals get no vhost (e.g. the seaweedfs
filer/master frontend on an onion node), so probing them can only fail
(TLSV1_UNRECOGNIZED_NAME / ERR_NAME_NOT_RESOLVED). Drop them from the
web-health expectation map. Mirrors the same suppression already added to
the CSP health probe.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 02:34:00 +02:00
kevinveenbirkenbachandClaude Opus 4.8 c8a4320478 feat(lookup): container_hostname — kernel-safe container hostname
`lookup('container_hostname', app)` returns the app's canonical primary
domain when it fits sethostname(2)'s 63-byte cap, else the short entity
name. Onion subdomains (sub.<56-char>.onion) overflow that limit and make
`docker run` fail with "sethostname: invalid argument", killing container
init on onion nodes. Replaces the seven roles that hostname'd their
container on the raw domain (bookwyrm, bridgy-fed, confluence, jira,
postmarks, pretix, xwiki) with the encapsulated lookup.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 02:34:00 +02:00
kevinveenbirkenbachandClaude Opus 4.8 10a11fc570 fix(svc-net-tor): stabilise the .onion egress path for downstream deploys
- dnsmasq: filter-rr=AAAA on the .onion split-DNS. Recent Tor answers the
  AAAA automap with a link-local-shaped v6 address that has no REDIRECT rule,
  so clients that prefer v6 (getaddrinfo) black-holed every onion request.
- self-reach gate: after wiring the egress router, block the role until the
  node onion actually answers over Tor. A (re)started Tor republishes its
  hidden-service descriptor for ~2-5 min; without the gate every later step
  touching an onion URL (front-proxy checks, injections, Playwright) failed
  as if transient. Any HTTP reaction — including OpenResty's 444
  connection-close for the bare node onion — counts as reachable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 02:34:00 +02:00
kevinveenbirkenbachandClaude Opus 4.8 4e6ef25423 fix(test): repo-lint compliance + seaweedfs api-primary readiness
- _shared.js: wrap the boot-poll delay in resolveTimeout (raw-timeout lint)
- 02_render_inventory: docker -> container wrapper (engine-abstraction lint)
- seaweedfs: api becomes the primary canonical so the deploy readiness
  probe hits a served vhost when the frontend (filer/master) is disabled
- seaweedfs frontend: exempt from variant pinning — the onion gate is a
  hard invariant, no variant may force the frontend on a tor node
- test_coverage: add the `# nocheck: variants-coverage` exempt (same
  mechanism as bond-baseline) for such invariant flags

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 02:34:00 +02:00
kevinveenbirkenbachandClaude Opus 4.8 32b2cf7e80 style: make autoformat pass (ruff line wraps, YAML comment spacing)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 02:33:59 +02:00
kevinveenbirkenbachandClaude Fable 5 88dc35fd00 docs(swarm-routine): sequence diagram with backup trigger points
Adds a routine/README.md describing what each phase of the swarm lab
run does (bring-up, the variant-matrix round loop, the DR drill's nine
steps, the chaos phase, teardown), with the moments a backup-executing
systemd unit actually starts highlighted, and links to the per-step
scripts below the diagram.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 01:47:52 +02:00
kevinveenbirkenbachandClaude Fable 5 81d4709e13 fix(swarm-drill): size the LUKS device image from the pulled tree
The DR drill's simulated USB was a fixed 512M loop image (~449M usable
after LUKS2 + ext4), but the deployed local-2-device unit syncs the
whole pulled backup root, which after the remote pull holds two
machine-id trees (the manager's docker+secrets repos and the NFS
export repo). For an app the size of akaunting that overflows the
image mid-rsync and the drill fails with ENOSPC at step [5/9]. base.sh
now measures the pulled tree (du -sm) and passes 2x its size + headroom
(floor 2G) to 02_luks_device.sh, which sizes the loop image from that
argument; fallocate keeps the image sparse so the larger floor is free.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 01:47:52 +02:00
kevinveenbirkenbachandClaude Fable 5 715b207990 docs(backup): deploy-time trigger flow diagram
Second diagram in the backup architecture doc: when the units run
during a deploy, keyed on MODE_BACKUP and install vs update, including
the preload instant snapshots, the end-of-play remote pull and the
abort-on-failed-snapshot semantics.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-13 01:47:52 +02:00