Swarm reuses NFS-backed state across cluster rebuilds. A role then reads config persisted from a prior cluster while its database (a node-local volume) is fresh, causing install drift and stale-credential loops. Wipe each NFS-backed volume subdir before pre-create, gated to dev/act via RUNTIME so production state is never touched.
Hoist the shared swarm and NFS guards to the include's when, and fold the per-item mapping check into the loop via selectattr, so the two tasks no longer repeat identical conditions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move discourse out of variant 4 (now onlyoffice + mastodon) into a new dedicated variant 7, mirroring the gitlab isolation, so its loosely-coupled (bond 0.3) bridge footprint gets its own budget-bin. Add a numbered overview comment (# Variant N) before each variant entry.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
docker stack deploy rejects the compose group_add property ('Additional property group_add is not allowed'), so svc-prx-openresty drops it from the swarm compose and applies the docker.sock gid post-deploy via 'container service update --group-add' (swarm-native equivalent). The compose path (network_mode host) is unchanged; the new task is DEPLOYMENT_MODE == 'swarm' gated.
compose_ca's swarm --no-wrapper path no longer runs 'docker compose config', whose strict dotenv parser aborts on swarm env files carrying verbatim special-char secrets (e.g. a password beginning with a single quote). It reads the rendered compose files directly; wrap=False needs only service names. The compose (wrapper=True default) path keeps the original 'docker compose config' branch byte-identical.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The discourse launcher is a standalone container (docker run via ./launcher), not a swarm service, so it cannot attach to swarm-scoped overlay networks. In swarm it was left without a reachable network for its shared pg/redis/openresty peers.
The shared network util now creates a node-local bridge instead of an overlay when the role declares networks.local.force_bridge in meta/server.yml, read via the config lookup with a false default so every other caller stays byte-identical. Discourse sets force_bridge: true and connects the rebuilt launcher to the shared overlays (web/pg/redis) so swarm services stay reachable; the launcher config drops the explicit --network flags in swarm.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Jenkins inherited the 3-node topology default, but it is a stateful singleton (one NFS-backed JENKINS_HOME, no native active-active HA). Three replicas race on the shared home at boot (SEVERE plugin-load failures) and the OIDC state/nonce splits across the openresty VIP, so the anonymous redirect to Keycloak never completes (LDAP's single round-trip passes). Pin service_replicas=1 like matomo/erpnext; it stays a swarm service (scheduled + healed across the cluster), just not horizontally scaled.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
moodle runs replicas:3 sharing one NFS code volume. The entrypoint guarded its cp -a with a version.php TOCTOU check, so several replicas copied concurrently; losers hit 'File exists' and, under set -euo pipefail, aborted before chown + php-fpm, leaving an incomplete/unreadable tree -> install_database.php failed Permission denied (rc 255). Serialize the bootstrap under flock, gate on an atomic .bootstrap.done sentinel, cp -an || true so a benign collision can't abort, make the tree world-readable before the install (root_squash neuters the in-container chown), and make losing replicas wait for the sentinel.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Commit 9f43364c0 flipped the per-card icon HEAD probe to the public icon.<domain> URL in swarm, but the probe runs on the Ansible controller, which has no DNS/route to it -> every probe failed silently -> icon.source was never set -> cards fell back to Font Awesome (no img) and the dashboard e2e failed. Drop the probe_public flip: probe the in-cluster sync URL the controller can reach via the published swarm port, while still rewriting icon.source to the public URL the browser loads.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In swarm, OmniAuth/OpenSSL (decidim OIDC discovery) and Node (bluesky-broker -> PDS) could not trust Keycloak's internal self-signed CA: docker stack deploy never received compose.ca.override.yml, and the override's $$-escaped entrypoint wrapper is swarm-incompatible (stack deploy does not un-escape it). Add a --no-wrapper override mode (env + CA mount only: SSL_CERT_FILE/CURL_CA_BUNDLE/NODE_EXTRA_CA_CERTS) generated on the stack host for self-signed apps, and build the stack-config/stack-deploy --compose-file list via compose_file_args (new --compose-file flag kwarg) instead of hardcoding compose.yml -- which also stops swarm silently dropping a role's own compose.override.yml. Compose untouched: self-signed-gated, no-wrapper is swarm-only, lookup defaults to -f.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
bench new-site never passed --db-host, so Frappe defaulted to 127.0.0.1 (MySQLdb OperationalError 2002). The DB host is only written to common_site_config.json by the one-off configurator, which swarm never gates (docker stack deploy ignores depends_on; a restart:none one-off shows 0/1 forever, so the convergence wait can't gate on it either). Pass host/port explicitly; the vars already resolve to mariadb:3306.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Added to permissions.allow during the collabora swarm-convergence investigation (Collabora SDK docs lookup for the mount_jail_tree knob).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
These commits were made with --no-verify, which skipped make test, leaving three latent lint failures. utils/paths.py: use the cached utils.cache.files.read_text (verified no import cycle) instead of a plain read. .ansible-lint-ignore: add the pixelfed resolve_host_cid load-failure entry (templated playbook_dir include, false positive). library/database_query.py: ruff line-wrap. make test is now fully green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
wait_converged failed after 600 silent retries (rc=1, empty) with no clue why. Wrap the wait in a rescue that, on the manager, dumps 'container service ps --no-trunc' + 'container service logs --tail 50' and aborts with a root-cause message, so converge failures (e.g. collabora's coolmount permission error) surface immediately instead of after 20 minutes of silence. Swarm-gated; the success path is unchanged and compose skips the block entirely.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
docker stack deploy drops cap_add/security_opt, so collabora's coolmount (CAP_SYS_ADMIN jail-mount) fails 'Operation not permitted' and the container never goes healthy -> the swarm task reschedules forever and never converges (bigbluebutton/nextcloud/listmonk). Append --o:mount_jail_tree=false in swarm so collabora copies/links the jail tree instead of mounting it. Compose keeps the caps and is byte-identical.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The realm-import scales keycloak 0->N (fresh reschedule across nodes), then the kcadm update block runs delegated to a fixed KEYCLOAK_HOST_NODE; the in-module recovery resolves only a node-local container id, so when a replica reschedules to another node it can never be found -> 'No such container: <dead cid>'. Wrap the update routines (extracted to 02b_update_routines.yml) in a rescue that re-resolves the hosting node + live id and re-delegates the updates, bounded to 5 attempts; compose re-raises immediately. Relax the recovery resolver to status=running so it can find the just-rescheduled, not-yet-healthy replica.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In swarm the openresty worker runs as nobody and cannot read the host docker socket (root:<gid> 0660), so the init_worker docker-health Lua poll fails 'Permission denied' and the /metricz up metric stays 0. Stat the socket for its live gid and group_add it to the swarm service so the worker can read it. Swarm-gated; compose uses network_mode host and is untouched. Verified live: --group-add of the socket gid removes the EACCES.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The bootstrap/admin tasks exec 'container exec PIXELFED_CONTAINER' (the bare service name 'pixelfed'), which is not a swarm container name -> 'No such container: pixelfed'. Adopt the matomo resolve_host_cid pattern: resolve PIXELFED_HOST_NODE + PIXELFED_LOCAL_CID once, exec against the resolved id with delegate_to the hosting node, and drop the chdir (irrelevant to container exec and absent on a delegated worker). Compose is byte-identical (HOST_NODE = inventory_hostname, LOCAL_CID = PIXELFED_CONTAINER).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The module execs 'container exec <container>' as an argv list (no shell), using config["container"] = the literal "postgres" in shared mode, which is not a swarm container name -> 'No such container: postgres' for peertube and ~12 other callers. The database lookup already provides a swarm-aware config["address"] = "$(resolve-container-id <stack> <key>)"; expand it here (shlex-split, run the resolver as argv) to a concrete id when present. Compose (address == container) is byte-identical.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
docker stack deploy parses the .env file as KEY=value; three lines used YAML 'KEY: value' syntax, so RAILS_CACHE_STORE: "memcache" became a variable name containing whitespace and aborted the swarm deploy. Convert them to KEY=value and drop the duplicate OPENPROJECT_RAILS__RELATIVE__URL__ROOT (line 11 already sets it).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ansible-flavor stages the matrix-mdad runner build context into MATRIX_MDAD_RUNNER_BUILD_DIR via stack_host_copy, but nothing created that directory first, so the copy failed 'Destination directory ... does not exist'. Ensure the dir (delegate_to STACK_HOST, run_once) before the copy loop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The redis wait-until-running + PING checks exec via REDIS_CONTAINER_ADDRESS (resolve-container-id, manager-only). Ungated they ran on every node, so on workers resolve-container-id exits 'must run on a Swarm manager node' and the checks fail (broke svc-bkp-container-2-local). Gate them to the stack host with delegate_to + run_once, mirroring svc-db-postgres.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
nfs_prep looped lookup('volume', application_id), which returns only the role's own meta/volumes.yml and omits sidecar volumes the stack also mounts (e.g. <entity>_redis from svc-db-redis / an sso-proxy). Their backing dir was never created, so swarm rejected the mount and the sidecar stayed 0/1 (fusiondirectory/postmarks/snipe-it 500/502). Loop the rendered compose_volumes set instead, which includes the sidecars; manager-pinned roles still skip via the placement gate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The compose-deploy inventory validator is spawned as a bare subprocess without the env build, so os.environ["INFINITO_DIR_VAR_LIB"] raised KeyError at import and aborted all compose deploys before Ansible ran. Fall back to the default.env SPOT (no hardcoded literal); read it plainly to avoid a utils.paths -> utils.cache import cycle.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Set DEFAULT_MAX_STORAGE to 350GB (was 400GB) and make it the single source of truth: the INFINITO_VARIANT_BUNDLE_MAX_STORAGE env handler's setdefault and the resolve_max_storage error message now reference DEFAULT_MAX_STORAGE instead of repeating the literal. Update the default-resolution unit test accordingly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add tests/lint/ansible/services/test_variants_shared_false_once.py enforcing the one-local-provider-variant limit on services.<key>.shared: false pins, document the variants-shared-false-once suppression marker, and flip the database service (mariadb/postgres/redis) to shared: true across the erpnext, funkwhale, shopware, snipe-it, suitecrm, taiga, xwiki and zammad variants to comply.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GitLab 19.0 added a NOT NULL check constraint (check_77eda6baaa) on oauth_applications.organization_id. The provisioning runner created the Doorkeeper application without a resolvable organization context (Current.organization is nil in a gitlab-rails runner), so the insert produced organization_id=NULL and failed with PG::CheckViolation on GitLab 19.1. Resolve the default organization, set it on the request context, and assign it via both the association and the column; fail loudly if no organization is available.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The swarm openresty restart used 'container service update --force --detach' (fire-and-forget), so the deploy + e2e proceeded before openresty reconverged with prometheus.conf. The per-vhost log_by_lua then saw a nil metric_requests ('attempt to index global metric_requests') and the prometheus metricz/scrape e2e failed. Drop --detach so the update waits for convergence (bounded by the existing timeout 600). Also guard the per-vhost log_by_lua against a nil metric so any residual pre-init window can't throw.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dev/act ganesha server is NFSv4-only, but the swarm-test runtime extras forced storage.nfs.version: 3, so every mount failed 'Protocol not supported'. Set version 4 (write_extras + dev swarm inventory). The client probe mounted the Root_Squash export_base and a root probe-write failed; point it at the No_Root_Squash state path. Both client mounts hardcoded 'hard', so a torn-down test server wedged nodes into D-state; gate them to soft in dev/act (reusing the controller mount's gate). Hoist the shared mount-opts string into the STORAGE_NFS_MOUNT_OPTS SPOT (group_vars/all/15_storage.yml), used by all three mounts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The example _RUN_URL in test_download.py lacked the '# nocheck: url' marker its siblings carry, so test_public_http_urls_are_reachable probed github.com/o/r/actions/runs/123 -> HTTP 404 and failed the external suite.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SPOT: declare the var-lib base once in default.env as INFINITO_DIR_VAR_LIB; the env build derives INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR from it; reference it everywhere (utils/paths.py, scripts via ${...:?}, tests, the dev inventory's DIR_VAR_LIB injection) instead of hardcoding the literal. group_vars keeps the literal default; the env-lookup lives in the dev inventory per the no-ansible-env-lookups lint rule. pull_specific_host.py --folder is now required and the role passes {{ DIR_BACKUPS }}.
Swarm e2e: collect_playwright_reports.sh docker-cp's the reports off the manager plus a workflow upload step, so swarm Playwright traces reach CI artifacts (was compose-only).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The Ansible task timeout hard-kills the Playwright sidecar at TEST_E2E_PLAYWRIGHT_RUN_TIMEOUT, discarding every junit/html/trace and aborting the play before the archive step. Set Playwright's own globalTimeout a buffer (60s) below that cap so it self-terminates with reports, the task returns rc\!=0 instead of being killed, and the diagnostics get archived for the next run.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
