The up-check filtered 'container ps' by com.docker.swarm.service.name set to SNIPE_IT_CONTAINER_ADDRESS (a container address, not a service name), so it never matched and the wait timed out. Use lookup('container_service', application_id, SNIPE_IT_SERVICE), the same form the keycloak CID resolver uses.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Swarm rejected the jitsi replicas with 'bind source path does not exist' for the jicofo/jvb/config/transcripts dirs, because 02_manager_ops pre-created only the prosody dirs. Add the remaining meta/volumes.yml bind sources to the dir-creation loop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ldapauth.config.php swarm config was mounted mode 0440 root:root, but Friendica runs as www-data and could not read it, so the container crash-looped and the schema task failed with 'No such container'. Widen to 0444; the config is mounted only into Friendica's own container, so the read stays container-local.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The administrator/migration include execed in the taiga_manager container before the flush_handlers that deploys the stack, so the swarm play failed with 'resolve-container-id: no running task for service taiga_manager'. Move the flush_handlers ahead of the include so the manager service exists before migrate runs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The svc-bkp play loads svc-db-postgres on every swarm node, so the un-delegated container readiness waits and the superuser-password task ran on workers and failed with 'resolve-container-id: must run on a Swarm manager node'. Add delegate_to STACK_HOST + run_once to the three core tasks, mirroring svc-db-mariadb.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reorder the clean-container-owned / clean-pycache-dirs / clean-stale-nfs Makefile targets into ascending order, set the executable bit on the swarm NFS cleanup scripts, and opt the CI-logs analyze and download-test file reads out of the cached-read lint (a runtime read of a user-supplied log, and a temp fixture written inside the same test).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In swarm the bare container name 'moodle' collides with the same-named overlay network, so 'container inspect moodle' returns the network object (which has no .State.Health.Status) and the health wait dies with 'map has no entry for key "State"'. Use MOODLE_CONTAINER_ADDRESS in the health-wait and the two rescue diagnostics, matching the install task that already resolves the container that way.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
python -m cli.meta.ci.logs.download fetches a run's job logs and artefacts in parallel, with conclusion filters (-s/-f/-c/-k), a jobs.json/summary.tsv manifest, per-request jitter to dodge API rate limits, and idempotent re-runs. python -m cli.meta.ci.logs.analyze clusters the downloaded logs by their root failing task.
Rename the agent CI-triage page debug/pipeline.md to debug/ci.md, point its download/inspect step at the new CLI, and reference the tooling from the contributing CI-debugging page. Add unit tests for both commands.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SSO-without-LDAP left the OIDC admin user without nextcloud admin rights -> the admin-settings Playwright addon tests hit 'Access forbidden' in every sso variant that lacked ldap (v2-v6), while v0 (sso+ldap) passed. Enable ldap (enabled+shared) in every sso variant and make mariadb shared everywhere except variant 1 for a consistent multi-app topology.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
database_query opens query_file with Path().open() on the managed host. In the swarm test the managed host is a remote SSH node that does not carry the repo files, so the controller-side path '{{ role_path }}/files/sql/...' does not exist there and the query aborts with 'could not read query_file ...: No such file or directory'. Pass the SQL as content read on the controller via lookup('file', ...) instead, which works in both compose and swarm.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 03_schemas include ran its ldapsm imports on every swarm node, so multiple nodes wrote the same custom schema into the single shared OpenLDAP concurrently (result 80 'AttributeType not found' on one node, result 20 'Type or value exists' on another). Apply run_once + delegate_to STACK_HOST to the include so the schema provisioning runs exactly once on the stack host, matching the other mutating tasks in the role.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
docker stack deploy rejects an env file whose variable name carries trailing whitespace ('variable LDAP_ENABLED contains whitespaces'), so the swarm deploy aborted before funkwhale started. The aligned LDAP_ENABLED / LDAP_BIND_PASSWORD / LDAP_START_TLS / DATABASE_URL lines used 'KEY = value'; collapse them to 'KEY=value'.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-role swarm overlay local subnet was a /28 (14 usable IPs). A role's swarm stack plus its sidecars and per-task network attachments can need more than that, which surfaces as 'node is missing network attachments, ip addresses may be exhausted' and leaves services (sso-proxy, app frontends) unresolvable mid-deploy. Re-allocate all 74 /28 roles to a unique /24 (254 usable) inside 192.168.0.0/16, chosen via the network address suggester so the assignments stay non-overlapping. Validated by tests/integration/infrastructure/networks/test_unique_valid_and_mapped.py and a full make test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When swarm replaces the keycloak replica mid-deploy, the cached container id dies and 'Update Client settings' aborts with 'No such container'. The kcadm module re-resolves a fresh health=healthy id and retries, but the 40 x 3s = 120s budget could expire before the new replica finished its cold boot (jgroups + infinispan + db index check), leaving the per-app SSO client unconfigured. Raise DEAD_CID_MAX_RETRIES to 80 (240s) so the recovery outlasts a keycloak cold boot.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Conflicts resolved keeping the branch's structure and taking main's version bumps: web-app-pihole kept the branch's lookup-based implementation (container_image/ports/volumes plus compose_volumes/networks lookups and meta/volumes.yml); web-app-erpnext took main's image version v16.25.0; web-app-gitlab took main's 19.1.1-ee.0; CHANGELOG took main's [10.1.1] entry. The branch's swarm-nfs structural changes (erpnext internal ports/proxy/backend, gitlab redis) were preserved. make test is green on the merged tree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
svc-storage-nfs-server gains a backend flavor selected by services.nfs-server.flavor (default "{{ 'ganesha' if RUNTIME in ['dev','act'] else 'kernel' }}"). The ganesha flavor installs nfs-ganesha + FSAL_VFS and renders /etc/ganesha/ganesha.conf (NFSv4, Pseudo == Path so clients mount the real paths unchanged, Root_Squash on the export base and No_Root_Squash on the state subdir), reloaded via systemctl restart. It runs in the dev/act DinD swarm-test, where the host kernel nfsd is not network-namespaced and a privileged node driving exportfs would wedge it into a D-state. The kernel flavor keeps nfs-kernel-server + exportfs -ra for production and every non-dev/act runtime. 01_core dispatches on the flavor, 03_prepare holds the shared export preparation, flavor/{kernel,ganesha}/main.yml hold the per-backend install/render/service steps, and handlers gains the ganesha restart.
docs/agents/action/iteration: role.md renamed to compose.md (Compose Loop); new swarm.md for the swarm deploy loop split out of workflow.md; workflow.md trimmed to GitHub Actions workflow iteration; all cross-references updated; the duplicated README gotcha removed; the NFS gotcha rewritten to the ganesha design.
Also drops a retired-flag history comment in group_vars/all/01_modes.yml and a redundant comment in the init selection script, and allows WebFetch(repology.org) used during ganesha packaging research.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace 'lookup(applications) | node_max_old_space_size(app, service)' with the 'lookup(node_max_old_space_size, app, service)' lookup form (it resolves the app config itself), dropping the boilerplate at all 4 call sites (nextcloud whiteboard, peertube x2, suitecrm). Extract the shared decimal to_bytes size parser into utils/sizes.py, used by the lookup and the peertube install_mem_limit filter (removing the cross-plugin import).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 512m right-sizing fell below the node_max_old_space_size filter minimum heap of 768 MB, which raised at compose generation and failed every nextcloud deploy variant (sys-svc-compose). 1g restores the floor (heap=768) with headroom. Verified peertube (8g) and suitecrm (2g) -- the other filter consumers -- are well above the floor.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
02a_manager_ops_on_host.yml set 'delegate_to: KEYCLOAK_HOST_NODE' directly on 'include_tasks: 02b_updates.yml', which Ansible rejects ("'delegate_to' is not a valid attribute for a TaskInclude"), so the playbook failed to parse (exit 4) for every SSO-consuming app, since keycloak is preloaded for them. The delegation is already supplied by the caller 02_manager_ops.yml, which includes 02a via apply: { delegate_to: KEYCLOAK_HOST_NODE }, so the include-level delegate_to was both redundant and invalid. Remove it; the sibling include 05_realm_import.yml already relies on the caller's apply, which is why it had no delegate_to.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
e791d2556 centralized the empty-services emit into base.yml.j2 (emit 'services: {}' when no local sidecar). But base.yml.j2 renders the shared 'services:' key that consuming roles nest their own service under (svc-db-mariadb, and every web-app whose backends are all shared). When base emitted 'services: {}' and the role appended ' <service>:' after the include, the result was 'services: {}' followed by a nested mapping -> invalid YAML ('services must be a mapping'), which failed ca-inject for mariadb on compose and the swarm pre-deploy 'compose build' for ~19 apps on swarm (more backends are central there, so more roles hit the empty-sidecar branch). Revert base to a bare 'services:' and restore discourse's local gate -- discourse is the only launcher-based role with no compose service, so it alone needs 'services: {}' when its sidecars are all shared. compose_ca's no-op-on-empty-services and the .ansible-lint-ignore 04_login entry from e791d2556 stay.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
base.yml.j2 now captures the sidecar service includes and emits 'services: {}' when none render, instead of a bare 'services:' which docker compose rejects ('services must be a mapping'). A launcher-based app with all backends shared (web-app-discourse) has no compose services because it runs its container via ./launcher, so its compose was previously invalid. web-app-discourse drops its local _has_local_sidecar gate and just includes base. compose_ca.py (ca-inject) no-ops -- writes an empty {'services': {}} override and exits 0 -- instead of dying with 'No services found after merging' when the merged compose has no services, since there is nothing to CA-inject and the downstream compose up/ps tolerate an empty stack; add test_main_empty_services_writes_noop_override. .ansible-lint-ignore: add the 04_login resolve_host_cid path_join include (a load-failure false-positive that strict ansible-lint profile:min flags). make test is green (lint, integration render, unit).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In swarm the standalone ./launcher container was created with only --network=DISCOURSE_NETWORK. With a shared redis/db the engine resolves DISCOURSE_REDIS_HOST/DISCOURSE_DB_HOST to the central aliases (redis-central, the postgres name) which live on the svc-db-redis / svc-db-postgres overlays, not the discourse network. So 'rake db:migrate' during './launcher rebuild' aborted with Redis::CannotConnectError 'redis://redis-central:6379' (resolving to 127.0.0.1:6379, refused). The compose-only db-bridge handler is skipped in swarm and 05_network's connect runs after the rebuild, so neither can help a migrate that runs during the rebuild. Attach the launcher to DISCOURSE_REDIS_NETWORK (when redis is shared) and DISCOURSE_PG_NETWORK (when the db is shared) at creation time, gated on the engine local lookup so embedded backends keep their sidecar on the discourse network. Multiple --network at docker run is honored on the targeted Docker 28/29.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The shared curl healthcheck template prepends a / before container_healthcheck, and COLLABORA_DISCOVERY_PATH already carries a leading slash, so the rendered probe was http://127.0.0.1:9980//hosting/discovery (double slash). COOLWSD (POCO HTTP) exact-matches the /hosting/discovery route and does not normalize //, so the probe returned non-2xx, curl -f failed its retries, swarm marked the task unhealthy and SIGTERM+rescheduled it on the healthcheck cadence (~3 min), so 'swarm | wait for collabora task to converge' never passed (failing collabora and the apps that preload it: bigbluebutton, nextcloud). Apply regex_replace('^/+', '') at the healthcheck assignment to render the single-slash URL. Fixed at the consumption site rather than the var because proxy.conf.j2 uses COLLABORA_DISCOVERY_PATH as the nginx 'location = /hosting/discovery', which requires the leading slash.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The openldap container is manager-pinned (meta/services.yml default_placement: manager), but the role's setup tasks run on every node. The six tasks that container-exec via OPENLDAP_CONTAINER_ADDRESS (which expands to a manager-only resolve-container-id subshell in swarm) therefore failed on the workers with 'resolve-container-id: must run on a Swarm manager node / invalid container name or ID: value is empty' (TASK 'Query data backend DN' on svc-bkp-container-2-local-swarm-wrk-01/02, rc=1). Add delegate_to: STACK_HOST + run_once: true to each of the six (01_core: bind check, module DN, query databases; 02_credentials: query DN, slappasswd, set RootDN/RootPW), mirroring the committed svc-db-mariadb pattern; run_once propagates the registered results to all hosts so the intervening all-host set_fact tasks still resolve. Compose-safe: STACK_HOST falls back to inventory_hostname and run_once is a no-op on a single host.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 'Resolve hosting node + container id' block was gated 'when: KEYCLOAK_HOST_NODE is not defined or none'. A block-level when propagates to the included resolve_host_cid tasks and is re-evaluated per task; resolve_host_cid sets <prefix>_HOST_NODE in its first task, so once KEYCLOAK_HOST_NODE was set the later container-id resolution and the KEYCLOAK_LOCAL_CID set_fact self-skipped, leaving KEYCLOAK_LOCAL_CID undefined. The 'Wait for keycloak container to be healthy' shell then failed finalizing _raw_params with 'KEYCLOAK_LOCAL_CID is undefined' across all 16 SSO-dependent swarm apps. Flatten the block and drop the self-mutating when so resolve_host_cid always runs and sets both KEYCLOAK_HOST_NODE and KEYCLOAK_LOCAL_CID; this also keeps both facts fresh after the realm-import scale reschedules keycloak.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the single utils/github/profile/role_summary.py script with a
focused CLI package split per responsibility (model, logparse, csvio,
render, github, sources). Output format is selectable via --format
(table default, plus markdown, csv, json); source is auto-detected as an
Ansible log, a role-runtimes CSV, or a GitHub Actions run/job URL (CSV
artifacts downloaded via gh).
The compose workflow now writes the CSV as a downloadable artifact and
appends the per-variant markdown to the step summary, and hard-fails
when the log is missing or carries no profile_roles data so a broken
metrics capture can no longer pass silently.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "Role runtimes" step read /tmp/ansible-<app>.log produced by a
docker cp out of the infinito container. That cp silently failed
(2>/dev/null || true) because the ansible log was not at the expected
in-container path, so role_summary reported "log not found".
The profile_roles table is already on the deploy step's stdout, so tee
it into the file role_summary reads and drop the broken cp (it also
raced the tee on the same path).
role_summary now segments the log by the matrix-deploy round/pass
markers and emits one per-role table per variant round; falls back to a
single combined table when no markers are present.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
_gh ran gh with check=True and capture_output=True, so on failure the gh error message (e.g. 'Not Found (HTTP 404)' when dispatching to a repo that lacks the workflow) was buried inside the CalledProcessError and only the command repr showed in the traceback. Check the return code explicitly, write gh's stderr through to our stderr, and raise SystemExit(returncode) so the real error is visible and the CLI exits with gh's non-zero code cleanly instead of a traceback. Keeps the explicit check=False for ruff PLW1510.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The local Path(__file__).resolve().parents[2] root computation tripped the project-root-import lint; use the canonical from utils import PROJECT_ROOT instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reclassify the nextcloud->onlyoffice bond from 1 to 0.5 and move onlyoffice out of the variant-0 host baseline into variant 4, so the document server is tested as a separate-host partner. Collabora stays the tightly-coupled (bond 1) default in the baseline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The resource right-sizing and variant rebalancing bring every variant well under the original 37GB/73GB, so lock in the tighter 32GB/64GB host budget.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Open a new CI runner bundle as soon as it would exceed INFINITO_VARIANT_BUNDLE_SIZE variants OR INFINITO_VARIANT_BUNDLE_MAX_STORAGE (default 400GB) cumulative min_storage, so storage-heavy variants are not stacked onto one runner that then runs too long. Every job now carries its explicit variant CSV in the name; empty only for roles without variants.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Disable talk_recording in variants where talk is off, and isolate gitlab into its own variant so it no longer shares a CI runner with discourse and mastodon.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collect and aggregate each service's min_storage (summed like mem) and surface it as a column in the detail, per-variant and per-role views, the JSON output and the filter/order fields.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
meta roles applications ressources gains two summary modes alongside the existing per-service detail table: (1) no --role: one row per role, each showing its heaviest variant (mem_reservation/mem_limit/pids_limit = the max per-variant sum, cpus = max), or the --variant N footprint of every role. (2) --role X without --variant: one row per variant of X (its sum/max). (3) --role X --variant N still renders the detailed per-service table - now the only way to reach it. --format/--order/--filter/--depth/--unshared apply in every mode. The table/JSON renderers are generalized (render_summary_text/json with a key_field) and a variant query accessor is added.
Useful for the CI runner-budget question: the heaviest-variant footprint per role/variant is now directly queryable. Tests cover all three modes (detail/all-roles/per-variant).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The no-direct-yaml lint (tests.lint.repository.yaml.test_no_direct_calls) flags any .py calling yaml.safe_load/safe_dump directly. The swarm_registry_sync regression test built its temp compose.yml with yaml.safe_dump; route it through utils.cache.yaml.dump_yaml instead and drop the now-unused yaml import. Includes the ruff line-wrap of the _sync helper signature. make test (incl test-lint) is green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When svc-db-qdrant is deployed as the shared/central qdrant its container is named 'qdrant'. flowise's compose template unconditionally defined its own 'qdrant' container, so docker compose up aborted with 'Error response from daemon: Conflict. The container name "/qdrant" is already in use'. Gate the embedded qdrant service block and flowise's depends_on on lookup('engine', 'qdrant', application_id, 'local'), so the embedded container is created only when flowise actually embeds qdrant; with a shared qdrant flowise reaches the central instance via the engine model and no longer collides.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the ten Keycloak update routines (client, mail, RBAC, LDAP, reCAPTCHA, userprofile, users) out of 02a_manager_ops_on_host.yml into 02b_updates.yml so 02a stays within the block-size limit; 02a now includes 02b. Delegation is preserved by the caller 02_manager_ops, which includes 02a with apply: delegate_to KEYCLOAK_HOST_NODE; that propagates to the nested update tasks. The include conditions (LDAP shared/disabled, reCAPTCHA, emailVerified, non-LDAP user provisioning) are moved verbatim.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On swarm the realm-import scales keycloak 0->N replicas; the rescheduled task is gone by the time 'Update Client settings' runs, so the kcadm exec hit 'Error response from daemon: No such container' and the in-module recovery gave up after DEAD_CID_MAX_RETRIES*DEAD_CID_RETRY_DELAY = 5*3 = 15s, far shorter than a keycloak restart + healthcheck + JVM warmup (~30-90s), so the resolver returned empty every attempt and the dead id was never replaced. Raise DEAD_CID_MAX_RETRIES to 40 (120s window) and add --filter health=healthy to KEYCLOAK_CID_RESOLVE_CMD so recovery only re-targets a ready replica, matching the login/realm-import resolvers. Compose is unaffected: the resolver command and current_cid are passed only when DEPLOYMENT_MODE == swarm, so the retry loop never engages and the larger constant is inert.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The synapse-admin task (tasks/flavor/ansible/05_proxy_wiring.yml) templates MARTIX_MDAD_CONTAINER_ADDRESS = lookup('container_address', application_id, 'matrix-mdad'), but meta/services.yml had no matrix-mdad key, so container_address aborted finalizing the shell cmd with "service 'matrix-mdad' missing in 'web-app-matrix' services config" (fatal: [localhost] in the compose job). The matrix-mdad DiD runner is rendered by the role's own flavor compose fragment and was never registered in meta. Add a name-only SPOT entry (taiga manager precedent): compose returns the bare name 'matrix-mdad' (= the fragment's container_name), swarm resolves <stack>_matrix-mdad. The stub has no enabled/image so the service-loader and the dynamic-flags/playwright lints ignore it, and test_resources only validates the primary entity.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The standalone Discourse bootstrap (./launcher rebuild) attaches --network={{ DISCOURSE_NETWORK }} (=entity_name 'discourse'), but that overlay was never created in swarm, so the rebuild aborted on every node with 'failed to set up container networking: Could not attach to network discourse: rpc error: NotFound desc = network discourse not found'. Create the role network before the launcher via sys-svc-compose network/create (overlay attachable, run_once + delegate STACK_HOST in swarm; bridge in compose), move the launcher tasks to 04_launcher gated on IS_STACK_HOST, and split the recreate-discourse handlers into stack_host.yml fired by a single IS_STACK_HOST-gated 'Handle stack host' handler that listens on 'recreate discourse', matching the established sys-service/sys-svc-sshd/svc-prx-openresty handler-include pattern. Gate the shared-db network connect/disconnect (05_network) on IS_STACK_HOST too.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ansible 2.19 parsed the task-level 'stdin:' written as a sibling of the free-form 'shell: >' as a second action module and aborted the play with '[ERROR]: conflicting action statements: shell, stdin'. Move stdin under 'args:' in 03_debug, 04_patch and 07_oidc_plugin so it is a shell module parameter, matching the canonical form already used by keycloak/mailu/friendica/decidim. This is a mode-independent parse error (failed compose and swarm).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
