BBB runs the external bigbluebutton/docker project (host networking,
node-local stateful stack) and cannot be a docker stack deploy service.
Add a per-role compose_mode override so it deploys via node-local
compose up on STACK_HOST even inside a swarm cluster:
- sys-svc-compose: compose_mode_force (per-role override) + computed
compose_mode (SPOT) in defaults; up.yml + handlers route on
compose_mode instead of DEPLOYMENT_MODE.
- container_address: honour compose_mode_force so a node-local role's
exec targets its compose container (bare name), not a swarm service.
- web-app-bigbluebutton: compose_mode_force=compose, ungate compose
generation for swarm, flush handlers in-scope, and add
services.bigbluebutton.ports.internal.http for the swarm proxy upstream.
- web-app-matrix: add services.element.ports.internal.http (swarm proxy).
- test_variant_bundles: pin compose mode in the main test so it is not
polluted by another test leaking INFINITO_DEPLOY_MODE=swarm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
database_query is a module: it opens query_file on the host it runs on. In swarm the task is delegated to the stack host, which has no repo checkout at role_path, so query_file is not found. A new action plugin (same name auto-wraps the module) reads query_file on the controller and passes the SQL as 'query', so every role's query_file: works in both compose and swarm with no call-site changes. Reverts the per-call-site fider workaround in favour of this root fix, which also covers mattermost, odoo, openproject and pixelfed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Floor-to-1 now applies only to the topology default (no override term). An explicit service_replicas=0 emits 'replicas: 0' (service created but unscheduled, to be scaled up by a later task) instead of being bumped to 1. Empty/missing inventory group on the default path still floors to 1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In swarm an app task can be scheduled on a worker, but the role exec'd
`container exec $(resolve-container-id ...)` on the manager -> "No such
container". Resolve the hosting node + node-local container id and delegate
each exec to that node.
- database.py: add a `service_name` want; resolve_host_cid/wait_converged
gain `*_service_name` for cross-stack services (mailu's shared mariadb)
- mailu wait-for-mariadb (DB probe + admin), mastodon db:migrate,
gitea migrate/admin-create, friendica manager-ops/patch/addons/bootstrap
(~13 exec sites) switch to resolve_host_cid + delegate_to
Compose-safe: resolve_host_cid degrades to HOST_NODE=inventory_hostname
(delegate is a no-op) + LOCAL_CID=bare compose container name, matching the
8 roles already using this pattern.
Note: friendica also needs service_replicas=1 (next bundle) so its resolved
container id does not churn across replicas.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve the 80 open CodeQL/code-scanning alerts on this branch.
Security-sensitive fixes:
- cli/administration/deploy/development: break the unsafe import cycle by
extracting the env/arg helpers into a leaf module `env.py`, giving the
load-time DAG compose -> common -> deps -> env.
- web-app-bluesky login-broker server.js: prototype-pollution guard on
cookie parsing, CR/LF + control-char log sanitisation, HTML-escaped
exception text, and a ReDoS-bounded handle regex.
- web-app-baserow SSO: harden the open-redirect `next` handling.
- network diagnose probes: require TLS >= 1.2.
- actionlint installer: validate tar members against path traversal.
- decidim/apt-purge lint regexes: rewrite to avoid catastrophic backtracking.
- 6x Playwright URL regexes anchored; crs-k/stale-branches pinned to a commit SHA.
Mechanical cleanups: unused imports/globals/locals, empty-except comments,
import-and-import-from, implicit-string-concat, multiple-definition,
unreachable/useless-assignment.
False positives are intentionally left in place (Django settings patches,
Odoo __manifest__, side-effect imports, cross-module symbols).
Recurrence guards:
- import-linter contract (pyproject + tests/lint) locks the dev-deploy
package acyclic; a new cycle now fails CI instead of only CodeQL.
- new lint test requires every bandit (S###) `noqa` to carry a justification.
- BLE001 (blind-except) added to ruff as a ratchet (tests/roles exempt,
current production offenders grandfathered).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Conflicts resolved:
- .claude/.gitignore: union of ignore patterns
- plugins/lookup/rbac_group_path.py: drop both unused conflicting imports
(body fetches apps via lookup_loader)
- roles/sys-svc-compose/handlers/main.yml: keep HEAD's refactor (compose-up
handlers live in compose.yml, included per DEPLOYMENT_MODE)
- utils/cache/base.py: keep HEAD's merge_with_defaults/FILE_TOKENS imports
Post-merge fixes for main's newly-enabled lints:
- pyproject.toml: grandfather 5 files under the BLE001 ratchet
- justify 6 pre-existing security # noqa markers (S108/S311/S701)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce plugins/lookup/path_absolute.py as the single source for the
repo-root anchor: lookup('path_absolute', 'roles/...') returns an absolute
path under playbook_dir, joining every term. Replace all hand-built
playbook_dir path constructions across .yml/.yaml and .j2 templates with it:
[playbook_dir, '<rel>'] | path_join -> lookup('path_absolute', '<rel>')
playbook_dir ~ '/<rel>' -> lookup('path_absolute', '<rel>')
{{ playbook_dir }}/<rel> -> lookup('path_absolute', '<rel>')
Unrelated repo_path symbols in cli/meta/roles/applications/resolution and the
single bare discover_playwright_roles(playbook_dir) call are left untouched.
Add tests/lint/ansible/test_playbook_dir_via_lookup.py: fails the build if any
of the three construction forms reappears in .yml/.yaml/.j2, pointing callers
at the lookup. Unit test covers the plugin.
Touching these files pulled their pre-existing narration comments into
test_comments_valid scope; clean them (delete restatement/banners, keep real
trip-wires as Workaround/Upstream/Invariant markers). Drop caution, important
and pitfall from the comment-lint marker vocabulary and remove the comments
that used them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The matomo bootstrap re-injects early-rendered web-svc consumers (such as
web-svc-logout) after obtaining the admin token, but in swarm this never ran:
roles_with_service filters by the manager's group_names, which omits
worker-placed web-svc services, so the re-injection loop was empty and the
logout page shipped without the matomo tracker.
- roles_with_service: add a scope kwarg; scope='all' bypasses the per-host
group_names filter (the merged applications config is already
deployment-scoped, so iterating all of it equals the deployment's consumers).
- 05_bootstrap: enumerate consumers with scope='all', stat each compose project
dir, and re-inject only deployed ones. Avoids a fatal chdir into an undeployed
consumer's missing /opt/compose/<entity>/ that scope='all' would otherwise hit.
- sys-front-inj-matomo: resolve matomo_effective_token (override else
lookup('users')); the disable-gate and the SitesManager getSitesId/addSite
calls use it, so the re-injection passes the fresh bootstrap token and bypasses
the play-start-stale controller-side users cache that left matomo disabled.
scope defaults to 'host' and the override defaults to empty, so every existing
caller and the per-consumer injection path are unchanged. Validated green in
both compose and swarm round 1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract the csp, domains, and networks blocks out of roles/*/meta/server.yml
into dedicated meta files, each mapping to its own top-level applications key
via the recursive meta-walk:
csp: -> meta/csp.yml -> applications.<app>.csp
domains: -> meta/domains.yml -> applications.<app>.domains
networks: -> meta/networks.yml -> applications.<app>.networks
Residual nginx/proxy keys (status_codes, client_max_body_size, locations,
ip6_enabled, config_upstream_url) stay in server.yml; an emptied server.yml is
deleted. 98 server.yml -> 249 new files, 74 deleted.
- Consumers repointed applications.<app>.server.{csp,domains,networks} ->
applications.<app>.{csp,domains,networks} across templates, vars, tasks,
lookups and filters.
- Infra readers migrated: canonical_domains_map, application_domain_index,
build_ldap_role_entries, service-registry overlay discovery,
roles_with_service, and the file-based readers utils/domains/list.py +
utils/meta/scan.py (read the now-deleted server.yml directly).
- Detection + mapping: _META_TOPICS gains csp/domains/networks;
ROLE_FILE_META_{CSP,DOMAINS,NETWORKS} constants + ROLE_FILES entries.
- New self-expiring lint guard (test_server_meta_no_split_fields) fails if a
server.yml still carries a split field; auto-fails from 2026-12-28 as a
delete reminder.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Consumer lookup plugins now reach the merged config through the Ansible
lookup loader instead of importing get_merged_applications /
get_merged_domains from utils.cache. The merge stays one cached
implementation surfaced via the applications/config/domains/users lookups;
consumers call lookup_loader.get(...).run() (the compose_networks idiom).
- get_merged_applications walks the whole meta/ tree recursively
(meta/**/*.yml to applications.<app>.<path>, addons and any depth) and
folds the raw meta/volumes.yml subtree onto applications.<app>.volumes
after the templar render so the Jinja source: strings stay unrendered.
- 31 consumer lookups rerouted; providers (applications/config/
applications_current_play/users/domains) and the raw-volume accessors
(volume/nginx) keep the cache import behind # nocheck: lookup-cache-import.
- New lint test_no_cache_imports_in_lookups enforces the contract.
- nfs_flavor is now the flavor SPOT (svc-storage-nfs-server + nfs_client_src
resolve via lookup('nfs_flavor')); removed the unused STORAGE_NFS_FLAVOR
group_var. Extracted extract_default_credentials to
utils/cache/schema_credentials.py to keep applications.py under the cap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
self._templar.template('{{ var }}') no-ops on untrusted strings in ansible 2.19+, so the nfs_* lookups emitted literal Jinja into /etc/exports and exportfs failed (the fsid=0 export never came up). The base lookups now read resolved inputs from 'variables'. nfs_flavor/nfs_client_src read the flavor from the applications variable (the variant-baked config provisioning bakes under applications:), not a re-merge of the role files via get_merged_applications. Adds a real-Templar regression test that fails on the old code.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Simpleicons: the reachability probe ran inside the controller-side Jinja filter, which has no route to the node's published port in swarm, so every card silently fell back to Font Awesome. Move the probe to a STACK_HOST-delegated uri_retry step and make the filter pure assembly. Compose output is byte-identical.
Tile reachability: add a per-card services.dashboard.iframe capability (default = enabled) so apps whose SPA top-redirects out of an embed (Keycloak admin console, iframe:false) open in a new tab; roles_with_service emits the flag into the target manifest and the spec branches on it. For iframe:false targets the spec also asserts the HTTP layer permits framing (no blocking X-Frame-Options, CSP frame-ancestors allows the dashboard) so the flag can never mask a real header block. Fix the Tab-button locator and reveal it from fullscreen before asserting.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve the 80 open CodeQL/code-scanning alerts on this branch.
Security-sensitive fixes:
- cli/administration/deploy/development: break the unsafe import cycle by
extracting the env/arg helpers into a leaf module `env.py`, giving the
load-time DAG compose -> common -> deps -> env.
- web-app-bluesky login-broker server.js: prototype-pollution guard on
cookie parsing, CR/LF + control-char log sanitisation, HTML-escaped
exception text, and a ReDoS-bounded handle regex.
- web-app-baserow SSO: harden the open-redirect `next` handling.
- network diagnose probes: require TLS >= 1.2.
- actionlint installer: validate tar members against path traversal.
- decidim/apt-purge lint regexes: rewrite to avoid catastrophic backtracking.
- 6x Playwright URL regexes anchored; crs-k/stale-branches pinned to a commit SHA.
Mechanical cleanups: unused imports/globals/locals, empty-except comments,
import-and-import-from, implicit-string-concat, multiple-definition,
unreachable/useless-assignment.
False positives are intentionally left in place (Django settings patches,
Odoo __manifest__, side-effect imports, cross-module symbols).
Recurrence guards:
- import-linter contract (pyproject + tests/lint) locks the dev-deploy
package acyclic; a new cycle now fails CI instead of only CodeQL.
- new lint test requires every bandit (S###) `noqa` to carry a justification.
- BLE001 (blind-except) added to ruff as a ratchet (tests/roles exempt,
current production offenders grandfathered).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause of the kernel-flavor swarm ENOENT: no export carried fsid=0, so the NFSv4 mount of the nested /srv/nfs/infinito-state sub-export had no pseudo-root anchor and the server returned NFS4ERR_NOENT on the PUTROOTFH+LOOKUP walk. crossmnt could not fix it (it only crosses a boundary mid-walk, never sets the starting filehandle). Kernel now exports the state dir itself at fsid=0 and clients mount the bare v4 root ':/' (zero LOOKUPs). ganesha keeps Pseudo=Path and v3 mounts the full path. The self-bind is kept so fsid=0 rides a real mount boundary (stable st_dev) under overlayfs DinD.
The NFS derivations move to per-value lookup plugins (nfs_flavor, nfs_client_src, nfs_fstype, nfs_mount_opts, nfs_state_path) backed by pure helpers in utils/storage/nfs.py with unit tests; group_vars STORAGE_NFS_* become thin lookup wrappers. The dev/act->ganesha flavor rule stays the single SPOT in the role config; nfs_flavor reads the merged value so client and server agree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In swarm the sibling exporters (alertmanager, cadvisor, node-exporter, blackbox) and the healthz backend are unreachable on the compose container_name / host.docker.internal / 127.0.0.1 addresses, so prometheus marked every target down and /healthz/ready returned 502. Add a scrape_target lookup emitting <host>:<port> per mode (swarm tasks.<entity>_<service>:<internal_port>, compose container_name/compose_host), reading the internal port from services.<svc>.ports.internal; declare those ports on the four exporters; route both healthz probes through the proxy_pass lookup via one set variable. Compose output stays byte-identical.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In swarm, OmniAuth/OpenSSL (decidim OIDC discovery) and Node (bluesky-broker -> PDS) could not trust Keycloak's internal self-signed CA: docker stack deploy never received compose.ca.override.yml, and the override's $$-escaped entrypoint wrapper is swarm-incompatible (stack deploy does not un-escape it). Add a --no-wrapper override mode (env + CA mount only: SSL_CERT_FILE/CURL_CA_BUNDLE/NODE_EXTRA_CA_CERTS) generated on the stack host for self-signed apps, and build the stack-config/stack-deploy --compose-file list via compose_file_args (new --compose-file flag kwarg) instead of hardcoding compose.yml -- which also stops swarm silently dropping a role's own compose.override.yml. Compose untouched: self-signed-gated, no-wrapper is swarm-only, lookup defaults to -f.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace 'lookup(applications) | node_max_old_space_size(app, service)' with the 'lookup(node_max_old_space_size, app, service)' lookup form (it resolves the app config itself), dropping the boilerplate at all 4 call sites (nextcloud whiteboard, peertube x2, suitecrm). Extract the shared decimal to_bytes size parser into utils/sizes.py, used by the lookup and the peertube install_mem_limit filter (removing the cross-plugin import).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
container_ports reads application_id from the play vars, where sys-stk-full leaves it as the include var '{{ sys_stk_full_application_id }}'. _as_str() stripped its TrustedAsTemplate tag, so templar.template() silently refused it on ansible-core >=2.19, leaving the raw string and failing every shared-service compose/swarm render ('Application ID {{ ... }} not found'). Re-trust via trust_as_template, mirroring plugins/action/stack_host_template.py, and import it directly (drop the dead <2.19 ImportError fallback).
Also fix a stale import in utils/roles/applications/config.py: AnsibleUndefined moved from ansible.utils.unsafe_proxy to ansible.template in ansible 2.20, so the old ImportError fallback always used a no-op stub and isinstance(obj, AnsibleUndefined) silently never matched. Import it directly from ansible.template.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When sys-stk-full renders a shared service (web-svc-logout, web-svc-collabora)
it restores application_id="{{ sys_stk_full_application_id }}"; that reaches the
container_ports lookup as the raw, un-templated string (the lookup reads it from
`variables`, not as a templated positional arg), so it raised "Application ID
'{{ sys_stk_full_application_id }}' not found" and broke the swarm deploy of
every app pulling in those shared services. Template it through self._templar,
the way container_depends_on already resolves the vars it reads. Add a
regression test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Take the host IP as an optional third element of each term
([service, protocol, host_ip]) instead of an ip= kwarg, and build the whole
ports: block from declared ports.local/public (host) + ports.internal
(container). Convert every compose template across 84 roles, folding the
remaining raw var/numeric/specific-ip mappings (including mailu's per-interface
mail ports bound to MAILU_IP4_PUBLIC). Multi-term calls render one term per line.
Add the compose-ports-must-use-container-ports lint: it forbids any raw
"<host>:<published>:<container>" publish item under a ports: block and requires
every term to declare ports.internal. Remove
tests/lint/roles/test_swarm_internal_ports.py, now subsumed.
Fix a container_depends_on rendering regression. The lookup self-indented
(default indent=4), so the added 4-space placement + | indent(4) triple-indented
the block. Flip the lookup default to column 0 (consistent with
container_networks), make every call use the 4-space + | indent(4) form, and add
a lint enforcing it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A swarm docker stack deploy env-file value must be a single physical line; the
multi-line GITLAB_OMNIBUS_CONFIG was read as separate whitespace-named env
entries and aborted the deploy. Add a depth-aware ruby_single_line filter that
collapses the Ruby program to one line (statement-boundary newlines at bracket
depth 0 become ';', newlines inside an open bracket or a string become a space
so hash/array literals like the OIDC omniauth_providers stay valid) and apply it
on the swarm branch only; compose keeps the multi-line value unchanged.
Covered by tests/unit/plugins/filter/test_ruby.py (10 cases incl. a realistic
omnibus config, escaped quotes and nested literals).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
lookup('volume', role) read the lazily-populated canonical-volumes registry directly, so a lookup running before any consumer built that role (nfs_prep's pre-create loop for an early unpinned engine such as ollama) saw an empty registry and skipped the subdir; the swarm bind mount then failed with 'no such file or directory'. Force get_application_defaults() on a miss, guarded so the populated hot path skips the deepcopy. Adds two guard tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Widen the openresty fan-in overlay /28 -> /24 (the /28 exhausted VIPs under mailu's services, deadlocking the stack). Pre-create NFS backing dirs for every volume of unpinned roles (placement decides, not the legacy per-volume nfs flag). Make email 'external' cluster-wide so swarm workers relay through the central mailu instead of a localhost root sender. Constructor: auto-place svc-prx-openresty, gate out unconsumed central engines, and load the svc-dns role group.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Foundation for the central svc-db-*/svc-dns-* engine roles: the engine lookup and engines descriptor, per-consumer engine credentials in the inventory manager, shared-net overlay rendering, container_depends_on, the sys-svc-engine provisioning dispatcher, and the compose_volumes NFS-placement split. base.yml.j2 gates the embedded redis sidecar on lookup('engine','redis',application_id,'local').
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A bridged addon's <ID>_ADDON_ENABLED flag is now true only when one of its
declared bridge partners is in the deployed playwright role set
(TEST_E2E_PLAYWRIGHT_APPS): integration specs skip cleanly when the partner is
not deployed and assert the real coupling when it is. App-addons without
bridges are unaffected; an unresolvable set falls back to the prior
enabled-AND-required behaviour (no regression).
Adds TestBridgeDeploymentGating (7 cases): partner-deployed, not-deployed,
multi-bridge any-match, no-bridge passthrough, disabled stays false, fallback
without the set, and string-set splitting.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add plugins/lookup/playwright_workers.py computing a conservative parallel-worker count (CPU/RAM-derived, app-under-test concurrency-capped, reduced on CI) and wire test_e2e_playwright_workers + test_e2e_playwright_fully_parallel + test_e2e_playwright_test_timeout into playwright.config.js (read from env) and 02_run_one.yml (passed via -e). fullyParallel stays false so parallelism is file-level only and intra-file ordering assumptions stay safe. Raise run_timeout to 3600 for addon-heavy roles (nextcloud ~50 specs).
Split the role's variables by Ansible precedence and intent: derived/fixed constants live in vars/main.yml (UPPERCASE, high precedence, not operator-tuned), operator-tunable knobs live in defaults/main.yml (lowercase, overridable). VARIANT_INDEX stays UPPERCASE with a single definition. Update references across tasks/templates/inventory/docs accordingly. Includes a unit test for playwright_workers and an autoformat pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the per-template Jinja loop (which mis-unpacked dict2items and rendered every flag as KEY_ADDON_ENABLED=false, silently skipping every addon spec) with a single lookup('addon_env_flags', application_id). The plugin is the single source of truth for the <ADDON_ID>_ADDON_ENABLED flags consumed by tests/addon-gating.js: env-key derivation matches addon-gating.js exactly, and a flag is true only when the addon is both enabled AND required (required:false addons skip). Switch all six role templates to the lookup, extend the env-file schema lint to recognise whole-line {{ ... }} expressions, and add the plugin unit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bare policies like no/off render as 'restart: no' which YAML parses as a boolean, so 'docker compose config' rejects the file. Quote the scalar ('restart: "no"') to keep it a string.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The net refactor (366dc7b70/cb27ff375) migrated every shared-net provider (postgres/mariadb/openldap/ollama/openresty/keycloak) to the overlay registry but missed seaweedfs and minio, so an object-store consumer like baserow never joined the provider's docker network and its S3 client hit ECONNREFUSED on seaweedfs:8333 (the intended direct data-plane path).
Add networks.overlay to both engines so consumers attach, and complete the multi-service shared-net-provider contract: (1) render.py emits a bare auto-IPAM project default for an own shared_net provider (no name, no subnet) so the entity-named external net created by network/create.yml is not re-requested -- this caused 'invalid pool request: Pool overlaps' on the first attempt. (2) a provider_self_alias flag lets the provider's sidecars (proxy/sso-proxy) JOIN the shared net WITHOUT publishing the <entity> alias, so the name resolves only to the main container and baserow's :8333 never round-robins onto a port-less sidecar. (3) seaweedfs/minio 01_core pre-create the external <entity> net via network/create.yml (mirroring web-app-prometheus) so compose adopts it as the project default instead of building <entity>_default.
Verified: baserow compose e2e test-seaweedfs.js passes (upload stored in the bucket, 3.9s); new TestOwnSharedNetProvider locks the contract; full make test green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
stack_host_copy is the copy: analogue of stack_host_template: it folds the IS_STACK_HOST gate into the module so callers do not silently fail on hosts whose destination tree is owned only by the stack host. In swarm an app's compose / build-context directory (/opt/compose/<app>) exists only on the manager, so a bare copy: into directories.instance also ran on workers and aborted the play with 'Destination directory does not exist'.
Migrate the six compose-dir copy tasks (baserow, jenkins, bookwyrm, postmarks, yourls, openldap) to stack_host_copy and drop the now-redundant when: IS_STACK_HOST. A new drift-guard lint test (test_stack_host_copy_migration) flags any future bare copy: into the compose dir, and a unit test covers the wrapper.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The server args were duplicated across the standalone provider stack (compose.yml.j2) and the local sidecar (service.yml.j2), so the -ip fix had to be applied in two places. They now live in one filter (plugins/filter/seaweedfs.py); both templates keep command: inline and render the args via it, appending -s3.config only when the caller passes the mounted s3.json path (the sidecar omits it). Adds a unit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add group_vars/all/18_api.yml defining the global API mapping (cloudflare, telegram, github, google, jira) and plugins/lookup/api.py — the single sanctioned reader that resolves a dotted 'provider.key' path against it and raises on unknown keys. Routing every proprietary/external credential access through one lookup keeps a single point to change how credentials are sourced. Includes the unit test covering nested resolution, subtree access, error paths, and templar handling.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reconcile main's object-store feature and pre-swarm deploy tasks with the branch's compose/networks/volumes SPOT and swarm lints.
objstore (seaweedfs/minio): standalone compose + embedded service templates migrated to the container_image/container_volumes/container_networks/compose_volumes/compose_networks/depends_on lookups; meta/volumes.yml on the canonical dict schema with s3.json and nginx.conf as type:config; embedded services made swarm-compatible via compose_only/compose_restart.
deploy tasks (erpnext, opentalk, seaweedfs, shopware, snipe-it, wordpress, yourls): container exec routed through the container_address resolver and gated on IS_STACK_HOST, shell module plus stack_host_template, no chdir or compose-verb forms.
Also removed orphaned vars, restored MAILU_INSTANCE_DIR, freed the seaweedfs subnet (192.168.206.0/24), stripped literal quotes from the nextcloud/bookwyrm env templates, and routed main's objstore compose_volumes tests through dir_var_lib.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
roles_with_service returns only consumer roles that are themselves deployed (in group_names) and excludes the service's own provider, so the logout/css/dashboard/matomo/prometheus injection specs no longer target undeployed or surface-less roles in a partial single-app deployment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Centralise the nginx proxy_pass directive in one source of truth (utils/networks/proxy.py + plugins/lookup/proxy_pass.py). Compose keeps the literal 127.0.0.1:<local> upstream; swarm resolves the service over the shared overlay at request time (variable upstream + http resolver) so a rescheduling backend never blocks openresty startup. Converts the html/media/upload/ws location partials, the keycloak oauth2 and login-iframe endpoints, and the logout endpoint to the lookup; adds the swarm resolver and the front-proxy service_key plumbing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The oauth2-proxy sidecar set cookie_domains to the app's primary domain only. A role served under several sibling subdomains (e.g. SeaweedFS filer/master/api under one base) therefore could not read its session cookie on the non-primary hosts, looping the auth chain. Add the common_dns_suffix filter and use it so cookie_domains becomes the longest DNS suffix shared by all of the app's domains: a single-domain app keeps its exact host (unchanged), a multi-domain app collapses to the shared parent so one session cookie spans every subdomain.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Switch penpot, nextcloud, gitea, mattermost, openwebui, funkwhale, baserow, openproject, pixelfed, mastodon, bookwyrm, listmonk, peertube and opentalk to the engine-agnostic objstore lookup: the minio service entries are pinned off, a seaweedfs service (gated on the provider being in group_names, public for the media apps) replaces them, and the matrix variants pin the new flag. OpenTalk keeps object storage mandatory through an embedded fallback and provisions its bucket via weed shell.
Add objstore lookup SPOT keys consumed by these roles: endpoint (host:port), public_domain, public_bucket (public S3 host plus bucket) and public_url (scheme plus public_bucket, scheme derived from the provider TLS state). This removes the repeated lookup('config', lookup('objstore', ...), 'server.domains.canonical.api') / host:port concatenations from the consumer templates. Listmonk settings render their JSON values via the to_json filter.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a database-style object-store abstraction: the objstore lookup resolves the active engine (seaweedfs|minio) per consumer and yields a uniform S3 payload, objstore_secret_key is auto-generated like database_password, sys-svc-objstore delegates per-consumer provisioning to the resolved provider's objstore_init entrypoint (lazy-loading the provider when it was not preloaded), and compose/network/volume plumbing supports both a shared central instance and an embedded storage-only instance per consumer.
Add the web-app-seaweedfs provider role: Filer and Master UIs behind the admin-only oauth2-proxy via a role-local nginx and proxy.conf.j2 (S3 API domain stays ungated, admin UIs deny with 403 when SSO is disabled), per-consumer S3 identities from s3.json, idempotent bucket provisioning via weed shell, a /24 on the central-provider subnet range, Playwright coverage for admin/biber/guest plus consumer bucket existence, and unit tests for the objstore lookup and the per-engine compose volumes.
Mark web-app-minio lifecycle: unsupported, move its mc client image into a dedicated services.client entry, align it with the objstore credential contract, and make its consumer provisioning idempotent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Convert the two registry-driven networks templates into pure-Python
rendering functions plus thin Ansible lookup plugins:
roles/sys-svc-compose/templates/networks.yml.j2 -> {{ lookup('compose_networks') }}
roles/sys-svc-container/templates/networks.yml.j2 -> {{ lookup('container_networks') | indent(4) }}
The rendering logic lives in utils/networks/render.py as pure-Python
functions that take a registry + closure-style config/database lookups,
making the logic directly unit-testable without a Jinja stub harness.
The lookup plugins are thin wrappers that resolve application_id,
DEPLOYMENT_MODE, and swarm.network.encryption from the templar, build
the service_registry, and forward to the render functions.
Migrate 233 include lines across 85 production templates. Convert
web-app-matrix's hand-coded MDAD networks block (default + conditional
postgres + conditional openldap) to the generic lookup; this adds an
openresty attachment when services.sso.enabled (harmless extra
interface, pinned in test_matrix_mdad_consumer_attaches_postgres_ldap_sso).
Mark local-DB / redis sidecars (sys-svc-rdbms/templates/services/,
svc-db-redis/templates/service.yml.j2) with `# nocheck: networks-literal`
because they intentionally attach to default-only.
Extend the lint to scan ALL roles/*/templates/**/*.yml.j2 (not only
compose.yml.j2) and to forbid the legacy include form; the marker
above is the documented escape hatch.
Coverage:
- tests/unit/utils/networks/test_render.py (30 cases incl. encrypted=false literal pin)
- tests/unit/plugins/lookup/test_compose_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/unit/plugins/lookup/test_container_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/integration/infrastructure/compose/test_networks_render.py (10 snapshot scenarios)
- tests/integration/infrastructure/compose/test_network_includes.py (3 lint rules)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The container_volumes lookup returns a YAML string starting with
`volumes:` at column 0. Templates consume it via
`{{ lookup('container_volumes', application_id, '<svc>') | indent(4) }}`
in 129 places.
Jinja's `indent(width)` only indents lines AFTER the first by
default (indentfirst=False), so the rendered output kept `volumes:`
at column 0 and the rest at column 4 — producing a top-level
`volumes:` map in the middle of a service block, which broke the
compose.yml structure (the following `ports:` / `networks:` keys
ended up inside the volumes list and `compose build` failed with
`did not find expected '-' indicator`).
Prepending a leading newline to the lookup's return value lets
`indent(4)` indent every meaningful line including the first one,
without forcing 129 templates to switch to `indent(4, True)` (which
would also need a separate audit of which call-sites already supply
their own leading whitespace).
Existing filter-side unit tests in
tests/unit/plugins/filter/test_container_volumes.py exercise the
underlying `container_volumes()` function which is unaffected — the
newline is added only at the lookup boundary.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
container_address + container_service are mode-aware lookups that
build a docker-exec-safe address from `application_id + service_key`.
Two adjustments to align them with how the rest of the project uses
service keys:
* Swarm-mode return value now uses the lookup's `service_key`
argument directly as the swarm service-suffix instead of
`services.<key>.name`. This matches the documented contract in
`roles/sys-svc-compose/files/resolve_container_id.sh` (the helper
composes `<stack>_<SERVICE_KEY>`) and what `docker stack deploy`
names a service after (the compose-yaml service key, not the
compose-side `container_name:` override).
* Compose-mode `_resolve_bare_name` falls back to
`services[entity_name]` when `service_key == 'application'` and
the meta has no top-level `application:` entry. 26 roles use
`application:` as their compose-yaml service key while keying
their meta/services.yml under the entity name (baserow, listmonk,
wordpress, mobilizon, etc.). Without the fallback, every consumer
of `container_address(application_id, 'application')` raised
`service 'application' missing in '<role>' services config` at
task-runtime. The fallback is intentionally narrow (only the
literal 'application' key) so an actual typo in another
service_key still raises clearly.
Unit-test contract update: test_container_address.py replaces
`test_swarm_mode_uses_resolved_service_name` with
`test_swarm_mode_uses_compose_yaml_service_key`, asserting the
resolver receives the service_key ('mattermost') rather than the
`services.<key>.name` value ('custom-mm-name'). Existing
`test_unknown_service_raises` still passes because the fallback only
fires for service_key='application'.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add plugins/lookup/container_image.py: returns the full `image: "<ref>"`
line so templates drop the redundant image: prefix and surrounding
quotes. Supports `custom=True` to auto-construct `<entity_name>_custom`
for locally-built images, plus explicit `image=`/`version=` overrides
for exotic cases (taiga back-auth/front-oidc variants, decidim infinito
suffix, etc.).
* 25 unit tests for the lookup cover compose vs swarm prefix, custom
flag, overrides, tag_only=True, missing version warnings.
* New strict lint (tests/lint/.../test_no_raw_image_in_compose_templates)
fails on any raw `image:` line in roles/*/templates/*.j2 unless
nocheck-suppressed. Legacy `image: "{{ lookup(...) }}"` form no
longer accepted.
* Migrate 159 image refs across ~80 roles to the standalone
`{{ lookup('container_image', application_id, '<service_key>') }}`
form. 26 roles use `custom=True` (mariadb, postgres, openldap,
jenkins, jira, gitlab, nextcloud, mediawiki, shopware, wordpress,
odoo, joomla, openproject, bookwyrm, espocrm, hugo, littlejs,
fediwall, roulette-wheel, openresty, confluence and more).
* Fix svc-db-redis application_id leak: service.yml.j2 is included
into consuming roles where application_id evaluates to the outer
role, so the image lookup needs the hardcoded 'svc-db-redis'.
* Integration tests (test_templates_require_image,
test_build_requires_image) recognise the standalone lookup as an
implicit `image:` at the same indent.
* Mediawiki swarm tasks: add explicit `:{{ MEDIAWIKI_VERSION }}` to
the registry-prefixed image string because the custom var is now
bare.
Only 3 legitimate exotic nocheck-suppressed image lines remain:
matrix-mdad runner (nested services.matrix.runner.image path),
matrix mautrix bridges (per-iteration dynamic) and svc-registry-docker
(chicken-and-egg self-reference).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two related fixes in the volume-rendering filters, both surfaced by
`docker stack deploy` failing on the rendered compose.yml.
1) Render Jinja in mount source/target via the lookup-supplied
render_jinja closure. `container_volumes` previously emitted the
`meta/volumes.yml` `source:` / `mounts[].target:` strings unchanged;
in swarm those flowed verbatim into compose.yml as
`{{ lookup('nginx', ...) }}:/usr/local/openresty/nginx/conf/conf.d`
and broke `docker stack deploy expanding mount target` with
`function "lookup" not defined`. `compose_volumes` had the analogous
bug for top-level `configs:` / `secrets:` source fields.
Add `_maybe_render` to `compose_volumes` and the matching
try/except-AnsibleUndefinedVariable render block to `container_volumes`
for bind source + mount target, raising `AnsibleFilterError` with the
offending expression so failures stay diagnosable.
2) Coerce config/secret `mode:` to int. The YAML/J2 idiom carries `mode`
as a string ("0440"); `docker stack deploy` rejects that as
"services.X.configs.N.mode must be a number". Add `_coerce_mode_int`
(octal-string -> int, with bool + empty + non-octal guards) and
convert before emission. Update the per-service config/secret unit
tests to expect `0o440` / `0o400` instead of the legacy strings.
Validated by the t5 -> t8 act-swarm-zombie progression (mode-int gate
cleared at t6; bind source/target + config/secret source render cleared
at t7 -> t8, lifting ok=452 to ok=798).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`get_canonical_volumes()` returns `meta/volumes.yml` `source:` / `target:`
strings via plain YAML load, so under Ansible 2.19+ they are not
`TrustedAsTemplate`-tagged. When `container_volumes` / `compose_volumes`
later asked their captured `templar.template()` to render those strings,
the trust gate short-circuited silently and the raw `{{ ... }}` flowed
verbatim into the rendered compose.yml, breaking `docker compose build`
and `docker stack deploy` with "service refers to undefined volume
{{ ... }}".
Replace the `contextlib.suppress(Exception)` wrapper in each lookup's
`render_jinja` closure with an unconditional `templar.template` call
whose input is wrapped in `_trust_as_template`, so the engine actually
compiles the template. Nested `lookup('container', ..., 'directories.x')`
calls inside canonical bind sources resolve through normal Jinja
plumbing once the outer string is trusted.
Validated by the t5 -> t8 act-swarm-zombie progression
(ok=261 -> ok=452 -> ok=798).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
