categories.yml moves from roles/ to the repository-root meta/, joining packages.yml. utils/roles/categories.py owns its location plus the tree-walking helpers that were stranded in entity/name.py, so no caller spells the path or duplicates the lookup.
meta/distros.yml becomes the SPOT for every distribution: os_family, Galaxy platform name, dev-environment base image, and the pkgmgr/pkgmgr-virgin/environment reference templates. It replaces seven scattered copies (DISTRO_FAMILY, VALID_DISTROS, _DISTROS, _CANONICAL_PLATFORMS, two shell arrays, a Makefile literal). INFINITO_DISTROS is no longer a static default but generated by utils/env/handlers/infinito/distros.py; the shell consumers read it through the env layer.
Drops dead CLI surface found along the way: --categories-file and --roles-dir on the provision CLI plus three unused parameters of generate_dynamic_inventory, and the positional roles_file of the invokable categories command.
tests/lint/repository/test_distros_spot.py pins the copies that cannot call the SPOT (workflow literals, docs table) and forbids composing a pkgmgr reference outside it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five helpers rescanned the whole roles/ tree on every call while their result is a pure function of the (static) tree, so callers that invoke them per-role/per-app/per-service turned the scan into O(N^2) filesystem ops. compute_application_gid was the measured hotspot: 254 rescans in one applications render (~25s of a 41s cold render). Memoize each tree-derived result with functools.cache keyed by the resolved roles_dir (the repo's existing idiom), returning tuples so the shared cache cannot be mutated; dependency_resolver uses an instance cache instead since functools.cache on a method would pin instances via self. Cold applications/domains render drops 41s -> ~8.5s (5.6x); behaviour is unchanged (GID/registry/placement/path values identical).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Transient CDN/mirror TLS resets are a recurring hard-fail class (docker.com, PyPI). Harden all three egress paths at their SPOT: the curl filter now emits --retry-all-errors (retries=0 opts a call out), so every ansible curl retries; a new lint requires --retry-all-errors on file-downloading curl and --tries on wget in verbatim files/*.sh (which cannot use the filter), fixing the moodle/alias downloads it flagged; and the update role configures native retries for every package manager - apt Acquire::Retries, dnf retries=, pacman XferCommand over retrying curl - covering all five CI distros (ubuntu shares apt, centos shares dnf).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The server.yml split hoisted csp to the top level; the filter kept
reading server.csp.whitelist and silently returned no mirrorable asset
hosts, leaving the mirror vhost whitelist and body rewrite empty.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The compose and swarm render paths launder dollars differently; the launched argv arrived with $pid collapsed to an empty string and $! doubled, and the sidecar crash-looped on a sh syntax error, so opentalk's wait for the embedded object store exhausted its retries. The bucket bootstrap now polls /status from a backgrounded subshell and exec makes the server PID 1, which needs no pid bookkeeping and receives docker stop's TERM directly; the poll loop dies with the container and readiness stays gated by the consumer's bounded wait.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
categories.yml now declares modes defaults per category (deepest match wins, parents apply as fallback): update, drv and svc-net disable compose and swarm (host-hardware and host-networking categories), svc-registry requires both enabled (CI bootstrap infrastructure). role_modes_defaults resolves a role's policy; the new category-modes lint requires every deploy-bearing service entry to declare exactly those modes.<mode>.enabled values explicitly, exempt via '# nocheck: category-modes' on the service key. update, the four drv roles and the three wireguard roles declare their values explicitly; the dict-valued modes meta key is excluded from the category-tree walkers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
roles/categories.yml now declares the ordered 'stages' and a per-category 'stage' (plus run_after and bootstrap). utils/roles/stage.py resolves a role to its stage and exposes stage_groups(); the new stage_groups lookup feeds the 01_constructor / 02_server / 02_workstation / 03_destructor group loops in place of their hardcoded lists. A static invariant test proves the lookup reproduces the exact previous loops, so the rendered plays (and deploy behavior) are unchanged.
cli.meta.roles.order.run orders its main phase by stage as a run_after tiebreak. plugins/filter/invokable_paths tolerates the new sibling 'stages' key, and the invokable-roles wiring test recognizes stage_groups-driven wiring.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.
Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolve every 3+-file same-prefix group flagged by the new prefix lint
by moving the files into a subfolder named after the shared prefix and
dropping the prefix from the file names, updating all references:
- utils/env/handlers/infinito_*.py -> infinito/ (dir/, package_cache/
nested a second level); registry imports and the four handler-dir
globs switched to iter_project_files
- utils/install/lint/ansible_*.py -> ansible/ (aliased imports)
- roles/sys-svc-compose/files/swarm_*.{py,sh} -> files/swarm/
- web-app-erpnext/web-app-zammad files/scripts/apply_*.{py,rb} -> apply/
- web-app-odoo files/sql/{select,update}-* and sql/oidc/{grant,insert,
select,update}-* -> matching subfolders
- web-app-keycloak tasks/update/05_ldap*.yml -> 05_ldap/ (internal
includes made role_path-absolute)
- plugins/filter/get_*.py -> plugins/filter/get/ (filter names are the
filters() dict keys, so Jinja usage is unaffected; ansible discovers
the one-level-deep subdir)
Ansible lookup dirs, addon dirs and .github/workflows keep their flat
layout (the filename is the public id / GitHub only discovers flat).
Unit tests mirror the new source layout.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The export subdir was created run_once on the NFS server only; a worker
that had already readdir'd the export kept a negative dentry and swarm
rejected every replicas>1 task with 'readdirent .../_data: no such file
or directory' for the whole converge window. Materialize the bind-device
dir through each node's own mount, guarded by a per-node mountpoint
assert so an unmounted node fails the deploy loudly instead of receiving
a local mkdir that splits the brain and trips the nfs-client
has-local-data assert on the next full deploy.
compose_volumes popped the volume 'nfs' metadata before nfs_prep could
read it, so declared uid/gid/mode never took effect and every subdir
fell back to root 0755; keep the mapping under the compose-spec
extension key x-infinito-nfs (ignored by docker, verified against stack
config and compose config). KIX declares uid/gid 110010100 mode 0770
through it, fixing the backend_api_token EACCES crash on its shared
volume.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The OpenTalk controller v0.34.1 verifies its bucket via S3 ListBuckets
membership, but SeaweedFS hides every bucket from an unconfigured
identity (confirmed identical on 4.38 and 4.39), and the embedded sidecar
started with no s3.config, so the controller's signed request got an
empty bucket list and crash-looped on 'bucket missing' although the
bucket existed. The sidecar now runs s3.configure to register the
consumer's access key with Read/Write/List/Tagging on its bucket, the
same grant the central provider gets from its mounted s3.json.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New web-svc-mirror role: an nginx vhost that proxies and caches exactly
the external asset hosts aggregated from all deployed applications' CSP
whitelists (external_asset_origins filter) under first-party paths
(/<origin-host>/...), refusing every other upstream. svc-net-tor
consumes the service, so any tor deployment automatically serves the
mirror on its onion.
For apps with services.tor.enabled the shared body filter rewrites
external asset URLs in HTML to the onion mirror (onion sessions only;
clearnet untouched) and the CSP builder allows the mirror origin in the
asset directives. The dashboard's keycloak-js vendoring is reverted:
onion sessions now load the adapter through the mirror, clearnet loads
jsdelivr directly as before.
Verified: variant=0 deploy green (failed=0), dashboard 14/14 over onion
with keycloak-js served through the mirror, mirror spec green, health
check accepts the 404 vhost root.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two wrapper behaviours move onto single sources of truth:
Exec timeout: SYS_TIMEOUT_CONTAINER_EXEC (07_schedule.yml, default 7d so a
hard kill never beats a production migration; the dev/CI inventory
tightens it to 60min) reaches every task host via the play environment as
CONTAINER_EXEC_TIMEOUT. The wrapper prefixes only the exec subcommand with
coreutils timeout --kill-after=15. Ansible task timeouts are SIGALRM-based
and slept through the last wedged docker socket for 4.5 hours; the OS
level kill still fires there and returns 124, which until/failed_when and
the rescue diagnostics react to. Unset or 0 keeps todays behaviour.
CA container paths: the bind-mount targets for the injected root CA and
wrapper script lived as string literals in container.py, compose_ca.py
and the filter plugin. They now live once in the CA_TRUST group_vars SPOT
(inject_cert_container, inject_wrapper_container) and reach container.py
as env vars (play environment plus CA_TRUST_SYSTEMD_ENV) and compose_ca.py
as CLI arguments built by the compose_ca_inject_cmd lookup. A caller that
provides the host CA env but not the container paths is half-migrated and
fails loudly instead of silently dropping CA injection; a host without any
CA env keeps the soft fallback. All tests read the paths from the SPOT,
and a regression test asserts the runtime scripts carry no literal copy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The controller crash-looped on a missing S3 bucket and the recorder on an unprovisioned OIDC client while the swarm converge gate blocked before any post-flush task could create either. The embedded seaweedfs sidecar now bootstraps its consumer bucket itself at startup (new seaweedfs_sidecar_script filter: entrypoint-wrapped server, bounded readiness poll, idempotent weed shell bucket create, TERM/INT forwarded to the server), and the Keycloak clients are provisioned before the handler flush. Post-flush bucket tasks stay as an idempotent backstop; the obsolete restart tasks are dropped. Live-validated against chrislusf/seaweedfs:4.38: bucket created, stop behavior identical to the stock entrypoint.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rename the meta/services.yml key across all roles, the roles_with_default_placement lookup (now roles_with_placement), the get_role_default_placement / iter_roles_with_default_placement helpers, and every consumer (swarm closure, compose_volumes filter, nfs_prep, container deploy template, constructor stage, extend_inventory) plus their tests and docs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New to_onion_url filter swaps a clearnet host suffix for the tor_node and forces http (no-op without tor_node or off-primary hosts). body_filter.lua uses it in onionize() so that .onion requests receive the onion CDN origin for injected head/body snippets (e.g. universal-logout.js), which the Tor browser cannot reach on clearnet. Covered by 7 unit tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A curl against an accepted-but-silent peer hangs forever: task retries
never fire because the first attempt never returns, and the CI job dies
at the 6h runner cut (proven by the mailu admin-API hang that took the
suitecrm/peertube swarm legs down). The new `curl` filter renders the
hardened base command ("{{ 30 | curl }}" -> curl -s --connect-timeout 5
--max-time 30); scaling composes with lookup('timeout').
test_curl_via_filter enforces it: bare curl invocations in role tasks,
templates and shipped scripts fail the lint with a pointer to the
filter. Plain .sh files, which cannot render Jinja, pass by pinning
--max-time inline; Docker healthchecks are exempt (their timeout field
caps them); `# nocheck: curl-timeout` opts out (catalogued in
suppression.md).
Convert all 34 existing invocations: mailu (replacing the short-lived
MAILU_CURL var), elasticsearch, checkmk, yourls (300s for the
schema-creating installer POST), the telegram alert script, and the
plain scripts of jellyfin, svc-runner, apt_repo, moodle (600s tarball)
and the mattermost provisioner. Mailu's user/token tasks additionally
carry task-level timeouts so a wedged container exec cannot stall the
play.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deploy GitLab from the official CNG CE images as six services: webservice (puma) and sidekiq scale to the topology default, workhorse is the sole HTTP entry, gitaly and a one-shot migrations job are pinned singletons, shell runs gitlab-sshd. The Omnibus container, its Dockerfile and the GITLAB_OMNIBUS_* config groups are removed. All rails/workhorse/shell/gitaly configs and shared secrets are pre-rendered by Ansible and mounted read-only; eleven credentials land in meta/schema.yml including an rsa_pem_2048 OIDC signing key (new value_generator algorithm). Postgres, redis and S3 object storage ride the existing lookups; OIDC, SMTP, the public SSH port and the root password are preserved. compose_volumes gains a per-volume nfs:false opt-out so the gitaly repositories volume stays node-local in swarm (upstream forbids git repositories on NFS); nfs_prep skips volumes without driver_opts accordingly. The Nextcloud gitlab integration execs the CNG rails runner. ruby_single_line dies with its only consumer; ruby_dq keeps test coverage. Ports (8181/8080/2222/8075/3808/9122) and service names resolve from single SPOT declarations in meta/services.yml via vars constants; the version pin sits on the webservice image pair so the updater can bump it. Live-validated in compose up to all six services healthy and migrations completed; fresh-install-only semantics for existing Omnibus data are documented.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Over Tor a page served on an .onion still advertised a clearnet-only CSP and oauth2-proxy config, so onion origins were blocked (dashboard could not frame auth.<onion>; oauth2-proxy rejected the redirect back to the onion app). build_csp_header now mirrors the clearnet primary to the node onion per token; the keycloak oauth2-proxy config derives whitelist_domains and cookie_domains from the app resolved domains. Exclusive (onion-only) apps get onion tokens only, primary apps get both, non-tor apps stay clearnet.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The swarm branch passed values through raw, so a value with interior
whitespace (MAIL_FROM_NAME='Infinito.Nexus by ...') landed unquoted in
the rendered .env and broke the env parser: the snipe-it service
crashlooped at 0/3 until the converge wait failed in every variant.
Quote (and escape backslash/double-quote) only when the value contains
space/tab/newline; everything else stays raw, so the swarm invariants
hold: no compose $$-doubling, raw $ and double quotes preserved,
and the playwright --env-file path (docker CLI keeps quotes literal)
is unaffected because no playwright env value can contain whitespace.
Compose quoting is untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
resource_filter now falls back service_name -> entity_name -> hard
default, so entity-level limits keep applying to per-service compose
entries. Compose service names drop the entity prefix (kix, jitsi,
zammad, bluesky); the docker container_name keeps it via compose_only.
A new lint forbids '<entity>-'-prefixed service names.
Service names and internal ports are declared once: vars/main.yml
holds <ROLE>_<SERVICE>_SERVICE constants (container names derived via
entity_name), meta/services.yml declares ports.internal consumed
through config lookups. sso-proxy upstreams stack-qualify the origin
host in swarm because short aliases are ambiguous on shared overlays.
Stale proxy.service_key entries fixed for erpnext/shopware/nextcloud.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
One entrypoint wrapper imports the project CA into the JVM truststore for penpot and jenkins: env-driven (CA_TRUST_CERT from sys-svc-compose-ca, set in both modes), fail-open without a cert (production), fail-loud with one; writable cacerts are patched in place, read-only layouts fall back to /tmp + JAVA_TOOL_OPTIONS. Replaces penpot's inline compose command and jenkins' role-local entrypoint, which silently skipped the import when cacerts were not writable. ca_cert_container becomes a plain constant since its only Jinja consumer moved into the wrapper.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ca_cert_host derives the host cert path from SOFTWARE_DOMAIN; ca_cert_container carries the fixed in-container bind-mount target. CA_ROOT.cert_host and the test fixtures consume the filter; the standalone runtime scripts (compose_ca.py, container.py) cannot import it and are pinned to the SPOT by the unit test instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
'login source already exists' was matched inline in both the LDAP and the OIDC setup; the filter is the single marker source and normalizes case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The two handler tasks carried the identical four-marker stderr chain inline and matched case-sensitively; the filter is the single marker source and normalizes case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The five docker network connect/disconnect tolerances were inline stderr substring chains, inconsistent between sites (the handler variants matched case-sensitively without a default). The docker_network_noop filter is the single marker source per operation; disconnect additionally tolerates absent networks/containers, which is the proven swarm failure ('No such container: postgres' when the central DB is a swarm service). Drop the now-orphaned DISCOURSE_ERROR_NOT_CONNECTED.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In the swarm-DinD test the ansible controller (ansible_connection: docker) cannot read the STACK_HOST-rendered xwiki.properties, so compose_volumes' content hash falls back to the source path and the config name is constant. xwiki deploys twice (bootstrap with the OIDC block off, final with it on after the extension installs), so the config content changes but the name does not, and the final docker stack deploy fails to update the immutable config with 'only updates to Labels are allowed'.
Honor an explicit name: on config/secret entries in compose_volumes (volumes already did), and give xwiki's properties config a name that rotates with the OIDC phase, so the final deploy creates a new config instead of updating in place. Backward-compatible: no other role sets name: on a config. make test green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
compose_volumes emitted a top-level configs:/secrets: entry for every declared
config/secret, but container_volumes already gates each per-service reference on
its mount's when:. So a config whose only mount was gated off (e.g. LDAP
disabled) became an orphan: a top-level object no service references, whose file:
source points at a file the role never renders (same flag). Under docker stack
deploy that source is read at deploy time and the deploy fails; under compose it
was a harmless unused entry.
Gate the top-level emission on the same mount_when_passes container_volumes uses,
so a top-level config/secret exists iff a service actually references it. To share
the predicate, promote the private _mount_when_passes from container_volumes into
utils/roles/applications/mounts and import it in both filters; the bare except
moves with it (inline noqa), so container_volumes no longer needs its BLE001
per-file-ignore.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Model change: per-user 'reserved: true|false' becomes 'accounts', a list of
host/mailbox/identity account types. Default when absent derives from roles
(bot/administrator -> [mailbox, identity], else [identity]) both for role-file
users and — via _apply_account_defaults after the inventory merge — for
inventory-only users, so users like biber keep their LDAP/Keycloak accounts.
Auto-generated role-suffix reservations get accounts: [].
New 'forward' field: mailu creates <username>@domain -> forward user's email
via the new 04b_manage_alias.yml (create-only; no_log on the loop so user
dicts never hit the log). Mailbox creation and password rotation now gate on
'mailbox' in accounts instead of role names; the mailu rbac role is renamed
mail-bot -> bot (token gate included).
svc-db-openldap drops the provision.reserved flag; OPENLDAP_USERS is always
the identity subset. reserved_usernames/non_reserved_users now classify by
'identity' in accounts, keeping the Keycloak deny-list semantics. root/backup
pin accounts: [host], administrator [host, mailbox, identity], helpdesk
[mailbox].
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extract the csp, domains, and networks blocks out of roles/*/meta/server.yml
into dedicated meta files, each mapping to its own top-level applications key
via the recursive meta-walk:
csp: -> meta/csp.yml -> applications.<app>.csp
domains: -> meta/domains.yml -> applications.<app>.domains
networks: -> meta/networks.yml -> applications.<app>.networks
Residual nginx/proxy keys (status_codes, client_max_body_size, locations,
ip6_enabled, config_upstream_url) stay in server.yml; an emptied server.yml is
deleted. 98 server.yml -> 249 new files, 74 deleted.
- Consumers repointed applications.<app>.server.{csp,domains,networks} ->
applications.<app>.{csp,domains,networks} across templates, vars, tasks,
lookups and filters.
- Infra readers migrated: canonical_domains_map, application_domain_index,
build_ldap_role_entries, service-registry overlay discovery,
roles_with_service, and the file-based readers utils/domains/list.py +
utils/meta/scan.py (read the now-deleted server.yml directly).
- Detection + mapping: _META_TOPICS gains csp/domains/networks;
ROLE_FILE_META_{CSP,DOMAINS,NETWORKS} constants + ROLE_FILES entries.
- New self-expiring lint guard (test_server_meta_no_split_fields) fails if a
server.yml still carries a split field; auto-fails from 2026-12-28 as a
delete reminder.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace 'lookup(applications) | node_max_old_space_size(app, service)' with the 'lookup(node_max_old_space_size, app, service)' lookup form (it resolves the app config itself), dropping the boilerplate at all 4 call sites (nextcloud whiteboard, peertube x2, suitecrm). Extract the shared decimal to_bytes size parser into utils/sizes.py, used by the lookup and the peertube install_mem_limit filter (removing the cross-plugin import).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A swarm docker stack deploy env-file value must be a single physical line; the
multi-line GITLAB_OMNIBUS_CONFIG was read as separate whitespace-named env
entries and aborted the deploy. Add a depth-aware ruby_single_line filter that
collapses the Ruby program to one line (statement-boundary newlines at bracket
depth 0 become ';', newlines inside an open bracket or a string become a space
so hash/array literals like the OIDC omniauth_providers stay valid) and apply it
on the swarm branch only; compose keeps the multi-line value unchanged.
Covered by tests/unit/plugins/filter/test_ruby.py (10 cases incl. a realistic
omnibus config, escaped quotes and nested literals).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Foundation for the central svc-db-*/svc-dns-* engine roles: the engine lookup and engines descriptor, per-consumer engine credentials in the inventory manager, shared-net overlay rendering, container_depends_on, the sys-svc-engine provisioning dispatcher, and the compose_volumes NFS-placement split. base.yml.j2 gates the embedded redis sidecar on lookup('engine','redis',application_id,'local').
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The server args were duplicated across the standalone provider stack (compose.yml.j2) and the local sidecar (service.yml.j2), so the -ip fix had to be applied in two places. They now live in one filter (plugins/filter/seaweedfs.py); both templates keep command: inline and render the args via it, appending -s3.config only when the caller passes the mounted s3.json path (the sidecar omits it). Adds a unit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reconcile main's object-store feature and pre-swarm deploy tasks with the branch's compose/networks/volumes SPOT and swarm lints.
objstore (seaweedfs/minio): standalone compose + embedded service templates migrated to the container_image/container_volumes/container_networks/compose_volumes/compose_networks/depends_on lookups; meta/volumes.yml on the canonical dict schema with s3.json and nginx.conf as type:config; embedded services made swarm-compatible via compose_only/compose_restart.
deploy tasks (erpnext, opentalk, seaweedfs, shopware, snipe-it, wordpress, yourls): container exec routed through the container_address resolver and gated on IS_STACK_HOST, shell module plus stack_host_template, no chdir or compose-verb forms.
Also removed orphaned vars, restored MAILU_INSTANCE_DIR, freed the seaweedfs subnet (192.168.206.0/24), stripped literal quotes from the nextcloud/bookwyrm env templates, and routed main's objstore compose_volumes tests through dir_var_lib.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The oauth2-proxy sidecar set cookie_domains to the app's primary domain only. A role served under several sibling subdomains (e.g. SeaweedFS filer/master/api under one base) therefore could not read its session cookie on the non-primary hosts, looping the auth chain. Add the common_dns_suffix filter and use it so cookie_domains becomes the longest DNS suffix shared by all of the app's domains: a single-domain app keeps its exact host (unchanged), a multi-domain app collapses to the shared parent so one session cookie spans every subdomain.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a database-style object-store abstraction: the objstore lookup resolves the active engine (seaweedfs|minio) per consumer and yields a uniform S3 payload, objstore_secret_key is auto-generated like database_password, sys-svc-objstore delegates per-consumer provisioning to the resolved provider's objstore_init entrypoint (lazy-loading the provider when it was not preloaded), and compose/network/volume plumbing supports both a shared central instance and an embedded storage-only instance per consumer.
Add the web-app-seaweedfs provider role: Filer and Master UIs behind the admin-only oauth2-proxy via a role-local nginx and proxy.conf.j2 (S3 API domain stays ungated, admin UIs deny with 403 when SSO is disabled), per-consumer S3 identities from s3.json, idempotent bucket provisioning via weed shell, a /24 on the central-provider subnet range, Playwright coverage for admin/biber/guest plus consumer bucket existence, and unit tests for the objstore lookup and the per-engine compose volumes.
Mark web-app-minio lifecycle: unsupported, move its mc client image into a dedicated services.client entry, align it with the objstore credential contract, and make its consumer provisioning idempotent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two related fixes in the volume-rendering filters, both surfaced by
`docker stack deploy` failing on the rendered compose.yml.
1) Render Jinja in mount source/target via the lookup-supplied
render_jinja closure. `container_volumes` previously emitted the
`meta/volumes.yml` `source:` / `mounts[].target:` strings unchanged;
in swarm those flowed verbatim into compose.yml as
`{{ lookup('nginx', ...) }}:/usr/local/openresty/nginx/conf/conf.d`
and broke `docker stack deploy expanding mount target` with
`function "lookup" not defined`. `compose_volumes` had the analogous
bug for top-level `configs:` / `secrets:` source fields.
Add `_maybe_render` to `compose_volumes` and the matching
try/except-AnsibleUndefinedVariable render block to `container_volumes`
for bind source + mount target, raising `AnsibleFilterError` with the
offending expression so failures stay diagnosable.
2) Coerce config/secret `mode:` to int. The YAML/J2 idiom carries `mode`
as a string ("0440"); `docker stack deploy` rejects that as
"services.X.configs.N.mode must be a number". Add `_coerce_mode_int`
(octal-string -> int, with bool + empty + non-octal guards) and
convert before emission. Update the per-service config/secret unit
tests to expect `0o440` / `0o400` instead of the legacy strings.
Validated by the t5 -> t8 act-swarm-zombie progression (mode-int gate
cleared at t6; bind source/target + config/secret source render cleared
at t7 -> t8, lifting ok=452 to ok=798).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Migrate every role's meta/volumes.yml from ad-hoc dict-of-named-volumes
to a canonical dict-of-dicts schema where the YAML key is the semantic
short name and the entry body carries type / name / source / mounts /
nfs / mode / read_only. The optional 'name:' field is the docker volume
name (defaults to the key); the previous 'docker_name:' field is gone.
What's in this commit (210 paths):
Infrastructure (23)
- New filter plugins/filter/{compose,container}_volumes.py — render the
top-level volumes:/configs:/secrets: section and per-service mount
blocks from the canonical meta. container_volumes uses the SEMANTIC
name as per-service mount source so it resolves against compose_volumes'
top-level dict key (docker_name as source produced 'undefined volume'
on swarm).
- New lookup plugins/lookup/{compose,container,volume}_volumes.py.
volume(application_id, name) returns a normalised single entry;
volume(application_id) returns the whole canonical dict.
- utils/cache/applications.py keeps a sibling registry
_CANONICAL_VOLUMES_BY_ROLE outside the merged applications payload so
raw Jinja source: strings never enter the templar walk (root-cause of
the 200x cache-miss + recursive get_merged_applications storm).
- utils/roles/applications/mounts.py — schema validator + normaliser;
list-shape is rejected.
- plugins/lookup/nginx.py: _volume_path_by_name reads canonical directly.
- scripts/tests/deploy/swarm/_context.sh: NFS_VOLUMES extraction handles
both legacy dict and canonical dict shapes.
- roles/svc-prx-openresty/handlers/restart_and_validate.yml: gate
compose-only validate+reload behind a single when-block; swarm uses
service-update instead (bind-mounts of a FILE pin source inode at
task-create time, so atomic write-tmp+rename leaves the running task
on a deleted inode).
- roles/sys-svc-compose/tasks/utils/nfs_prep.yml: switch to
lookup('volume', application_id) | dict2items so NFS subdir
pre-creation works under the new cache split.
Tests (12)
- New strict lints flagging inline mount sections in compose templates,
single-file bind mounts, and legacy lookup('config', ..., 'volumes.X.Y')
call sites — all volume-mount surfaces MUST flow through container_volumes
/ compose_volumes / volume lookups.
- Integration: test_volumes_compose_chain + test_volume_constants_are_lookups
accept both the legacy and the new lookup('volume', app, 'X').name forms
during the migration window.
- Unit tests for the new lookup, filter, mounts helper, plus updated
nginx lookup test.
- Prometheus role-wiring test reads canonical meta for the
alert_rules.yml bind mount.
Role migrations (175)
- 62 meta/volumes.yml converted to dict-of-dicts canonical shape
- 63 templates/compose*.yml.j2 replace inline volumes:/configs:/secrets:
with lookup('container_volumes', application_id, '<service>') |
indent(N) per service + a single trailing lookup('compose_volumes',
application_id) for the top-level block
- 50 vars/main.yml drop dead volume constants and refactor 23 remaining
legacy lookup('config', ..., 'volumes.X.Y') to lookup('volume',
application_id, 'X').Y across 9 roles
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The mapping from compose-style `docker_restart_policy` (`no`,
`on-failure[:N]`, `always`, `unless-stopped`) to swarm's
`deploy.restart_policy.condition` (`none`, `on-failure`, `any`) was
seven lines of inline Jinja `{% set / if / elif / else %}` in
`roles/sys-svc-container/templates/deploy.yml.j2`. That made the
template noisy at the call site, hid the typo-safety logic from
unit tests, and forced every reviewer to re-derive the mapping table
from a chain of conditionals.
Move the mapping into a dedicated filter plugin so the call site is
a single readable expression and the mapping is exercised by a
focused test suite.
- plugins/filter/swarm_restart_condition.py: pure function, no Jinja
context dependency (the input is the only thing that matters).
Strips + str-casts the input so YAML-literal slop like `" no "`
or a numeric `0` does not defeat the mapping. Output is always a
member of the closed swarm condition set `{none, on-failure, any}`;
unknown / typo inputs map to `any` (safe default -- a misconfigured
long-running service still runs and the operator notices the typo
via the literal `condition: any` in the rendered file, instead of
silently degrading to `none` which would skip respawning a real
service).
- tests/unit/plugins/filter/test_swarm_restart_condition.py: 12
cases. Direct mapping per branch, `on-failure:N` suffix form
(suffix dropped because swarm carries max-attempts in a separate
`max_attempts` key), None / empty / whitespace fallback,
stringification of non-string input, strip semantics, an explicit
unknown-value test that pins the safe-default rationale, and a
property test over twelve inputs that asserts the output is always
inside the closed swarm condition set. A second test class
verifies the filter is registered under the exact name
`swarm_restart_condition` so a typo in the FilterModule shim does
not silently fall back to the identity filter.
- roles/sys-svc-container/templates/deploy.yml.j2: thirteen lines of
inline Jinja (`{% set _drp = … %}` + four `{% set
_swarm_restart_condition = … %}` branches) collapse to a single
filter expression:
condition: {{ docker_restart_policy
| default(DOCKER_RESTART_POLICY)
| swarm_restart_condition }}
The runtime mapping is identical to the pre-refactor inline
version (verified by the new unit tests against the same input
buckets), so nothing changes for existing roles.
docker stack deploy passes env-file values verbatim, including outer
double quotes - the container reads KEY="value" instead of KEY=value
and DB auth, URL parsers, and int(os.environ['KEY']) consumers break.
Compose keeps stripping the quotes as before.
- plugins/filter/dotenv.py: read DEPLOYMENT_MODE from the render
context via @jinja2.pass_context. compose mode keeps the old quoting
(escape $$, \, "). swarm mode passes the value through verbatim.
Missing or unknown mode falls back to compose quoting so a typo
never silently leaks unquoted values into compose. Direct Python
callers without a Jinja context keep the compose-style default.
- tests/unit/plugins/filter/test_dotenv_quote.py: keep the 12
direct-call cases, add 8 TestDotenvQuoteModeAware cases rendering
the filter through a real jinja2.Environment.
- 4 OIDC_TITLE templates (bookwyrm, confluence, jira, pretix): replace
the manual "{{ X | replace('\"','\\\"') }}" Compose-only escape
with {{ X | dotenv_quote }}.
- 3 env.j2 templates (nextcloud, taiga, wordpress): drop literal
outer quotes that were Compose-only decoration; keep | dotenv_quote
on the password / token assignments.
- sys-svc-rdbms/templates/env/mariadb.env.j2: strip literal quotes
unconditionally - the file name is not env.j2 so the dotenv-quote
integration test does not require the filter and the generated DB
passwords are alphanumeric.
