New roles/sys-svc-container/templates/env.j2 sets TZ={{ HOST_TIMEZONE }} and LANG=C.UTF-8. Every role env.j2 includes it at the top; it renders via stack_host_template, the same engine that resolves the includes in compose.yml.j2, so it works in both compose and swarm. TZ/LANG are removed from the individual role env.j2 so each value stays single-source.
New lint tests/lint/ansible/test_env_base_include.py verifies every role env.j2 includes the base and never re-sets a base-owned key (keys derived from the base file).
mattermost and gitea meta/volumes.yml drop the /etc/timezone host-path bind: swarm hard-rejects a task whose bind source is missing on the node (Debian-trixie nodes have no /etc/timezone). TZ now comes from the base env. /etc/localtime bind also dropped; the container timezone follows TZ.
Non-directive comments were removed from the touched env.j2 (banners, restatements, reference URLs, narration) so the changed files pass the comment-lint; only tool directives remain.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The variants-static-override integration test flags variant overrides that repeat a static meta/services.yml literal. The tor enabled:false pins across the WebRTC roles' variants are intentional hard-invariant documentation, so annotate them with nocheck: variants-static-override like their shared:false siblings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
coturn, bigbluebutton, jitsi and opentalk move WebRTC media over UDP (STUN/TURN, SRTP, JVB, LiveKit SFU), which Tor cannot carry. Pin their tor service block to enabled/shared false in meta/services.yml and in every variant, so no deploy scope or variant ever injects onion domains for them and every consumer lookup renders their clearnet URLs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rename the meta/services.yml key across all roles, the roles_with_default_placement lookup (now roles_with_placement), the get_role_default_placement / iter_roles_with_default_placement helpers, and every consumer (swarm closure, compose_volumes filter, nfs_prep, container deploy template, constructor stage, extend_inventory) plus their tests and docs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New lint test_no_raw_goto.py forbids raw `page.goto(...)` in role playwright specs + persona flows (the gotoOnion wrapper in personas/utils is exempt) so navigation retries transient Tor-transport errors over onion. Migrates every raw call across ~75 roles (343 sites) to `gotoOnion(page, ...)` and teaches gotoOnion to detect relative onion URLs (`isOnionCanonical`). gotoOnion is a behaviour-identical drop-in on clearnet (single attempt, same return). Verified: no-raw-goto lint green, eslint 0 errors, make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add the global TIMEOUT_FACTOR and a shared timeouts.js helper: resolveTimeout(base) scales by TIMEOUT_FACTOR and, over a .onion target, an extra multiplier, so Tor's per-request latency no longer trips hard-coded timeouts. Migrate every raw timeout in the specs (timeout:/timeout=/setTimeout/waitForTimeout) to resolveTimeout, stage the helper in all three staging paths, and add test_no_raw_timeouts to forbid raw numeric timeouts. Clearnet behaviour is unchanged (factor 1: resolveTimeout(N) === N).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
make autoformat output across role meta files, a task file and 19_tor.yml (single->double quotes on mode values, Jinja line reflow). No semantic changes. Kept separate from the onion CSP / oauth2-proxy fixes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pure formatting from make autoformat (a folded long Jinja line in svc-runner vars; double->single space before # nocheck comments in jitsi/openproject/rdr-www services and mailu/openwebui/snipe-it/web-svc-file variants). No semantic changes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Onion routing is transport-layer with no persona/UI surface, so tor cannot be gated from the per-persona Playwright specs. Mark the tor block with '# nocheck: playwright-service-flag' in the 60 domain roles that ship a playwright.env.j2, per test_env_services_match's documented exemption for services that legitimately cannot be Playwright-gated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Each role's meta/variants.yml now pins the tor onion gate both ways so the matrix exercises it: variant 0 pins services.tor.enabled: true, every non-baseline variant pins enabled: false + shared: false (# nocheck: variants-static-override, since tor.shared is a static literal). This gives real test_coverage / test_bond_baseline coverage without the removed per-key code exemptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Domain-serving roles reference the svc-net-tor provider via a tor: block. Consumers declare enabled (dynamic group_names gate) and shared: true (static — onion is always the shared node model, inline nocheck: dynamic-flag). The provider svc-net-tor additionally declares exclusive/primary defaults that consumers inherit via resolve_service_config unless overridden.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Container-name vars derived their value inconsistently (hard-coded literals, entity_name composites) instead of the role-owned SPOT. Add a lint requiring every *_CONTAINER var to resolve through lookup('config', app, 'services.<x>.name'), the bare entity_name form, or the engine lookup; give each multi-container worker a name: in meta/services.yml (with realistic resource limits) and switch its var to the lookup. Extend the engine lookup with a shared-aware 'container' field (central provider container when shared, embedded <entity>-<engine> otherwise) and gate zammad's embedded memcached sidecar on not-shared so its container_name never collides with the central memcached.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
resource_filter now falls back service_name -> entity_name -> hard
default, so entity-level limits keep applying to per-service compose
entries. Compose service names drop the entity prefix (kix, jitsi,
zammad, bluesky); the docker container_name keeps it via compose_only.
A new lint forbids '<entity>-'-prefixed service names.
Service names and internal ports are declared once: vars/main.yml
holds <ROLE>_<SERVICE>_SERVICE constants (container names derived via
entity_name), meta/services.yml declares ports.internal consumed
through config lookups. sso-proxy upstreams stack-qualify the origin
host in swarm because short aliases are ambiguous on shared overlays.
Stale proxy.service_key entries fixed for erpnext/shopware/nextcloud.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every compose.yml.j2 now declares services as '{{ service_name }}:' with
its '{% set %}' knobs on two-space lines above the key, includes at four
spaces and if/for tags aligned to the content they wrap. The layout
renders correctly because stack_host_template enables lstrip_blocks for
compose.yml.j2; indent(4, True) in the container base and '{%+' guards
in whitespace-sensitive scalars keep the shared fragments identical
under both render modes. A new lint test enforces the skeleton.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
meta/volumes.yml stages six type:bind config directories on the stack host; a worker-scheduled task lacks the source paths and the stack never converges. Unpinning requires migrating them to nfs-backed named volumes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Role README Credits now name galaxy_info.author (plain bold or wrapped
in a Markdown link), matched byte-for-byte against the role's own
meta/main.yml, instead of a hard-coded maintainer name. Regenerate all
245 role README Credits blocks to the new 'Implemented by ...' format.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Autoformat (mbake/yamllint) re-indented sequence items under their mapping key
across the regenerated meta files and wrapped one long URL scalar in the matomo
inject task. Whitespace/formatting only, no semantic change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_name_emoji_and_length.py: every Ansible task
name and GitHub Actions workflow/job/step name must start with an emoji (first
char not 0-9/A-Za-z) and be <= 80 chars.
- emit emoji names at the source: cli/build/include generator now writes
"🔧 setup <app>" / "🚿 flush handlers after <app>" so generated
tasks/groups/*-roles.yml satisfy the convention too
- convert existing task/step/job/workflow names to emoji-prefixed, choosing a
task-fitting icon per action (📦 install, ✅ ensure, 🔍 probe, 🥾 bootstrap,
🚀 deploy, 🧩 service-module load, ⬇️ checkout, 📂 include-routines, ...)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract the csp, domains, and networks blocks out of roles/*/meta/server.yml
into dedicated meta files, each mapping to its own top-level applications key
via the recursive meta-walk:
csp: -> meta/csp.yml -> applications.<app>.csp
domains: -> meta/domains.yml -> applications.<app>.domains
networks: -> meta/networks.yml -> applications.<app>.networks
Residual nginx/proxy keys (status_codes, client_max_body_size, locations,
ip6_enabled, config_upstream_url) stay in server.yml; an emptied server.yml is
deleted. 98 server.yml -> 249 new files, 74 deleted.
- Consumers repointed applications.<app>.server.{csp,domains,networks} ->
applications.<app>.{csp,domains,networks} across templates, vars, tasks,
lookups and filters.
- Infra readers migrated: canonical_domains_map, application_domain_index,
build_ldap_role_entries, service-registry overlay discovery,
roles_with_service, and the file-based readers utils/domains/list.py +
utils/meta/scan.py (read the now-deleted server.yml directly).
- Detection + mapping: _META_TOPICS gains csp/domains/networks;
ROLE_FILE_META_{CSP,DOMAINS,NETWORKS} constants + ROLE_FILES entries.
- New self-expiring lint guard (test_server_meta_no_split_fields) fails if a
server.yml still carries a split field; auto-fails from 2026-12-28 as a
delete reminder.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Swarm rejected the jitsi replicas with 'bind source path does not exist' for the jicofo/jvb/config/transcripts dirs, because 02_manager_ops pre-created only the prosody dirs. Add the remaining meta/volumes.yml bind sources to the dir-creation loop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-role swarm overlay local subnet was a /28 (14 usable IPs). A role's swarm stack plus its sidecars and per-task network attachments can need more than that, which surfaces as 'node is missing network attachments, ip addresses may be exhausted' and leaves services (sso-proxy, app frontends) unresolvable mid-deploy. Re-allocate all 74 /28 roles to a unique /24 (254 usable) inside 192.168.0.0/16, chosen via the network address suggester so the assignments stay non-overlapping. Validated by tests/integration/infrastructure/networks/test_unique_valid_and_mapped.py and a full make test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The container_ports migration left make test red; reconcile it:
- Delete 10 port vars orphaned when their compose mappings moved to the
container_ports lookup (OPENLDAP_DOCKER_PORT_OPEN, RABBITMQ_INTERNAL_PORT,
REGISTRY_CACHE_PORT, ERPNEXT_FRONTEND_PORT, FLOWISE_PORT_PUBLIC,
JITSI_WEB_HTTP_PORT, KEYCLOAK_SERVER_HOST, MINIO_API_PORT_INTERNAL,
OPENCLOUD_HTTP_PORT, OPENWEBUI_PORT_PUBLIC).
- Add PORT_BANDS.public entries for the fixed protocol ports the migration
declared (smtp/smtps/submission/pop3/pop3s/imap/imaps/sieve, rtmp, c2s/s2s).
- Route the container_ports internal-declared lint through the SPOT constants
(ROLE_FILE_TEMPL_COMPOSE / ROLE_FILE_META_SERVICES), cached read_text, and
load_yaml_any, per repo conventions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Take the host IP as an optional third element of each term
([service, protocol, host_ip]) instead of an ip= kwarg, and build the whole
ports: block from declared ports.local/public (host) + ports.internal
(container). Convert every compose template across 84 roles, folding the
remaining raw var/numeric/specific-ip mappings (including mailu's per-interface
mail ports bound to MAILU_IP4_PUBLIC). Multi-term calls render one term per line.
Add the compose-ports-must-use-container-ports lint: it forbids any raw
"<host>:<published>:<container>" publish item under a ports: block and requires
every term to declare ports.internal. Remove
tests/lint/roles/test_swarm_internal_ports.py, now subsumed.
Fix a container_depends_on rendering regression. The lookup self-indented
(default indent=4), so the added 4-space placement + | indent(4) triple-indented
the block. Flip the lookup default to column 0 (consistent with
container_networks), make every call use the 4-space + | indent(4) form, and add
a lint enforcing it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`bond` is a 0..1 factor for how strongly a service is tied to the overall
integration: bond: 1 = always required, bond: 0 = omittable. Groundwork for
tooling that later scores service importance and decides what can be dropped.
Named `bond` (not `bound`) to avoid clashing with the resource bounds/limits.
Add test_service_bond.py (lint): every enabled + shared service must declare a
valid `bond` (number in 0..1). Seed `bond: 1` as a placeholder on all 638
enabled+shared service entries across 88 roles, to be refined later.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reconcile main's object-store feature and pre-swarm deploy tasks with the branch's compose/networks/volumes SPOT and swarm lints.
objstore (seaweedfs/minio): standalone compose + embedded service templates migrated to the container_image/container_volumes/container_networks/compose_volumes/compose_networks/depends_on lookups; meta/volumes.yml on the canonical dict schema with s3.json and nginx.conf as type:config; embedded services made swarm-compatible via compose_only/compose_restart.
deploy tasks (erpnext, opentalk, seaweedfs, shopware, snipe-it, wordpress, yourls): container exec routed through the container_address resolver and gated on IS_STACK_HOST, shell module plus stack_host_template, no chdir or compose-verb forms.
Also removed orphaned vars, restored MAILU_INSTANCE_DIR, freed the seaweedfs subnet (192.168.206.0/24), stripped literal quotes from the nextcloud/bookwyrm env templates, and routed main's objstore compose_volumes tests through dir_var_lib.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Give each reverse-proxied app a swarm-resolvable upstream: add services.<entity>.ports.internal.http to the role metadata and rename the main compose service from the generic 'application' to the entity name, so its short DNS alias on the shared overlay is unique. Thread the service name through compose, vars (*_SERVICE_NAME), volume mounts and the oauth2 origin host. Consolidate the split image/ports blocks in bridgy-fed and postmarks into the entity block.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Convert the two registry-driven networks templates into pure-Python
rendering functions plus thin Ansible lookup plugins:
roles/sys-svc-compose/templates/networks.yml.j2 -> {{ lookup('compose_networks') }}
roles/sys-svc-container/templates/networks.yml.j2 -> {{ lookup('container_networks') | indent(4) }}
The rendering logic lives in utils/networks/render.py as pure-Python
functions that take a registry + closure-style config/database lookups,
making the logic directly unit-testable without a Jinja stub harness.
The lookup plugins are thin wrappers that resolve application_id,
DEPLOYMENT_MODE, and swarm.network.encryption from the templar, build
the service_registry, and forward to the render functions.
Migrate 233 include lines across 85 production templates. Convert
web-app-matrix's hand-coded MDAD networks block (default + conditional
postgres + conditional openldap) to the generic lookup; this adds an
openresty attachment when services.sso.enabled (harmless extra
interface, pinned in test_matrix_mdad_consumer_attaches_postgres_ldap_sso).
Mark local-DB / redis sidecars (sys-svc-rdbms/templates/services/,
svc-db-redis/templates/service.yml.j2) with `# nocheck: networks-literal`
because they intentionally attach to default-only.
Extend the lint to scan ALL roles/*/templates/**/*.yml.j2 (not only
compose.yml.j2) and to forbid the legacy include form; the marker
above is the documented escape hatch.
Coverage:
- tests/unit/utils/networks/test_render.py (30 cases incl. encrypted=false literal pin)
- tests/unit/plugins/lookup/test_compose_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/unit/plugins/lookup/test_container_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/integration/infrastructure/compose/test_networks_render.py (10 snapshot scenarios)
- tests/integration/infrastructure/compose/test_network_includes.py (3 lint rules)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add plugins/lookup/container_image.py: returns the full `image: "<ref>"`
line so templates drop the redundant image: prefix and surrounding
quotes. Supports `custom=True` to auto-construct `<entity_name>_custom`
for locally-built images, plus explicit `image=`/`version=` overrides
for exotic cases (taiga back-auth/front-oidc variants, decidim infinito
suffix, etc.).
* 25 unit tests for the lookup cover compose vs swarm prefix, custom
flag, overrides, tag_only=True, missing version warnings.
* New strict lint (tests/lint/.../test_no_raw_image_in_compose_templates)
fails on any raw `image:` line in roles/*/templates/*.j2 unless
nocheck-suppressed. Legacy `image: "{{ lookup(...) }}"` form no
longer accepted.
* Migrate 159 image refs across ~80 roles to the standalone
`{{ lookup('container_image', application_id, '<service_key>') }}`
form. 26 roles use `custom=True` (mariadb, postgres, openldap,
jenkins, jira, gitlab, nextcloud, mediawiki, shopware, wordpress,
odoo, joomla, openproject, bookwyrm, espocrm, hugo, littlejs,
fediwall, roulette-wheel, openresty, confluence and more).
* Fix svc-db-redis application_id leak: service.yml.j2 is included
into consuming roles where application_id evaluates to the outer
role, so the image lookup needs the hardcoded 'svc-db-redis'.
* Integration tests (test_templates_require_image,
test_build_requires_image) recognise the standalone lookup as an
implicit `image:` at the same indent.
* Mediawiki swarm tasks: add explicit `:{{ MEDIAWIKI_VERSION }}` to
the registry-prefixed image string because the custom var is now
bare.
Only 3 legitimate exotic nocheck-suppressed image lines remain:
matrix-mdad runner (nested services.matrix.runner.image path),
matrix mautrix bridges (per-iteration dynamic) and svc-registry-docker
(chicken-and-egg self-reference).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Migrate every role's meta/volumes.yml from ad-hoc dict-of-named-volumes
to a canonical dict-of-dicts schema where the YAML key is the semantic
short name and the entry body carries type / name / source / mounts /
nfs / mode / read_only. The optional 'name:' field is the docker volume
name (defaults to the key); the previous 'docker_name:' field is gone.
What's in this commit (210 paths):
Infrastructure (23)
- New filter plugins/filter/{compose,container}_volumes.py — render the
top-level volumes:/configs:/secrets: section and per-service mount
blocks from the canonical meta. container_volumes uses the SEMANTIC
name as per-service mount source so it resolves against compose_volumes'
top-level dict key (docker_name as source produced 'undefined volume'
on swarm).
- New lookup plugins/lookup/{compose,container,volume}_volumes.py.
volume(application_id, name) returns a normalised single entry;
volume(application_id) returns the whole canonical dict.
- utils/cache/applications.py keeps a sibling registry
_CANONICAL_VOLUMES_BY_ROLE outside the merged applications payload so
raw Jinja source: strings never enter the templar walk (root-cause of
the 200x cache-miss + recursive get_merged_applications storm).
- utils/roles/applications/mounts.py — schema validator + normaliser;
list-shape is rejected.
- plugins/lookup/nginx.py: _volume_path_by_name reads canonical directly.
- scripts/tests/deploy/swarm/_context.sh: NFS_VOLUMES extraction handles
both legacy dict and canonical dict shapes.
- roles/svc-prx-openresty/handlers/restart_and_validate.yml: gate
compose-only validate+reload behind a single when-block; swarm uses
service-update instead (bind-mounts of a FILE pin source inode at
task-create time, so atomic write-tmp+rename leaves the running task
on a deleted inode).
- roles/sys-svc-compose/tasks/utils/nfs_prep.yml: switch to
lookup('volume', application_id) | dict2items so NFS subdir
pre-creation works under the new cache split.
Tests (12)
- New strict lints flagging inline mount sections in compose templates,
single-file bind mounts, and legacy lookup('config', ..., 'volumes.X.Y')
call sites — all volume-mount surfaces MUST flow through container_volumes
/ compose_volumes / volume lookups.
- Integration: test_volumes_compose_chain + test_volume_constants_are_lookups
accept both the legacy and the new lookup('volume', app, 'X').name forms
during the migration window.
- Unit tests for the new lookup, filter, mounts helper, plus updated
nginx lookup test.
- Prometheus role-wiring test reads canonical meta for the
alert_rules.yml bind mount.
Role migrations (175)
- 62 meta/volumes.yml converted to dict-of-dicts canonical shape
- 63 templates/compose*.yml.j2 replace inline volumes:/configs:/secrets:
with lookup('container_volumes', application_id, '<service>') |
indent(N) per service + a single trailing lookup('compose_volumes',
application_id) for the top-level block
- 50 vars/main.yml drop dead volume constants and refactor 23 remaining
legacy lookup('config', ..., 'volumes.X.Y') to lookup('volume',
application_id, 'X').Y across 9 roles
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The compose/swarm split on `depends_on:` shape (map form with
`condition:` vs. list form, per `docker stack deploy` constraint)
was being hand-coded as a `{% if DEPLOYMENT_MODE == 'swarm' %}` gate
around every block. Repetitive, easy to forget, and templates that
did remember it had to repeat the same six-line pattern per service.
The new `depends_on` lookup encapsulates the shape switch:
{{ lookup('depends_on', {SVC: 'service_healthy', …}) }} # mapping
{{ lookup('depends_on', [SVC_A, SVC_B]) }} # all default
- Compose -> map form with explicit `condition:` per entry.
- Swarm -> list form (just names; swarm ignores depends_on
ordering semantically but accepts the syntax).
- Mapping `None` value uses the default condition (`service_started`,
matching Docker Compose's behaviour when no condition is given).
- Plain list of names uses the default for every entry.
The output is shaped so the call site reads naturally at the
service-member column:
some_service:
image: …
{{ lookup('depends_on', {DB: 'service_healthy'}) }} # at col 4
volumes: …
Concretely the lookup emits line 1 WITHOUT leading whitespace and
lines 2+ with the `indent` kwarg (default 4) plus the YAML-relative
nested indent. Jinja substitution applies the template's leading
spaces to line 1 only, so the column-4 call placement makes
`depends_on:` land at column 4 and the children at 6 / 8. Lines 2+
carry their own indent because Jinja does not propagate template-side
leading whitespace to continuation lines.
Plugin + tests:
- plugins/lookup/depends_on.py: ~190 lines. Mapping / list / empty
input handling, default_condition kwarg, condition validation
against the closed set `{service_started, service_healthy,
service_completed_successfully}`, DEPLOYMENT_MODE resolution via
templar with `mode=` test-only override, indent kwarg semantics
documented in the module docstring.
- tests/unit/plugins/lookup/test_depends_on.py: 27 cases covering
TestInputNormalisation (mapping / list / empty / typo / single
string rejection / non-string sequence rejection / unsupported
type rejection / arity errors), TestModeBehaviour (compose / swarm
/ unknown-falls-to-compose / missing-DEPLOYMENT_MODE-falls-to-compose
/ mode-kwarg override), TestOutputFormatting (line-1 unindented
exact-string assertion, indent=0 edge case, indent=6 edge case,
insertion-order stability for diff-stable output).
Lint:
- tests/lint/ansible/jinja/test_compose_template_uses_depends_on_lookup.py:
rule key `compose-depends-on-must-use-lookup`. Flags three patterns
in any `*compose*.yml.j2` under `roles/`:
1. literal `depends_on:` keyword line
2. flow-form `depends_on: [a, b]`
3. literal `condition: service_*` line
The pre-existing `lookup('container_depends_on', application_id)`
is intentionally not flagged: its call form contains no literal
`depends_on:` token in source. Scan target broadened to
`name.endswith('.yml.j2') and 'compose' in name` so
`compose.override.yml.j2` and `compose-inits.yml.j2` are covered
alongside `compose.yml.j2` and `*.compose.yml.j2`.
Migration (43 depends_on blocks across 17 templates):
- erpnext (6): configurator + backend + websocket + scheduler +
queue-short + queue-long + frontend chains.
- mailu (8): imap + smtp + oletools + antispam + antivirus + webdav +
fetchmail + webmail.
- zammad (4): rails + 3 worker shapes.
- shopware (4): web + worker + scheduler + opensearch, init-gated.
- jitsi (3): three single-key dependencies on JITSI_PROSODY_CONTAINER.
- kix (3): backend + frontend + proxy.
- bigbluebutton/compose.override.yml.j2 (2): bbb-web + etherpad,
inside the existing BBB_COLLABORA_ENABLED gate.
- moodle (2): nginx + cron, both depending on moodle.
- opentalk (2): application + recorder.
- taiga (2): events + gateway.
- matrix (1): loop-driven via `{% set deps = {} %} +
__setitem__` escape hatch over MATRIX_BRIDGES.
- matomo (1): bootstrap service.
- nextcloud (1): whiteboard sidecar.
- openproject (1): proxy.
- funkwhale (1): front.
- magento (1): nginx.
- flowise (1): postgres + redis.
Pre-existing `lookup('container_depends_on', application_id)` calls
(erpnext, nextcloud, moodle, zammad, taiga, matrix, etc.) are left
untouched -- that lookup auto-detects DB / redis from the application
config and emits its own block; it is the app-aware counterpart to
this structural lookup.
Every call placement repositioned to column 4 (standard
compose-service-member indent) so the new line-1-unindented semantic
renders correctly. Eight templates needed the column-4 reposition
(via a mechanical one-shot script that was deleted after).
`make test` green; new lint + 27 unit tests green; independent
re-grep across all `*compose*.yml.j2` finds zero residual raw
`depends_on:` / `condition:` lines.
Across roles/web-app-*, the typical body shape is:
- include_role: sys-stk-full (or sys-svc-compose) cluster prereqs
- <host-fs writes, container exec, compose exec> stack-host only
Under swarm-mode delegation the second group only makes sense on the
host that holds the compose stack (manager), but the role plays on
every host in groups[<role>] so workers abort on missing paths.
Two workflow sweeps (gate-manager-ops + gate-manager-ops-01core, 58
agents total, discover/plan/adversarial-verify per role) plus three
manual fixes surfaced 27 roles to refactor. Each role keeps
sys-stk-full ungated so worker-side prereqs (sys-svc-container, NFS
client) still bootstrap; the trailing manager-only block either moves
into a new tasks/02_manager_ops.yml included with
when: IS_STACK_HOST | bool (SUB_FILE pattern) or gets the gate
inlined on the one or two trailing tasks (INLINE pattern). In
compose mode IS_STACK_HOST stays true so single-host deploys are
unchanged.
Manual catches the workflow agents missed:
- web-app-fediwall: utils/once/flag.yml lives AT THE END of 01_core,
not at the start; preserved verbatim outside the gate so workers
still flip the run_once_<role> fact.
- web-app-xwiki: a pre-existing block(...) wraps sys-stk-full plus
_flush_config.yml; gated only the _flush_config.yml include so
sys-stk-full keeps running on workers.
- web-app-dashboard: 02_javascript.yml (creates /opt/compose/dashboard/
JS dirs) ran BEFORE sys-stk-full and was ungated; added the gate.
Roles touched (INLINE): akaunting, jenkins, magento, mobilizon
Roles touched (SUB_FILE): baserow, dashboard, decidim, erpnext,
fediwall, friendica, gitea, jitsi, keycloak, listmonk, littlejs,
mailu, matomo, moodle, odoo, openproject, opentalk, pixelfed,
shopware, suitecrm, taiga, xwiki, zammad
Single SPOT lookup `plugins/lookup/compose_volumes.py` absorbs the
applications-lookup pipe step and auto-wires `DEPLOYMENT_MODE` plus
the `storage` group var from the templating context. Callers only
pass what genuinely varies per service (`application_id` and an
optional `extra_volumes` dict).
Behavior change (bug fix): the filter was always invoked with the
default `deployment_mode='compose'` because no caller passed it
explicitly, which silently skipped the swarm-NFS driver-opts
injection path. The lookup reads `DEPLOYMENT_MODE` from globals,
so swarm-NFS volumes now render the `driver: local` / `driver_opts:
type: nfs` block correctly. Callers can still override via
`deployment_mode=` / `storage=` kwargs when a template needs
non-default rendering.
Other coordinated changes:
- 81 template call sites migrated from pipe form to lookup form
(single-line and multi-line shapes).
- `FilterModule` registration in plugins/filter/compose_volumes.py
removed so the deprecated pipe form fails fast at render time.
The underlying `compose_volumes()` function stays exported and is
imported by the new lookup plus the existing 21-test filter unit
test module.
- 10 new lookup unit tests pin the contract (term shape, kwargs
pass-through, auto-wiring, override precedence, error cases).
- Integration test `test_compose_volumes_call_required.py` updated:
asserts the new lookup form is present in every `compose.yml.j2`
and adds a second test forbidding the deprecated pipe form
anywhere under `roles/**/templates/`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace per-key `compose_container_name` lookup with a generic
`compose_only(key, value)` SPOT that emits `key: "<value>"` only in
compose mode and nothing in swarm. Same gating logic now serves
container_name (157 call sites) and pull_policy (17 call sites)
without duplicate plugin code.
Why this consolidation:
- The earlier `build-key` lint rule was misdirected. Swarm tolerates
`build:` with a harmless warning when `image:` is pre-pushed (via
the existing handlers/swarm_registry_push.py flow in
roles/sys-svc-compose). The actual swarm-rejected key is
`pull_policy:` ("Additional property pull_policy is not allowed"),
and 10 templates were silently broken because they declared inline
`pull_policy: never` outside the existing `{% if DEPLOYMENT_MODE
\!= 'swarm' %}` gate in roles/sys-svc-container/templates/build.yml.j2.
- A single generic lookup is cleaner than N per-key wrappers when
the gating logic is identical (emit-key-or-nothing).
Lint changes in tests/lint/ansible/jinja/test_mode_incompatible_keys.py:
- New `pull-policy-key` rule, regex `^\s*pull_policy\s*:`, remediation
points at compose_only.
- Existing `container-name` rule keeps its raw-key regex; remediation
now also points at compose_only.
- Broken-split / broken-embed regexes rekeyed on the `compose_only`
lookup name (was `compose_container_name`).
- `build-key` rule and its 12 nocheck annotations across templates
removed.
Coordinated changes:
- 86 templates rewritten (157 container_name calls migrated;
17 inline pull_policy lines replaced).
- 1 redundant `{% if DEPLOYMENT_MODE \!= 'swarm' %}` guard dropped in
build.yml.j2 (the lookup self-gates).
- 12 unit tests pin the compose_only contract.
- Catalog (suppression.md) updated: `pull-policy-key` row added,
`container-name` row remediation updated.
- Old `plugins/lookup/compose_container_name.py` and its test module
deleted.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
container_name is illegal in swarm mode when replicas > 1 (docker rejects
with "services.deploy.replicas: can't set container_name and X as container
name must be unique"). All compose.yml.j2 templates now route through a
single lookup that suppresses the key in swarm mode and emits it in compose
mode.
SPOT: plugins/lookup/compose_container_name.py
Reads DEPLOYMENT_MODE from the templating context, returns
'container_name: "<name>"' under compose and an empty string under swarm.
Call sites read:
{{ lookup('compose_container_name', MY_CONTAINER) }}
No explicit DEPLOYMENT_MODE pass-through needed.
Sweep: 82 templates rewritten across roles/web-app-*, roles/web-svc-*,
roles/svc-*, plus 4 partial includes (sso_proxy, redis, mariadb, postgres).
For container names assembled from multiple Jinja fragments the args use
tilde concat (X ~ '-' ~ Y) so the lookup receives a single resolved string
and the call stays a single well-formed Jinja expression.
Lint: tests/lint/ansible/jinja/test_no_raw_container_name.py blocks
(a) raw 'container_name:' YAML keys in any roles/*/templates/**/*.j2,
(b) split-Jinja regression where the outer braces close mid-call:
{{ lookup('compose_container_name', A }}-{{ B) }},
(c) embedded-Jinja regression inside a string argument:
lookup(..., "{{ X }}_suffix") (removed in ansible-core 2.23).
The (b) and (c) regexes accept either single- or double-quoted lookup
name and scan the full file so multi-line shapes also surface. Suppress
per-site with '# nocheck: container-name' on the same or preceding line.
Tests: tests/unit/plugins/lookup/test_compose_container_name.py pins the
lookup contract: compose vs swarm output, whitespace tolerance, default-to
compose when DEPLOYMENT_MODE is absent, AnsibleError on missing or excess
terms, non-string name coercion.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Follow-up to the initial web-app-jitsi commit. `make test` flagged 11
contract violations on the newly-added role; each is addressed here.
- group_vars/all/08_networks.yml: declare PORT_BANDS.public.media
(10000–10099) so the JVB media plane port (UDP/10000) falls in band.
- roles/web-app-jitsi/templates/env.j2: pipe every Jicofo/JVB/Jigasi/Jibri
XMPP password through `| dotenv_quote`.
- roles/web-app-jitsi/meta/services.yml: make `email` flag dynamic
(`'web-app-mailu' in group_names`) and add `# nocheck: dynamic-flag`
+ `# nocheck: playwright-service-flag` above `javascript:` (gated on
sso.enabled, not on group_names; runtime DOM patch, no probe-able
surface flag).
- roles/web-app-jitsi/meta/variants.yml: re-add explicit
email.enabled / email.shared pins per variant (variant-coverage rule
requires every dynamic flag to have both true- and false-pinning
variants).
- roles/web-app-jitsi/meta/schema.yml: drop unused jwt_app_secret
credential.
- roles/web-app-jitsi/vars/main.yml: drop unreferenced JITSI_HOSTNAME,
JITSI_WEB_PROTOCOL, JITSI_JWT_APP_SECRET.
- roles/web-app-jitsi/templates/playwright.env.j2: drop unused
ADMIN_EMAIL / JITSI_BASE_URL / OIDC_ISSUER_URL keys and collapse the
PERSONA_*_BLOCKED comment block to one line per key (env-file schema
rule allows at most one comment line above an entry).
- roles/web-app-jitsi/README.md: replace the dead
jitsi-contrib/prosody-plugins/tree/main/token_lobby URL with the
repository root.
Final result: `make test` 5/5 targets ✅ (186 tests OK),
`make autoformat` clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the four-container Jitsi Meet stack (web, prosody, jicofo, jvb) with a
3-variant matrix: V1 oauth2-proxy gate + Keycloak LDAP federation, V2
no-auth anonymous, V3 direct prosody mod_auth_ldap2 against svc-db-openldap.
Auth model is oauth2-proxy at the reverse proxy because Jitsi has no
native OIDC adapter — JWT-via-Keycloak would require a custom token
server outside this repo. The role injects a fixed-position
`[data-injected-logout]` button via templates/javascript.js.j2 so the
no-built-in-logout React SPA satisfies the persona contract's
no-direct-URL logout rule. V2/V3 declare PERSONA_BIBER_BLOCKED /
PERSONA_ADMINISTRATOR_BLOCKED because the shared persona helpers only
model the OIDC redirect chain.
Final FULL-matrix `mode=reinstall full_cycle=true` run green across all
three variants (V1: 6/6 pass, V2/V3: 2 pass + 4 blocked skips).
Also lands utils/parse_trace.py — a small zipfile helper used during
the iteration to extract URL traces from playwright .trace bundles.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
