SOFTWARE_NAME, SOFTWARE_DOMAIN, SOFTWARE_URL and ORGANIZATION are declared once in group_vars/all/00_general.yml and derive from each other. Eleven templates wrote the value instead, so a rename would reach the four declarations and every consumer that reads them, and silently miss the literal.
The lint reads the name from the declaration rather than carrying it, and searches case-insensitively: the domain is the name lowered, the URL is https:// plus the domain and the organization is the name again, so one literal finds the value of all four. A hit is reported with the variable it should have been, derived from the scheme in front of it and its case. The escape is nocheck: software-identity, same line or the one above.
Every migrated template renders the same bytes as before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The list form could publish a host:container pair and nothing else, so thirteen blocks stayed hand-written: the ones needing a protocol suffix, a port range, the swarm long syntax with mode host, or a container-port-only publish. Each of those duplicated numbers the role already declares in meta/services.yml, which is the drift the lookup exists to remove.
The lookup keeps the list form unchanged for its 93 existing call sites and gains a dict form: transport emits one item per protocol, container 'same' serves the services that cannot remap (STUN/TURN, media relays), mode 'host' emits the long block an edge role needs in swarm so the client address survives the hop, publish 'ephemeral' emits the container port alone, and key names the block for Discourse's launcher file, whose expose: list is what becomes docker run -p there. A declared {start, end} mapping renders as start-end on both sides, so a relay range needs no term of its own.
svc-prx-openresty had no ports block at all and bound 80 and 443 as literals in its template, while torrc.j2 carried a third literal 80 for the hidden service. None of the three referenced the others. openresty now declares the ports, the compose template reads them, and the onion's port 80 comes from ports.onion.http through TOR_ONION_EXTRA_PORTS.
Three blocks stay raw, each with the reason it cannot route through one call: keycloak's sso_proxy maps a published port owned by the consuming role to a container port owned by web-app-keycloak, and matrix's ansible flavor takes element on 8080 and federation on 8048 from the upstream playbook rather than from this repo's ports.internal.
The enforcing lint scanned only files with compose in the name and only double-quoted list items, so nine of the thirteen blocks were invisible to it. It now reads every role template that can carry a ports block, catches unquoted items, protocol suffixes, ranges, flow style and the long syntax, and tracks the block by the indentation of its key: keying on the next line closed the block on anything shaped like a key, which target, published, protocol and mode all are. Under expose: only a real mapping counts, because there the single-segment form documents a container port and publishes nothing.
Verified by rendering all 110 container_ports call sites through the real templar, the real lookup and the real merged applications view.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every dashboard persona login spent 25s times the onion multiplier in
clickOidcLoginLink before it clicked anything. The markup probe matched
a[href*='openid-connect'], and the dashboard header renders the Account
dropdown's hidden Logout item, whose Keycloak end-session URL contains
the same string, before its Login link. .first() pinned the hidden item
until the 5s probe expired; the exact-name probe then missed for 20s
because Playwright counts the Font Awesome ::before glyph in the link's
accessible name. The probe now skips logout hrefs and resolves to the
Login anchor the loose probe used to click.
The adapter readiness wait cost every other role 15s times the
multiplier, because only the dashboard sets __oidcLoginReady. The helper
now waits for the load event and then only while the flag is false. The
dashboard loader, an inline head script, declares it false before the
Login link renders; oidc.js still sets it true once the click handler
is wired.
Traces of run 34421944328 show the two probes expiring at 5.07s + 20.00s
on clearnet and 25.01s + 100.00s over Tor. Across runs 34421944328 and
34419507038 the Tor dashboard persona test took 150-156s in all 102
compose runs against 32-34s in the 92 clearnet ones.
Replayed on the dashboard HTML those traces captured, the helper reaches
the Keycloak authorize endpoint in 0.08s instead of 40.09s, and still
waits for the adapter (1.52s) when the flag flips to true after 1.5s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mount pipeline was split across two layers: compose_volumes and
container_volumes lived as filter modules that ansible's filter loader
rejected on every start ("has no attribute 'FilterModule'", two warnings
per run), while the same-named lookups wrapped them thinly. The filter
bodies now live inside their lookups (the tor_socks precedent), the
filter files are gone, and the render tests moved next to the lookup
tests.
As lookups they see the templating context, so the application_id term
every call site passed back to its own role is gone too: both lookups
read application_id from context and fail loudly when it is missing.
107 templates and nfs_prep.yml drop the term; the calls collapse to
lookup('compose_volumes') and lookup('container_volumes', service_name).
The container_volumes call is now canonical - four leading spaces,
service_name in the service position, | indent(4) - enforced by a new
lint (test_container_volumes_call_shape). Normalizing the 85 literal
call sites surfaced that erpnext keyed its meta/volumes.yml mounts by
container name instead of compose service key, the only role to do so;
its meta now uses the service keys. wordpress's column-0 call, which
rendered its first line at top level, is indented like every other.
Template logic that only restated meta/volumes.yml is deleted: mailu fed
compose_volumes an extra_volumes dict whose eight names were looked up
from the very meta file compose_volumes reads itself; zammad's init and
railsserver extra mounts were the volumes.yml entries minus the two
services, which the meta now lists (the when-gated OIDC binds included).
Conditionality that meta cannot express stays in code, but in role-local
lookup plugins instead of template loops: matrix_bridge_mounts renders
the per-bridge registration mounts, nextcloud_extra_volumes the
feature-gated whiteboard/recording volumes (type: volume entries are
declared unconditionally by compose_volumes, so gating must happen
before it).
Parity is proven, not assumed: every mechanically rewritten template is
byte-identical to HEAD after the transformation regexes (95/95), and the
semantic roles render identically through the real functions against
HEAD's meta - zammad across six sso/tls polarities, all seven erpnext
services, mailu's top level. make test-lint passes complete.
Also: suitecrm's compose template pins container_extra_hosts with
host_alias (the SOCKS proxy URL host.docker.internal never resolved in
compose+tor), and the five disabled mautrix bridge addons use the
empty-default config lookup so templating the applications dict stops
emitting 125 errors per batch in every CI job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
75 commits: the socialhome role reimplemented against the project it actually
deploys, native-login specs for funkwhale and shopware, a window.open IdP
handoff and a settings-link fallback in the shared persona helpers, a swarm
converge budget that gives up on a terminal task, and a long tail of role
fixes.
Eleven conflicts. The shared persona helpers (keycloak.js, logout.js) take the
incoming structure — they gained real behaviour this branch never had — with
this branch's resolveTimeout convention re-applied over it and
confirmKeycloakLogoutIfPrompted re-attached at every return point; without it
the Keycloak SSO session outlives the logout. playwright.config.js keeps this
branch's version: both sides set actionTimeout, but ours routes through an env
var and 15s is too short behind an onion. socialhome's spec takes the incoming
content — 2dbd6eb10 found the role described the wrong upstream project
entirely — and keeps gotoOnion and expectHstsWhenTls, since demanding HSTS over
a plaintext onion is a false failure. Both READMEs were regenerated rather than
merged by hand.
Three collisions arrived with no conflict marker at all, because each side
changed a different place in a different file:
- group_vars/all/18_swarm.yml declared swarm_stack_converge_retries twice.
Same value, so nothing breaks today; editing the documented one would have
silently changed nothing.
- svc-net-tor and web-app-socialhome both claimed 192.168.74.0/24. Socialhome,
the newer role, moves to 192.168.75.0/24.
- web-app-socialhome/meta/schema.yml arrived as a new file, so ef5ea7cb5's
rename never saw it. Its credential would not have resolved and the role
would have deployed with an unusable administrator account.
The incoming specs also needed this branch's conventions: raw timeouts wrapped,
four page.goto calls routed through gotoOnion, a request.get given an
onion-aware timeout, and 21 roles marked playwright-service-flag over the tor
block this branch added to them.
make quality green on the merge result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
22 commits, mostly dependabot and docker image-version bumps, plus two fixes:
5153f3030 loads keycloak-js 26 as an ES module, 87a053518 makes the coredns
Corefile readable to the container user.
One conflict, in roles/web-app-dashboard/templates/javascript/oidc.js.j2, resolved
as a union because the two sides touched the same line for unrelated reasons.
edb96e9ae had hoisted the asset URL into KC_JS_ASSET_URL and rewrote it onto the
onion mirror; 5153f3030 changed the path to lib/keycloak.js and declared a local
Keycloak binding, since the adapter now arrives as a module default export instead
of a window global. The onion rewrite stays at the conflict site, the local binding
is kept because loadKeycloakAdapter returns the constructor rather than setting a
global, and the path change moved to the constant's definition.
An import() fetch always runs in CORS mode, which is why 5153f3030 gave the CDN
aca_origin. The onion route reaches the same asset through web-svc-mirror, whose
proxy.conf.j2 already sets that header, so the onion flavor needs no counterpart.
make quality green on the merge result: test-external, test-integration, test-lint,
test-unit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
keycloak-js dropped the UMD build in 26.0.0. The package now ships only
lib/keycloak.js with "type": "module" and a default export, so the CDN
path dist/keycloak.js resolves to a file that no longer exists, and a
classic script tag could not parse the file that does. The adapter never
loaded, window.__oidcLoginReady was never set, and every app job that
co-deploys the dashboard failed its Playwright run on the resulting
console error.
The loader now uses a dynamic import and takes the constructor from the
module's default export. That also retires the window.Keycloak check,
which had nothing left to check once the global stopped existing.
Module fetches always run in CORS mode, unlike classic script loads. The
internal CDN emitted no Access-Control-Allow-Origin at all, so it could
not have served an ES module to the dashboard's origin. jsdelivr sends
"*", which is why the external flavor would have hidden the gap.
Every init option the template passes carries the same default in 24.0.5
and 26.2.4, and the silent-SSO postMessage contract is unchanged, so the
adapter's behaviour is untouched beyond the packaging.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Blocking a persona removes the only end-to-end proof that a role's auth chain
works. docs/contributing/artefact/files/role/playwright.specs.js.md:146 has
always required each flag to carry a one-line nocheck stating the role contract
that justifies it, plus a matching paragraph in the role's README or TODO.
Nothing enforced it, so 89 flags across 46 roles carried neither.
The lint mirrors test_has_spec's shape and uses is_suppressed_at with
mode="line-above", so the marker must sit directly above its flag -- a blank
line between them breaks the association, which is what the persona helper's
reader will look at. A literal =false blocks nothing and is exempt; the
sso-conditional ternary is not, because it blocks in every variant that pins
services.sso.enabled false.
Writing 89 reasons turned out to be the point. Five roles audited in one pass
blocked for five entirely different mechanisms, all of which read as the same
shrug behind a bare =true: pgAdmin's login id is the account e-mail rather than
the Keycloak username; phpMyAdmin authenticates as the MariaDB root account;
GitLab's seeded password belongs to the built-in root, not the administrator;
Mastodon's tootctl accounts create never sets a password; phpLDAPadmin binds
with an LDAP DN and a different secret. Further out: MediaWiki removes the form
that would accept the Keycloak secret ($wgPluggableAuth_EnableLocalLogin=false)
and bounces anonymous requests back into the IdP, so the unauthenticated
landing the contract asserts can never be observed; Mattermost Team Edition has
no native OIDC, so Keycloak rides the GitLab OAuth slot behind a relabelled
link; MinIO's console returns redirectRules: null and renders no SSO button at
all; XWiki's administrator is an XWikiUsers object with no password property.
Two flags turned out to be unjustifiable and are gone rather than annotated.
web-app-mailu gated PERSONA_GUEST_BLOCKED on services.dashboard.enabled, but
runGuestFlow does one anonymous goto and asserts no signed-in marker -- it
never touches the dashboard. A leftover from a helper version that entered
through the dashboard account menu. And web-app-mig's rationale claimed the
role is an in-app module of web-app-dashboard; it has its own compose stack,
healthcheck and canonical domain, and dashboard is merely a tile consumer.
Nine of the reasons say plainly that the block is not a role property but a gap
in the shared persona helpers -- decidim's sign-out is a data-method link the
generic click cannot activate, fider's SSO entry is a two-step modal, taiga,
wordpress, xwiki, odoo and penpot bind OIDC on a path the personas never visit,
jenkins blocks unconditionally although its OIDC variant looks runnable, and
opentalk's own spec proves both personas work. That list did not exist before;
it is the actual yield here, not the 89 comment lines.
45 pre-existing comments came into the comment lint's scope because this sweep
touched their files -- it only inspects files that differ from HEAD. 28 of them
were the banner "Service gate flags rendered for closure.", every one sitting
above a line that is not a service gate flag at all, orphaned by the
alphabetical re-sort these templates once got.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven include templates under sys-svc-container decided how a container was
probed, and they read their inputs from template-local `{% set %}` scope.
That scope leaks: a value set for one service silently reached every service
below it, which is how taiga's front and gateway ended up probing a port
declared for a different service, and how bluesky's frontend probed a PDS
endpoint.
Probes now live in utils/docker/healthcheck.py, one class per flavor behind a
PROBES registry, and each service declares its own:
healthcheck:
flavor: curl
path: health/ready
start_period: 20m
`port_key` picks another entry under ports.internal for the services that
expose their probe elsewhere (keycloak on management, nextcloud talk on
signaling). Without a flavor the entry must carry an explicit test argv, so a
missing declaration fails loudly instead of rendering a probe against nothing.
Rendering HEAD's templates against the new output confirms 49 of 56 call
sites are byte-identical; the remaining seven are covered by an Ansible run.
Three probes change on purpose: erpnext's backend gains a check it never had,
bluesky's frontend drops the leaked PDS path, and msmtp_curl moves from a
hardcoded localhost:80 to the service's own port on 127.0.0.1 like every
other flavor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The asset lookup hardcoded https for the internal CDN origin, but
.onion vhosts are plaintext with no 443 listener, and the dashboard
mirror-rewrite sent even onion-local asset URLs through the mirror
(404). The adapter never loaded, so the OIDC code was never exchanged
and Login stayed visible after the round-trip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add lookup('asset', app, pkg, path) and lookup('asset_host'): both resolve
the asset origin through one shared resolve_host, returning the internal
CDN domain when web-svc-cdn is deployed with flavor internal and
cdn.jsdelivr.net otherwise, so template URLs and CSP origins can never
diverge. Versions come from the role's committed files/package-lock.json.
Roles declare frontend dependencies in files/package.json; web-svc-cdn
aggregates every role lockfile, runs one npm ci per role in a node
container and mirrors the packages under /npm/<pkg>@<version>/ following
jsdelivr's path scheme, so internal and external URLs differ only by host.
Convert web-app-dashboard (keycloak-js 24.0.5 UMD) and web-app-littlejs
(bootstrap, fontawesome) to the asset lookup, drop the vendored keycloak
blob, and add a repository lint forbidding hardcoded external CDN hosts
in templates and CSP metas; pre-existing external CDN entries are checked
out with nocheck markers and a migration TODO.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The applications menu and the platform/company header interpolated operator and role free text as plain YAML scalars, so any description containing a colon-space broke the rendered config at the from_yaml parse in the branding-asset task; moodle variant 0 hit it as the only deploy combining an active dashboard with web-app-lam, whose description carries 'directories: managing'. All name, description, icon, url and title fields now render through to_json, which yields quoted scalars for arbitrary text, matching what to_nice_yaml already did for the cards block.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New web-svc-mirror role: an nginx vhost that proxies and caches exactly
the external asset hosts aggregated from all deployed applications' CSP
whitelists (external_asset_origins filter) under first-party paths
(/<origin-host>/...), refusing every other upstream. svc-net-tor
consumes the service, so any tor deployment automatically serves the
mirror on its onion.
For apps with services.tor.enabled the shared body filter rewrites
external asset URLs in HTML to the onion mirror (onion sessions only;
clearnet untouched) and the CSP builder allows the mirror origin in the
asset directives. The dashboard's keycloak-js vendoring is reverted:
onion sessions now load the adapter through the mirror, clearnet loads
jsdelivr directly as before.
Verified: variant=0 deploy green (failed=0), dashboard 14/14 over onion
with keycloak-js served through the mirror, mirror spec green, health
check accepts the 404 vhost root.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Chromium never treats plaintext .onion origins as a secure context, so
keycloak-js (PKCE via crypto.subtle) could not run. A custom Playwright
browser fixture launches Firefox with the Tor-Browser prefs
(dom.securecontext.allowlist_onions, origin/mixed-content handling) for
onion targets and keeps Chromium for clearnet; all dashboard specs share
it via the register pattern.
keycloak-js is now vendored from the deployment's own CDN as an ES
module (jsdelivr is unreachable over Tor) with CORS opened on the CDN
proxy. Personas and the oidc-login spec wait for adapter readiness and
navbar settle before driving login/logout, which the async code->token
exchange needs over Tor latency.
Verified: full deploy matrix green (onion 14/14 twice, clearnet pass).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
port-ui prefetches every card icon over HTTPS at boot before it binds, so the 30s tcp-healthcheck default reaps it mid-prefetch on loaded swarm workers and the stack never converges. Set container_healthcheck_start_period to 10m (matching the heavier siblings).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The client-side iframe.js postMessage handler validated message origins and redirect targets against a single .DOMAIN_PRIMARY suffix, which rejected the onion mirror domain and broke iframe sync over Tor. Allow the svc-net-tor node suffix alongside clearnet when tor is deployed; both the origin and redirect checks stay restrictive.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a lint that rejects hardcoded container_port literals and the
host-published ports.local port; container_port is the port the container
listens on, so it must resolve from
lookup('config', application_id, 'services.<svc>.ports.internal.<name>')
or a constant built that way. Bring 42 roles into compliance: move each
literal into services.<svc>.ports.internal.<name> and reference it via the
lookup, using entity_name/service_name where they equal the service key.
Values are preserved (mattermost/prometheus pointed at ports.local with the
same value as internal). Drop the dead container_port from fider and
prometheus, and the narrative comments the edits pulled into the comment
lint's changed-file scope.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every compose.yml.j2 now declares services as '{{ service_name }}:' with
its '{% set %}' knobs on two-space lines above the key, includes at four
spaces and if/for tags aligned to the content they wrap. The layout
renders correctly because stack_host_template enables lstrip_blocks for
compose.yml.j2; indent(4, True) in the container base and '{%+' guards
in whitespace-sensitive scalars keep the shared fragments identical
under both render modes. A new lint test enforces the skeleton.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Convert the two registry-driven networks templates into pure-Python
rendering functions plus thin Ansible lookup plugins:
roles/sys-svc-compose/templates/networks.yml.j2 -> {{ lookup('compose_networks') }}
roles/sys-svc-container/templates/networks.yml.j2 -> {{ lookup('container_networks') | indent(4) }}
The rendering logic lives in utils/networks/render.py as pure-Python
functions that take a registry + closure-style config/database lookups,
making the logic directly unit-testable without a Jinja stub harness.
The lookup plugins are thin wrappers that resolve application_id,
DEPLOYMENT_MODE, and swarm.network.encryption from the templar, build
the service_registry, and forward to the render functions.
Migrate 233 include lines across 85 production templates. Convert
web-app-matrix's hand-coded MDAD networks block (default + conditional
postgres + conditional openldap) to the generic lookup; this adds an
openresty attachment when services.sso.enabled (harmless extra
interface, pinned in test_matrix_mdad_consumer_attaches_postgres_ldap_sso).
Mark local-DB / redis sidecars (sys-svc-rdbms/templates/services/,
svc-db-redis/templates/service.yml.j2) with `# nocheck: networks-literal`
because they intentionally attach to default-only.
Extend the lint to scan ALL roles/*/templates/**/*.yml.j2 (not only
compose.yml.j2) and to forbid the legacy include form; the marker
above is the documented escape hatch.
Coverage:
- tests/unit/utils/networks/test_render.py (30 cases incl. encrypted=false literal pin)
- tests/unit/plugins/lookup/test_compose_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/unit/plugins/lookup/test_container_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/integration/infrastructure/compose/test_networks_render.py (10 snapshot scenarios)
- tests/integration/infrastructure/compose/test_network_includes.py (3 lint rules)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add plugins/lookup/container_image.py: returns the full `image: "<ref>"`
line so templates drop the redundant image: prefix and surrounding
quotes. Supports `custom=True` to auto-construct `<entity_name>_custom`
for locally-built images, plus explicit `image=`/`version=` overrides
for exotic cases (taiga back-auth/front-oidc variants, decidim infinito
suffix, etc.).
* 25 unit tests for the lookup cover compose vs swarm prefix, custom
flag, overrides, tag_only=True, missing version warnings.
* New strict lint (tests/lint/.../test_no_raw_image_in_compose_templates)
fails on any raw `image:` line in roles/*/templates/*.j2 unless
nocheck-suppressed. Legacy `image: "{{ lookup(...) }}"` form no
longer accepted.
* Migrate 159 image refs across ~80 roles to the standalone
`{{ lookup('container_image', application_id, '<service_key>') }}`
form. 26 roles use `custom=True` (mariadb, postgres, openldap,
jenkins, jira, gitlab, nextcloud, mediawiki, shopware, wordpress,
odoo, joomla, openproject, bookwyrm, espocrm, hugo, littlejs,
fediwall, roulette-wheel, openresty, confluence and more).
* Fix svc-db-redis application_id leak: service.yml.j2 is included
into consuming roles where application_id evaluates to the outer
role, so the image lookup needs the hardcoded 'svc-db-redis'.
* Integration tests (test_templates_require_image,
test_build_requires_image) recognise the standalone lookup as an
implicit `image:` at the same indent.
* Mediawiki swarm tasks: add explicit `:{{ MEDIAWIKI_VERSION }}` to
the registry-prefixed image string because the custom var is now
bare.
Only 3 legitimate exotic nocheck-suppressed image lines remain:
matrix-mdad runner (nested services.matrix.runner.image path),
matrix mautrix bridges (per-iteration dynamic) and svc-registry-docker
(chicken-and-egg self-reference).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Migrate every role's meta/volumes.yml from ad-hoc dict-of-named-volumes
to a canonical dict-of-dicts schema where the YAML key is the semantic
short name and the entry body carries type / name / source / mounts /
nfs / mode / read_only. The optional 'name:' field is the docker volume
name (defaults to the key); the previous 'docker_name:' field is gone.
What's in this commit (210 paths):
Infrastructure (23)
- New filter plugins/filter/{compose,container}_volumes.py — render the
top-level volumes:/configs:/secrets: section and per-service mount
blocks from the canonical meta. container_volumes uses the SEMANTIC
name as per-service mount source so it resolves against compose_volumes'
top-level dict key (docker_name as source produced 'undefined volume'
on swarm).
- New lookup plugins/lookup/{compose,container,volume}_volumes.py.
volume(application_id, name) returns a normalised single entry;
volume(application_id) returns the whole canonical dict.
- utils/cache/applications.py keeps a sibling registry
_CANONICAL_VOLUMES_BY_ROLE outside the merged applications payload so
raw Jinja source: strings never enter the templar walk (root-cause of
the 200x cache-miss + recursive get_merged_applications storm).
- utils/roles/applications/mounts.py — schema validator + normaliser;
list-shape is rejected.
- plugins/lookup/nginx.py: _volume_path_by_name reads canonical directly.
- scripts/tests/deploy/swarm/_context.sh: NFS_VOLUMES extraction handles
both legacy dict and canonical dict shapes.
- roles/svc-prx-openresty/handlers/restart_and_validate.yml: gate
compose-only validate+reload behind a single when-block; swarm uses
service-update instead (bind-mounts of a FILE pin source inode at
task-create time, so atomic write-tmp+rename leaves the running task
on a deleted inode).
- roles/sys-svc-compose/tasks/utils/nfs_prep.yml: switch to
lookup('volume', application_id) | dict2items so NFS subdir
pre-creation works under the new cache split.
Tests (12)
- New strict lints flagging inline mount sections in compose templates,
single-file bind mounts, and legacy lookup('config', ..., 'volumes.X.Y')
call sites — all volume-mount surfaces MUST flow through container_volumes
/ compose_volumes / volume lookups.
- Integration: test_volumes_compose_chain + test_volume_constants_are_lookups
accept both the legacy and the new lookup('volume', app, 'X').name forms
during the migration window.
- Unit tests for the new lookup, filter, mounts helper, plus updated
nginx lookup test.
- Prometheus role-wiring test reads canonical meta for the
alert_rules.yml bind mount.
Role migrations (175)
- 62 meta/volumes.yml converted to dict-of-dicts canonical shape
- 63 templates/compose*.yml.j2 replace inline volumes:/configs:/secrets:
with lookup('container_volumes', application_id, '<service>') |
indent(N) per service + a single trailing lookup('compose_volumes',
application_id) for the top-level block
- 50 vars/main.yml drop dead volume constants and refactor 23 remaining
legacy lookup('config', ..., 'volumes.X.Y') to lookup('volume',
application_id, 'X').Y across 9 roles
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add the compose_replicas lookup plugin as the single source of truth for
swarm replica calculation. Default emits `replicas: len(groups[application_id])`
(topology-driven); override term forces an explicit value. In compose mode
the lookup returns the empty string because docker-compose ignores
deploy.replicas.
Refactor deploy.yml.j2 to use the lookup. The previous service_replicas
default in defaults/main.yml is no longer needed and is dropped.
Add a lint test that walks every roles/*/templates/compose*.yml.j2 and
verifies each service entry under the top-level services: declares
replica handling via one of three valid paths:
1. Include sys-svc-container/templates/base.yml.j2 (full inheritance).
2. Include sys-svc-container/templates/deploy.yml.j2 directly (escape
hatch for services that own their restart/logging/env_file and
therefore can't use base.yml.j2 without duplicate-key conflicts).
3. Call lookup('compose_replicas', ...) explicitly for handcrafted
deploy: blocks.
Fix the 7 web-app compose templates flagged by the new lint (bluesky,
dashboard, magento, matrix, moodle, nextcloud, opentalk) by including
deploy.yml.j2 directly and setting service_name as a SPOT before each
service entry, so the service name in the yaml key and the resource_filter
lookups in deploy.yml.j2 both flow from one variable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Single SPOT lookup `plugins/lookup/compose_volumes.py` absorbs the
applications-lookup pipe step and auto-wires `DEPLOYMENT_MODE` plus
the `storage` group var from the templating context. Callers only
pass what genuinely varies per service (`application_id` and an
optional `extra_volumes` dict).
Behavior change (bug fix): the filter was always invoked with the
default `deployment_mode='compose'` because no caller passed it
explicitly, which silently skipped the swarm-NFS driver-opts
injection path. The lookup reads `DEPLOYMENT_MODE` from globals,
so swarm-NFS volumes now render the `driver: local` / `driver_opts:
type: nfs` block correctly. Callers can still override via
`deployment_mode=` / `storage=` kwargs when a template needs
non-default rendering.
Other coordinated changes:
- 81 template call sites migrated from pipe form to lookup form
(single-line and multi-line shapes).
- `FilterModule` registration in plugins/filter/compose_volumes.py
removed so the deprecated pipe form fails fast at render time.
The underlying `compose_volumes()` function stays exported and is
imported by the new lookup plus the existing 21-test filter unit
test module.
- 10 new lookup unit tests pin the contract (term shape, kwargs
pass-through, auto-wiring, override precedence, error cases).
- Integration test `test_compose_volumes_call_required.py` updated:
asserts the new lookup form is present in every `compose.yml.j2`
and adds a second test forbidding the deprecated pipe form
anywhere under `roles/**/templates/`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace per-key `compose_container_name` lookup with a generic
`compose_only(key, value)` SPOT that emits `key: "<value>"` only in
compose mode and nothing in swarm. Same gating logic now serves
container_name (157 call sites) and pull_policy (17 call sites)
without duplicate plugin code.
Why this consolidation:
- The earlier `build-key` lint rule was misdirected. Swarm tolerates
`build:` with a harmless warning when `image:` is pre-pushed (via
the existing handlers/swarm_registry_push.py flow in
roles/sys-svc-compose). The actual swarm-rejected key is
`pull_policy:` ("Additional property pull_policy is not allowed"),
and 10 templates were silently broken because they declared inline
`pull_policy: never` outside the existing `{% if DEPLOYMENT_MODE
\!= 'swarm' %}` gate in roles/sys-svc-container/templates/build.yml.j2.
- A single generic lookup is cleaner than N per-key wrappers when
the gating logic is identical (emit-key-or-nothing).
Lint changes in tests/lint/ansible/jinja/test_mode_incompatible_keys.py:
- New `pull-policy-key` rule, regex `^\s*pull_policy\s*:`, remediation
points at compose_only.
- Existing `container-name` rule keeps its raw-key regex; remediation
now also points at compose_only.
- Broken-split / broken-embed regexes rekeyed on the `compose_only`
lookup name (was `compose_container_name`).
- `build-key` rule and its 12 nocheck annotations across templates
removed.
Coordinated changes:
- 86 templates rewritten (157 container_name calls migrated;
17 inline pull_policy lines replaced).
- 1 redundant `{% if DEPLOYMENT_MODE \!= 'swarm' %}` guard dropped in
build.yml.j2 (the lookup self-gates).
- 12 unit tests pin the compose_only contract.
- Catalog (suppression.md) updated: `pull-policy-key` row added,
`container-name` row remediation updated.
- Old `plugins/lookup/compose_container_name.py` and its test module
deleted.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mirror compose_container_name pattern: plugins/lookup/compose_restart.py
emits `restart: <policy>` only in compose mode. In swarm the key is
silently ignored (replaced by deploy.restart_policy) and produces
"Ignoring unsupported options" warnings at docker stack deploy time.
Sweep 19 raw restart: sites across 10 templates to the no-arg form
`{{ lookup('compose_restart') }}` (the lookup falls back to
DOCKER_RESTART_POLICY from ansible-vars).
base.yml.j2 keeps the explicit-arg form
`{{ lookup('compose_restart', docker_restart_policy | default(DOCKER_RESTART_POLICY)) }}`
so the 5 templates that override the policy via Jinja-scope
`{% set docker_restart_policy = ... %}` (erpnext, zammad, shopware,
matomo, openproject) retain their override path. Jinja-scope bindings
are visible only in the rendering template, never in the Ansible
variable context that the lookup sees, so the caller must resolve them.
Generalize the container-name lint into a data-driven multi-rule
scanner at tests/lint/ansible/jinja/test_mode_incompatible_keys.py
covering both container-name and restart-key. Adding a new rule is one
ModeRule row sharing the file-walk / suppression / line-grouping
scaffolding.
Also gate roles/svc-db-redis/templates/service.yml.j2's
resource.yml.j2 include with DEPLOYMENT_MODE \!= 'swarm'. Mirrors
base.yml.j2's existing pattern; prevents cpus / mem_limit /
mem_reservation / pids_limit from leaking into swarm output where
deploy.resources.* in deploy.yml.j2 already handles them.
11 unit tests pin the compose_restart lookup contract. Both rules
registered in docs/contributing/actions/testing/suppression.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
container_name is illegal in swarm mode when replicas > 1 (docker rejects
with "services.deploy.replicas: can't set container_name and X as container
name must be unique"). All compose.yml.j2 templates now route through a
single lookup that suppresses the key in swarm mode and emits it in compose
mode.
SPOT: plugins/lookup/compose_container_name.py
Reads DEPLOYMENT_MODE from the templating context, returns
'container_name: "<name>"' under compose and an empty string under swarm.
Call sites read:
{{ lookup('compose_container_name', MY_CONTAINER) }}
No explicit DEPLOYMENT_MODE pass-through needed.
Sweep: 82 templates rewritten across roles/web-app-*, roles/web-svc-*,
roles/svc-*, plus 4 partial includes (sso_proxy, redis, mariadb, postgres).
For container names assembled from multiple Jinja fragments the args use
tilde concat (X ~ '-' ~ Y) so the lookup receives a single resolved string
and the call stays a single well-formed Jinja expression.
Lint: tests/lint/ansible/jinja/test_no_raw_container_name.py blocks
(a) raw 'container_name:' YAML keys in any roles/*/templates/**/*.j2,
(b) split-Jinja regression where the outer braces close mid-call:
{{ lookup('compose_container_name', A }}-{{ B) }},
(c) embedded-Jinja regression inside a string argument:
lookup(..., "{{ X }}_suffix") (removed in ansible-core 2.23).
The (b) and (c) regexes accept either single- or double-quoted lookup
name and scan the full file so multi-line shapes also surface. Suppress
per-site with '# nocheck: container-name' on the same or preceding line.
Tests: tests/unit/plugins/lookup/test_compose_container_name.py pins the
lookup contract: compose vs swarm output, whitespace tolerance, default-to
compose when DEPLOYMENT_MODE is absent, AnsibleError on missing or excess
terms, non-string name coercion.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Rename credential basic_auth_password → api_bot_password and var
ZAMMAD_ADMINISTRATOR_PASSWORD → ZAMMAD_API_BOT_PASSWORD to reflect
that the value is the role-internal bot's local password, not the
administrator's (reused for the wizard-bypass admin only because both
need the same shell-safe alphanumeric form).
- Introduce ORGANIZATION constant (defaults to SOFTWARE_NAME) and migrate
organization-string consumers (mattermost team, shopware shop title,
dashboard login/logout descriptions, simpleicons "provided by",
user-administrator description/lastname).
- Add HOST_LL_CC_TAG (BCP-47 form, e.g. en-gb) and use in auto_wizard.
- Add NETWORK_TRUSTED_PROXY_CIDRS SPOT and consume from suitecrm,
shopware and zammad reverse-proxy configs.
- Extract ZAMMAD_WEB_PROTOCOL via lookup('tls', …, 'protocols.web')
instead of inline 'https' if … else 'http'.
- Use database_query for CREATEDB grant so it works on shared+non-shared
databases; back the SQL with files/sql/grant_createdb.sql SPOT.
- Read administrator firstname/lastname from users lookup in
auto_wizard.json.j2 instead of hardcoding.
- Drop redundant elasticsearch: block from web-app-zammad services.yml
(no consumer reads it).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements docs/requirements/021-sso-flavor-migration.md atomically:
* Schema: every role's meta/services.yml replaces the parallel
`oauth2:` / `oidc:` top-level blocks with a single `sso:` block
carrying `enabled`, `shared`, and a `flavor` discriminator
(`oidc` | `oauth2` | `saml`; default `oidc`). Flavor-specific
sub-keys live under `sso.<flavor>.*` (`sso.oauth2.{origin,acl,
allowed_groups}`, `sso.oidc.plugin`).
* Provider role: roles/web-app-oauth2-proxy/ is deleted in its
entirety; its 5 sidecar templates move under
roles/web-app-keycloak/templates/sso_proxy/, and the per-consumer
rendering task lives at roles/web-app-keycloak/tasks/sso_proxy.yml.
web-app-keycloak now declares `provides: sso`.
* Runtime: new resolver utils/roles/applications/services/sso.py
centralises the SSO predicates (`is_enabled`, `is_proxy_gated`,
`is_oidc_native`) and the oauth2 sub-values
(`oauth2_origin_{host,port}`, `oauth2_acl`, `oauth2_allowed_groups`).
Three lookup plugins consume the resolver: `sso` (single-app
predicate access), `sso_proxy_consumers` (enumerates oauth2-flavored
consumers), `sso_oidc_plugin` (renamed from `oidc_flavor`, Nextcloud
plugin selector). sys-stk-backend, sys-svc-compose, sys-svc-proxy
and plugins/filter/compose_volumes.py all gate on
`lookup('sso', ..., 'is_proxy_gated')` instead of manually combining
`enabled` and `flavor`.
* Identifier renames repo-wide: `services.<entity>.ports.local.oauth2`
→ `ports.local.sso`; `PORT_BANDS.local.oauth2` → `PORT_BANDS.local.sso`;
`OAUTH2_PROXY_*` env/Jinja vars → `SSO_PROXY_*`;
`oauth2_proxy_cookie_secret` → `sso_proxy_cookie_secret`;
`OAUTH2_SERVICE_ENABLED` + `OIDC_SERVICE_ENABLED` collapse into
`SSO_SERVICE_ENABLED`; Playwright gating helpers
(isServiceEnabled/skipUnlessServiceEnabled/requireService/safeIsEnabled/
safeSkipUnlessEnabled) move from `oidc`/`oauth2` to `sso`.
* Dual-block resolution: per req's Background-Table, web-app-bookwyrm
is `flavor: oauth2`; web-app-gitea and web-app-friendica keep
`flavor: oauth2` with their existing `oauth2.acl.blacklist`
(/user/login) and `oauth2.acl.whitelist` (federation + discovery
endpoints) — preserving the operative oauth2-proxy gating.
* Tests: the two mutual-exclusion guards
(tests/integration/iam/oauth2_oidc/test_mutual_exclusive.py and
test_acl_mutual_exclusive.py) are deleted per Decision #10;
test_oauth2_contract.py is renamed and rewritten to
test_sso_contract.py, asserting that `flavor: oauth2` roles carry
`oauth2.origin.{host,port}` plus `ports.local.sso`; test_sso.py
(web-app integration) now requires the unified `sso:` block with an
explicit flavor or a `# nocheck: sso` opt-out; new
tests/lint/repository/no_legacy_sso_paths/ guards the source tree
against re-introducing the legacy strings.
* Docs: req 21 itself has every AC checked off, and reqs 006, 008,
009, 013, 015, 016, 019, 020 plus 10+ role READMEs and
docs/contributing/actions/testing/suppression.md are updated to the
unified SSO naming.
Empirically validated end-to-end against the validation app set
(web-app-keycloak, web-app-nextcloud, web-app-bookwyrm, web-app-gitea,
web-app-friendica, web-app-dashboard) in matrix variants 0 and 1; v2
also passes the 4 target roles' Playwright after a documented
pre-existing KC_HOSTNAME-drift workaround (separate concern, not
caused by this refactor).
BREAKING CHANGE: every consumer of `services.oauth2.*`,
`services.oidc.*`, `OAUTH2_PROXY_*`, `OAUTH2_SERVICE_ENABLED`,
`OIDC_SERVICE_ENABLED`, `oauth2_proxy_cookie_secret`,
`oauth2_proxy_application_id`, `ports.local.oauth2`, the
`web-app-oauth2-proxy` role, or the legacy gating helper service names
(`oidc`/`oauth2`) must migrate to the unified `sso` shape documented
in docs/requirements/021-sso-flavor-migration.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Dashboard card icons were rendered with `http://172.17.0.1:8044/<slug>.svg`,
the in-cluster sync URL that only the Ansible runner can reach. The
browser saw `NS_ERROR_CONNECTION_REFUSED` for every Simple Icons card.
The `add_simpleicon_source` filter now takes two URLs: it probes the
local sync URL (plain HTTP, no TLS, no redirect chain — reachable
from the deploy container's `requests` without a CA bundle) and
writes the public `lookup('tls', 'web-svc-simpleicons', 'url.base')`
URL into `icon.source`. Cards render `<img src="https://icon.<DOMAIN>/<slug>.svg">`,
which is reachable from the browser and CSP-permitted (`img-src *`).
Drops the dead local-cache scaffolding (static-subdir vars, volume
mount, `Ensure directory` task, refresh-download step) since the
browser now fetches Simple Icons directly. Renames the
previously-static `templates/style.css.j2` to `files/style.css`
(no Jinja markers, matches the new template-uses-jinja lint warning).
Adds a Playwright spec `test-icons.js` that asserts the regression:
no card may still reference `172.17.0.1`, at least one card must
resolve under `icon.<DOMAIN>`, with a favicon fallback path when the
Simple Icons service is disabled.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Rename env/static.env → env/default.env across the entire repo
(string label, doc links, Python file-path constants) — the
`parse_static_env*` function names stay as a stable API surface.
Sort all `*.env` / `*.env.j2` entries alphabetically (52 in
default.env, 4 in ci.env, 46 of 49 j2 templates; 3 templates
with Jinja2 `{% %}` control flow are skipped because conditional
blocks would tear apart under a naive sort). Sort all 83
Makefile target blocks alphabetically; .PHONY group headers
stay in place.
Collapse the two remaining multi-line Makefile recipe comments
(dotenv-force, test-signed) to single-line WHY comments.
Add four lint tests under tests/lint/filesystem/ to keep the
invariants enforced going forward:
- env/test_alphabetical_sort.py
- env/test_single_line_comments.py (# nocheck: markers exempt)
- makefile/test_alphabetical_targets.py
- makefile/test_single_line_comments.py (# nocheck: markers exempt)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Browser vendors deprecated X-Frame-Options/X-XSS-Protection/X-Download-Options/
X-Permitted-Cross-Domain-Policies; CSP (rendered from each role's
meta/server.yml via build_csp_header) is the modern replacement. To make the
SPOT actually win against upstream apps (e.g. Keycloak/Quarkus still emits
X-Frame-Options), the proxy now hides those headers on every location type.
Includes a fail-loud lint that catches any future add_header / Header set/add
re-introduction in .conf / .conf.j2 / .htaccess / .j2 / .jinja / .tmpl files.
- sys-svc-proxy: new upstream_strips.conf.j2 partial bundles HSTS + the 4
deprecated header strips; included in html/media/ws/upload + web-svc-logout
+ web-app-oauth2-proxy locations (nginx array-replacement inheritance forces
per-location placement)
- web-app-keycloak: meta/server.yml frame-ancestors allows the primary-domain
wildcard with the dynamic TLS protocol so dashboard silent-check-sso iframe
loads cleanly
- web-app-dashboard/playwright: drop isOidcSilentCheckNoise — root cause is
now fixed, no noise filter needed
- web-app-dashboard/nginx + web-app-nextcloud/nginx: remove the deprecated
add_header lines themselves
- tests + docs: new lint registers `deprecated-http-header` suppression rule
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Console cleanliness moved to its own scenario
---------------------------------------------
``test-header-navbar-logos.js`` failed in CI when port-ui's Keycloak
silent-check-sso iframe was blocked by frame-ancestor CSP — the
``expectNoUnexpectedDiagnostics`` call at the test's end caught a
console.error logged by chrome for the blocked iframe (chrome-error://
chromewebdata + ``Framing 'https://auth.…'``) and failed every retry.
That diagnostic was a generic "no surprises" tail, unrelated to the
logo assertions the test is named after.
- Drop the diagnostic-tail call from the logo test; it now does
exactly what its name says.
- New ``test-no-console-noise.js`` runs the diagnostic check as its
own scenario, with two noise filters enabled
(``ignoreMatomoConsoleNoise`` + ``ignoreOidcSilentCheckNoise``).
- New ``isOidcSilentCheckNoise`` helper in ``_shared.js`` matches the
``chrome-error://chromewebdata`` URL plus the ``Framing '…'`` /
``silent-check-sso`` text patterns. These are test-env config
artefacts (Keycloak ``frame-ancestors`` not whitelisted), not
dashboard regressions.
- ``expectNoUnexpectedDiagnostics`` learns a corresponding
``ignoreOidcSilentCheckNoise`` option.
_shared.js no longer hides single-consumer code
-----------------------------------------------
Audited every export of ``_shared.js`` against every ``test-*.js``
consumer; ten symbols had exactly one caller and were therefore not
"shared". Moved each to the test file that owns it:
- ``runGuestFlow`` / ``runBiberFlow`` / ``runAdminFlow`` →
matching ``test-*-persona.js`` (now ``require("./personas")`` direct)
- ``assertCspMetaParity`` / ``assertCspResponseHeader`` /
``expectNoCspViolations`` / ``installCspViolationObserver`` plus
``canonicalDomain`` → ``test-csp-canonical-domain.js`` (CSP observer
install is now a local ``test.beforeEach`` in that file)
- ``dashboardTargetRoles`` parser → ``test-tile-reachability.js``
- ``cdnBaseUrl`` + each prefix (sharedCss / roleCss / sharedJs) →
``test-shared-css.js`` / ``test-cdn-role-stylesheet.js`` /
``test-logout-js-injection.js``
- ``platformLogoUrl`` + ``platformFaviconUrl`` →
``test-header-navbar-logos.js``
- ``appBaseUrl`` / ``oidcIssuerUrl`` / ``loginUsername`` /
``loginPassword`` / ``expectedOidcAuthUrl`` → ``test-oidc-login.js``
Dropped completely as dead test code: ``companyLogoUrl`` (no
spec ever read it) and its ``COMPANY_LOGO_URL`` env render in
``playwright.env.j2``. ``DASHBOARD_COMPANY_LOGO_URL`` in
``vars/main.yml`` stays — ``config.yaml.j2`` still consumes it for
the port-ui company.logo.source field.
``_shared.js`` is down to genuinely-shared state: ``dashboardJsBaseUrl``
+ ``matomoBaseUrl`` (each ≥ 2 callers), the diagnostic plumbing
(``attachDiagnostics`` / ``expectNoUnexpectedDiagnostics`` /
``waitForDashboardReady`` / ``waitForResourceResponse``), the noise
predicates, ``beforeEach``, and the ``skipUnlessServiceEnabled``
re-export. 116 LOC of state/exports collapse to 63.
Hitchhikers
-----------
- ``vars/main.yml`` loses an outdated docstring above the branding
SPOTs (the SPOTs themselves stay; ``config.yaml.j2`` consumes them).
- ``sys-svc-compose/defaults/main.yml``: linter re-joined the
multi-line ``compose_repository_pull`` default onto one line; pure
formatting, no semantic change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Playwright spec split
---------------------
The 747-line ``playwright.spec.js`` monolith is split into a thin
orchestrator + ``_shared.js`` (env, ``beforeEach``, common helpers) +
one ``test-*.js`` per scenario, matching the convention used by
web-app-fediwall/friendica/nextcloud. Each scenario file holds its own
specialised helpers (e.g. the auth/dropdown helpers live in
``test-oidc-login.js``, image-loaded assertions live in
``test-header-navbar-logos.js``); only helpers used by ≥ 2 tests stay
in ``_shared.js``. ``dashboard-card-flow.js`` becomes
``test-tile-reachability.js`` (git-detected rename) — the
``runDashboardCardScenario`` helper had a single consumer, so the
indirection is gone.
SPOT for branding URLs
----------------------
The ``'https:' ~ source if source.startswith('//') else source``
transformation lived inline in ``config.yaml.j2`` and was duplicated in
``playwright.env.j2`` when the logo assertion was added. Three new
SPOTs in ``vars/main.yml``:
- ``DASHBOARD_PLATFORM_LOGO_URL``
- ``DASHBOARD_PLATFORM_FAVICON_URL``
- ``DASHBOARD_COMPANY_LOGO_URL``
Both templates now reference these vars; the inline transformations
and the now-redundant ``{% set ... %}`` lines in ``config.yaml.j2``
are removed. The slurp + parse of the rendered config in
``tasks/01_core.yml`` stays because it covers the host-override case
(``DASHBOARD_CONFIG_INV_PATH`` template may use different branding),
but is now documented as the override-discovery path rather than the
default source of truth.
port-ui 2.0.0 + test alignment
------------------------------
- ``meta/services.yml``: port-ui pinned ``1.2.0 → 2.0.0`` (probe-first
asset resolver, see kevinveenbirkenbach/port-ui#2.0.0).
- ``test-header-navbar-logos.js``: drop the ``asset`` skip-gate (port-ui
>= 2.0.0 embeds the URL directly instead of producing the broken
``/static/<absolute-url>``), replace the shape regex with an
exact-match against the SPOT ``PLATFORM_LOGO_URL`` env var, and add
a favicon assertion that checks the rendered ``<link rel="icon">``
href against the SPOT ``PLATFORM_FAVICON_URL``.
- ``test-cdn-role-stylesheet.js``: keep the ``asset`` skip-gate — that
test exercises ``web-svc-cdn`` which depends on ``web-svc-asset`` as
its file backend, unrelated to the port-ui bug.
- ``playwright.env.j2``: rendered ``ASSET_SERVICE_ENABLED``,
``PLATFORM_LOGO_URL``, ``PLATFORM_FAVICON_URL``, ``COMPANY_LOGO_URL``.
- ``meta/services.yml``: drop the now-stale
``# nocheck: playwright-service-flag`` annotation on the ``asset``
service block.
.claude/settings.json
---------------------
Resort ``permissions.allow`` after the runtime appended
``WebFetch(domain:cloud.veen.world)`` at the end of the array during
the asset-resolver work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Removes direct process.env.<X>_SERVICE_ENABLED reads (and bespoke
per-role <ROLE>_OIDC_ENABLED / SERVICE_* / OIDC_ENABLED env flags) in
favor of skipUnlessServiceEnabled/isServiceEnabled from
roles/test-e2e-playwright/files/service-gating.js, per
docs/contributing/artefact/files/role/playwright.specs.js.md service-
gating contract.
Per-role changes:
- discourse, matrix, dashboard: replace process.env.<X>_SERVICE_ENABLED
with the helper; drop now-redundant nocheck markers on the canonical
env flags.
- taiga: drop TAIGA_OAUTH2_ENABLED / TAIGA_OIDC_ENABLED env lines;
consume OAUTH2_SERVICE_ENABLED / OIDC_SERVICE_ENABLED via the helper.
- opentalk, opencloud, nextcloud: drop <ROLE>_OIDC_ENABLED (and
NEXTCLOUD_LDAP_ENABLED) env lines; spec switches to
isServiceEnabled('oidc'|'ldap').
- moodle: drop SERVICE_OIDC / SERVICE_LDAP / SERVICE_LAM in favor of
the canonical OIDC_SERVICE_ENABLED / LDAP_SERVICE_ENABLED plus a new
LAM_SERVICE_ENABLED rendered from 'web-app-lam' group membership.
- decidim: drop OIDC_ENABLED env line; SSO-button-visible scenario
uses skipUnlessServiceEnabled('oidc').
Dashboard asset test split:
The previously bundled "dashboard loads core css, javascript,
simpleicons, and logo assets" test triple-gated (css/cdn/logout)
skipped the entire surface when any single shared service was off.
Split into six focused tests so disabling one service skips only its
specific assertion:
- shared CSS injection (gate: css)
- CDN-served role-local stylesheet (gate: cdn)
- logout.js injection (gate: logout)
- simpleicon-backed cards (gate: simpleicons; nocheck dropped)
- role-core JS modules + logos (ungated, role-core)
- iframe-sync postMessage handler (ungated, role-core)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Root causes diagnosed inside the running container per
docs/agents/action/iteration/playwright.md, fixed in repo, validated
via inner-loop rerun-spec.sh, and finally re-deployed via FULL matrix
variant=0 (all dashboard tests green) and reuse-kept variant=1
(persona/asset/login-swap tests collapse cleanly via env + test.skip
gates, 2 passed + 5 skipped, failed=0).
- templates/nginx/sso.html.conf.j2: serve the standard Keycloak
silent-check-sso body — `parent.postMessage(location.href,
location.origin)`. Without that script, the hidden iframe never
reports its URL back to the parent and `keycloak.init()` hangs
forever; the role's OIDC adapter then never wires its click
interceptor and the persona helper races into the static `href`
without PKCE state.
- templates/javascript/oidc.js.j2: set `window.__oidcLoginReady = true`
after `wireLoginLogoutClicks()` so persona helpers can detect that
the JS-wrapped click handler is in place BEFORE clicking. The flag
name SPOT is `OIDC_LOGIN_READY_FLAG` in
`roles/test-e2e-playwright/files/personas/utils/keycloak.js`.
- personas/utils/keycloak.js: new `clickOidcLoginLink(page, loginLink)`
helper — single source of truth for the OIDC-link click ritual
(waitFor visible → waitFor `__oidcLoginReady` → click → waitForURL
`/openid-connect/auth/`). Replaces the inline duplicated block in
biber.js + admin.js, and removes the static-href direct-nav fallback
that broke PKCE-enforced clients silently.
- personas/biber.js + admin.js: delegate to `clickOidcLoginLink` and
drop the `^...$` regex anchors on the Login link locator. Bootstrap
`data-bs-toggle="tooltip"` moves the link's `title` into
`data-bs-original-title` and the accessibility tree pulls THAT into
the link's accessible name (e.g. "Login to Infinito.Nexus on
infinito.example") — the anchored regex matched zero links. The
`\s*` between `log` and `in` keeps the substring pattern from
matching `Logout`/`logoff`.
- templates/playwright.env.j2: render
`PERSONA_{BIBER,ADMINISTRATOR}_BLOCKED` from
`services.oidc.enabled`. Variant=1 disables every shared service —
with OIDC off there is no Login link to click, so the persona
scenarios MUST collapse to a clean skip instead of failing loudly.
- files/playwright.spec.js: add `test.skip(\!cdnEnabled || \!logoutEnabled, ...)`
on the asset-loading test and `test.skip(\!oidcEnabled, ...)` on the
Login → Account swap test. The bespoke tile-reachability loop
already skips cleanly when `DASHBOARD_TARGET_ROLES_JSON` is empty,
so no further gate needed there.
- TODO.md: removed. The autonomy escape clause is satisfied.
- docs/requirements/019-playwright-meta-services-parity.md: strike
through `web-app-dashboard` row (v0+v1 closed) and document the
silent-check-sso / `__oidcLoginReady` / Bootstrap tooltip
contracts inline in the row's notes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Filter no longer takes a `database_volume=` parameter. The volume name is
derived from the central provider's `services.<dbtype>.name` (shared) or
`<entity>_database` (dedicated) — mirroring `plugins/lookup/database.py`'s
`volume_prefix + host` construction. Simultaneous postgres + mariadb on
the same role now raises an AnsibleFilterError with a clear message;
support is deferred per the docstring TODO.
The integration test (renamed from `test_shared_service_volumes.py` to
`test_compose_volumes_call_required.py`) now requires every
`roles/*/templates/compose.yml.j2` to call `compose_volumes`, so the
top-level `volumes:` block is always rendered. Without it, flipping a
shared service to dedicated mode adds a named-volume reference that
Compose rejects with `service "<name>" refers to undefined volume`.
Adds the call to 14 templates that were missing it (dashboard, hugo,
sphinx, bridgy-fed, fediwall, littlejs, mig, mini-qr, navigator,
roulette-wheel, simpleicons, collabora, logout, openresty) and strips
the `database_volume=...` argument from every other role plus Taiga's
now-unused `TAIGA_DATABASE_VOLUME` var.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the verbose
`(lookup('applications', application_id).server.domains.canonical | first)`
chain with the dedicated `lookup('domain', application_id)` plugin
across every playwright.env.j2 that renders `CANONICAL_DOMAIN`. The
helper already exists at plugins/lookup/domain.py and resolves the
canonical primary domain via the cached domain-merge pipeline, so the
inline canonical-list traversal is unnecessary noise that obscures
the actual contract.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Render BIBER_USERNAME/BIBER_PASSWORD on the role-local env templates
that drive the shared biber persona helper and, where the
administrator branch shares the same template, ADMIN_USERNAME/
ADMIN_PASSWORD too. Without these the shared persona helper read an
empty credential pair, skipped the Keycloak login step, parked on the
auth page, and the auth-marker assertion failed loudly with no clue
that the cause was a missing env declaration.
For web-app-pixelfed the previous indirection through
PIXELFED_PLAYWRIGHT_FIRST_USER_KEY / _SECOND_USER_KEY rendered the
same biber/administrator credentials under the role-local
LOGIN_USERNAME_{FIRST,SECOND} aliases. Drop the indirection and
consume BIBER_*/ADMIN_* directly in the role's spec so the env
contract matches the rest of the persona-driven roles.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Strip every `req <NNN>`, `req-<NNN>`, `Requirement <NNN>`,
`requirement-<NNN>`, and `docs/requirements/<NNN>-...md` mention from
non-Markdown source files. Comments anchored to requirement numbers
rot when requirements are renumbered or rewritten and conflate
present-state behaviour (which the comment should describe) with the
audit trail of how the code got there (which belongs in commit
messages, PR descriptions, and the requirement file itself).
* Document the rule in `docs/contributing/documentation.md` under
"Comments". Code MUST describe the *why* in present-state language;
contract enforcement lives in tests, not folklore.
* Clean 254 source files across `roles/`, `tests/`, `plugins/`,
`cli/`, `scripts/`, `utils/`, `group_vars/`, and the repo-root
`eslint.config.js`. Pure comment / docstring / log-message edits;
no behavioural change.
* Add `tests/lint/repository/test_no_req_references_in_code.py` to
enforce the rule going forward across `.js`, `.py`, `.yml`,
`.yaml`, `.j2`, and `.sh` files. The legacy
`test_no_stale_requirement_refs.py` is exempt as the only file
whose purpose is to detect these very tokens.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cross-service surface assertions for dashboard, prometheus and matomo
no longer run in every consumer role's persona. They are owned by the
provider's own spec, parameterised once over a `<NAME>_TARGET_ROLES_JSON`
manifest rendered into the provider's `templates/playwright.env.j2` via
the new `lookup('roles_with_service', '<svc>')` Ansible lookup.
Changes
-------
- New `plugins/lookup/roles_with_service.py` enumerates roles that
declare `services.<svc>.{enabled, shared}` truthy in the merged
applications config, returning `[{id, canonical_domain}, …]`.
- `roles/web-app-{dashboard,prometheus,matomo}/files/playwright.spec.js`
parse `<NAME>_TARGET_ROLES_JSON` and parameterise the per-consumer
assertion (dashboard tile reachability, prometheus `up=1`, matomo
SitesManager registration). The matomo spec gains an explicit
biber-denial test (request body validated against admin DOM markers
per the body-marker rule); the prometheus admin-reach test drops
the dashboard-iframe path in favour of direct goto.
- Persona helpers `roles/test-e2e-playwright/files/personas/{admin,
biber,guest}.js` enter at `${APP_BASE_URL}/` directly. The
`clickRoleTileFromDashboard` step and the `assertPrometheus*` /
`assertMatomo*` cross-service probes are gone; `utils/{dashboard,
prometheus,matomo}.js` are deleted. `utils/csp.js` switches to
literal `process.env.<KEY>` reads so the env-keys-used parity guard
recognises matomo's CSP injector consumption.
- 42 consumer-role `templates/playwright.env.j2` templates drop
`DASHBOARD_BASE_URL`, `DASHBOARD_SERVICE_ENABLED`, `PROMETHEUS_BASE_URL`,
`PROMETHEUS_SERVICE_ENABLED`. `MATOMO_BASE_URL` and
`MATOMO_SERVICE_ENABLED` stay because the per-role CSP injection
helper still gates on matomo's tracker host.
- `tests/integration/roles/test_playwright_env_services_match.py`
globally exempts `dashboard` and `prometheus` from the
services.yml-to-env parity rule.
- `tests/unit/plugins/lookup/test_roles_with_service.py` adds 14
unit tests for the lookup, including an explicit-contract test
that pins the over-inclusive behaviour on raw Jinja strings (the
lookup expects a templar-rendered payload).
- Docs: `docs/requirements/019-playwright-meta-services-parity.md`
rewords Rules 1, 3, 12 and adds Rule 13 describing the SPOT shift.
`docs/contributing/artefact/files/role/playwright.specs.js.md`
reflects the new persona shape and per-service catalogue, and is
cleaned per `docs/contributing/documentation.md` (no SPOT self-
labels, no req-references, em dashes replaced with complete
sentences, multi-clause lines split onto separate lines, broken
anchor `#two-personas-fixed-flow-` corrected).
Verification
------------
- 418 passed / 1 skipped / 0 failed across `tests/lint/`,
`tests/integration/roles/`, `tests/unit/plugins/lookup/`.
- `make lint-javascript`: 0 errors.
- `markdownlint-cli2`: 0 errors on all touched docs.
- `make autoformat`: clean (FURB171 single-item-set inequality fixed).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduces roles/test-e2e-playwright/files/personas/ — runGuestFlow,
runBiberFlow, runAdminFlow plus 10 utils (csp, dashboard, env, keycloak,
landing, logout, matomo, prometheus, interaction, index) — and wires
both the Ansible runner (run_one.yml) and the local rerun helper
(scripts/tests/e2e/rerun-spec.sh) to stage the tree alongside
service-gating.js so specs can `require("./personas")`.
First batch of role specs migrated to consume the shared library
(dashboard, keycloak, mailu, matomo, prometheus, web-svc-cdn,
web-svc-simpleicons) — each gets the spec.js persona scenarios plus
the matching meta/services.yml + playwright.env.j2 entries; matomo is
marked role-closed in req 019. dashboard/keycloak/mailu TODO.md
updated; docs/requirements/019 reflects matomo closure.
Reason: req 019 Rule 3 mandates that every role's guest/biber/
administrator flow runs the same canonical steps; centralising them
in one shared library keeps per-role drift from re-introducing the
stub-shaped bodies the new lint forbids.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hard-coded self-path references rot the moment the file moves: a header
comment ``# utils/foo/bar.py`` at the top of a module, a docstring
mentioning ``utils/foo/bar.py``, or a docs entry linking
``utils/foo/bar.py`` from inside ``utils/foo/bar.py`` itself all silently
desync from reality after the next ``git mv``.
The new lint walks every project file, computes its POSIX-style relative
path from PROJECT_ROOT, and reports any line whose text contains that
path as a substring. Files at the repo root (no ``/`` in their relative
path) and setuptools-generated ``*.egg-info/`` artefacts are exempt.
Per-line opt-out via ``# nocheck: self-path-reference`` on the offending
line, registered in the suppression catalog.
Existing offenders cleaned up:
- 72 redundant ``# <self-path>`` header lines removed across roles,
cli, plugins, scripts, tests, and utils.
- 4 prometheus templates that told consumers to "update
<self-template-path>" rewritten to drop the redundant self-reference.
- 8 docstring/comment self-references in cli, plugins, tests, utils,
and the dashboard oidc.js.j2 header rewritten to describe the layout
without naming the file's own path.
- 17 legitimate usage examples (shell scripts, test data lists, one
template self-include doc-block) marked with ``nocheck:
self-path-reference``.
- ``container.sh`` heredoc usage strings switched to ``${0}`` so the
marker stays out of the user-visible help output.
- ``healthcheck/http.yml.j2`` doc-block rewritten to prose because
``{# ... #}`` Jinja comments cannot host an inner ``{# nocheck #}``
(the inner ``#}`` closes the outer comment).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Touches wordpress, nextcloud, yourls, and dashboard. Drops redundant
| default(false) where the value is contracted-set.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
