nginx resolves its upstreams once, at start, so the frontend starts through
a guard that waits until `backend` and `exporter` resolve. A container
resolves with a search domain, and the zone of a deployment answers every
name with the address of the ingress. A service that Docker's DNS did not
know yet therefore resolved anyway, as `backend.<domain>`. The guard let
nginx start at once and nginx pinned the ingress as its backend. Every API
call then failed with 502 and the SPA showed "Bad Gateway" instead of the
login form, until the container was restarted.
The guard now asks for the absolute names, `backend.` and `exporter.`. The
search domain is not applied to those, so the lookup fails until the
services themselves resolve.
Closes#806
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The mount pipeline was split across two layers: compose_volumes and
container_volumes lived as filter modules that ansible's filter loader
rejected on every start ("has no attribute 'FilterModule'", two warnings
per run), while the same-named lookups wrapped them thinly. The filter
bodies now live inside their lookups (the tor_socks precedent), the
filter files are gone, and the render tests moved next to the lookup
tests.
As lookups they see the templating context, so the application_id term
every call site passed back to its own role is gone too: both lookups
read application_id from context and fail loudly when it is missing.
107 templates and nfs_prep.yml drop the term; the calls collapse to
lookup('compose_volumes') and lookup('container_volumes', service_name).
The container_volumes call is now canonical - four leading spaces,
service_name in the service position, | indent(4) - enforced by a new
lint (test_container_volumes_call_shape). Normalizing the 85 literal
call sites surfaced that erpnext keyed its meta/volumes.yml mounts by
container name instead of compose service key, the only role to do so;
its meta now uses the service keys. wordpress's column-0 call, which
rendered its first line at top level, is indented like every other.
Template logic that only restated meta/volumes.yml is deleted: mailu fed
compose_volumes an extra_volumes dict whose eight names were looked up
from the very meta file compose_volumes reads itself; zammad's init and
railsserver extra mounts were the volumes.yml entries minus the two
services, which the meta now lists (the when-gated OIDC binds included).
Conditionality that meta cannot express stays in code, but in role-local
lookup plugins instead of template loops: matrix_bridge_mounts renders
the per-bridge registration mounts, nextcloud_extra_volumes the
feature-gated whiteboard/recording volumes (type: volume entries are
declared unconditionally by compose_volumes, so gating must happen
before it).
Parity is proven, not assumed: every mechanically rewritten template is
byte-identical to HEAD after the transformation regexes (95/95), and the
semantic roles render identically through the real functions against
HEAD's meta - zammad across six sso/tls polarities, all seven erpnext
services, mailu's top level. make test-lint passes complete.
Also: suitecrm's compose template pins container_extra_hosts with
host_alias (the SOCKS proxy URL host.docker.internal never resolved in
compose+tor), and the five disabled mautrix bridge addons use the
empty-default config lookup so templating the applications dict stops
emitting 125 errors per batch in every CI job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/volumes.yml says where a volume lands inside its container. Wherever a
second file spelled the same path out, the two drifted independently: the mount
moves, the consumer keeps pointing at the old place, and nothing fails until the
process looks for content that is not there. e031b7f61 fixed exactly this for
web-app-moodle; this generalises it and makes the rule hold.
The lookup gains what a caller needs. `targets` maps each mounting service to
its container path, and `target` is the one path an entry mounts at -- present
only when every mount agrees. An entry that lands at different paths per service
omits `target` rather than picking one, and the caller names the service:
"{{ lookup('volume', application_id, 'prometheus_config').target }}"
"{{ lookup('volume', 'web-app-prometheus', 'host_sys').targets['node-exporter'] }}"
66 literals across 48 role files now derive. Replacement only happened where the
path stood alone -- /srv/app inside /srv/app/data is a different path -- and
every rewritten file was re-parsed, Jinja and YAML alike. Three sites the tool
refused rather than guess were done by hand: zammad's path sat in a Jinja list
literal, seaweedfs' inside a quoted string feeding a filter, and openresty's
/var/run/docker.sock is not a mount target at all but the socket on the host,
which only happens to spell the same string. That one carries the single
nocheck.
tests/lint/repository/spot/test_volumes_path_single_spot.py keeps it that way.
roles/*/files/ is out of scope by construction: it is copied verbatim into an
image and executed there, so a Dockerfile or entrypoint has no Ansible to
evaluate a lookup with.
One defect this found, and it was in the tooling rather than the roles: deciding
ambiguity AFTER filtering short paths made web-app-prometheus' host_sys look
single-valued -- it mounts /sys for cadvisor and /host/sys for node-exporter,
and /sys is below the length filter. The template would have rendered an
undefined `.target`. Both the converter and the rule now decide ambiguity over
every target and filter afterwards, and that call reads targets['node-exporter'].
Verified by resolving rather than by reading: all 70 accesses in the diff were
run against the real registry and every one yields a non-empty path equal to the
literal it replaced. make test-lint: 328 passed.
Not verified: no Ansible render, so a call site whose surrounding context cannot
build the applications registry would still fail at deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
extra_hosts is the one compose key a service may not carry twice. The onion SSO
back-channel pin was emitted by a Jinja partial that six roles included, while
six other services wrote the key by hand; nothing stopped a service from
receiving both. YAML then either rejects the document or keeps one list and
drops the other, silently.
The partial becomes plugins/lookup/container_extra_hosts.py, and every pin now
goes through it. Callers add their own entries with extra_hosts=[...], the way
container_volumes already takes extra_volumes=, and the lookup merges them after
the automatic pin and deduplicates in first-seen order. With nothing to emit it
returns an empty string.
The swarm address reasoning from 7b1839681 is preserved and now written down in
the plugin rather than only in that commit: the pin must NOT be host-gateway in
swarm, because that alias resolves per node while openresty publishes :80 with
mode: host under node.role == manager. It carries the stack host's address,
which reaches every task because the compose file is rendered once with the
manager as inventory host. The mode is read as compose_mode_force or
DEPLOYMENT_MODE, not the bare cluster mode, so a role with an override does not
lose its pin and its overlay at once.
Two roles kept a pin that now collides with the automatic one, and both are
resolved the same way -- the hand-written entry stays for clearnet and yields
the .onion case to the lookup, which picks the address per mode. In
web-app-mailu the three conditional entries move into a role-local
mailu_extra_hosts lookup exposed as MAILU_EXTRA_HOSTS, because an eight-line
conditional expression in the template argument is worse than the block it
replaced. It returns its entries as separate results, so vars/main.yml uses
query(); plain lookup() would join them into one comma-separated string.
tests/lint/ansible/jinja/test_compose_extra_hosts_use_lookup.py fails on any
literal extra_hosts: key in a role template, so the invariant survives the next
author who needs a pin. It follows test_compose_ports_use_container_ports_lookup:
the raw construct is forbidden, the lookup is not mandated per service -- a
service with no pins is correct without it, and the swarm branch raises when
facts are absent, which would turn a no-op into a failure surface for containers
that never speak OIDC.
Equivalence checked by rendering rather than by reading: under the trim_blocks +
lstrip_blocks the stack-host action forces for compose.yml.j2, both former call
site shapes (indented, and matrix's column 0) emit byte-identical text to the
new form when the block applies. The no-op case differs -- a whitespace-only
line where the include emitted none -- and yaml.safe_load returns the same
document for both.
31 unit tests; make test-lint 325 passed. Does not touch web-app-espocrm, whose
missing pin is a separate claim and gets its own commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
75 commits: the socialhome role reimplemented against the project it actually
deploys, native-login specs for funkwhale and shopware, a window.open IdP
handoff and a settings-link fallback in the shared persona helpers, a swarm
converge budget that gives up on a terminal task, and a long tail of role
fixes.
Eleven conflicts. The shared persona helpers (keycloak.js, logout.js) take the
incoming structure — they gained real behaviour this branch never had — with
this branch's resolveTimeout convention re-applied over it and
confirmKeycloakLogoutIfPrompted re-attached at every return point; without it
the Keycloak SSO session outlives the logout. playwright.config.js keeps this
branch's version: both sides set actionTimeout, but ours routes through an env
var and 15s is too short behind an onion. socialhome's spec takes the incoming
content — 2dbd6eb10 found the role described the wrong upstream project
entirely — and keeps gotoOnion and expectHstsWhenTls, since demanding HSTS over
a plaintext onion is a false failure. Both READMEs were regenerated rather than
merged by hand.
Three collisions arrived with no conflict marker at all, because each side
changed a different place in a different file:
- group_vars/all/18_swarm.yml declared swarm_stack_converge_retries twice.
Same value, so nothing breaks today; editing the documented one would have
silently changed nothing.
- svc-net-tor and web-app-socialhome both claimed 192.168.74.0/24. Socialhome,
the newer role, moves to 192.168.75.0/24.
- web-app-socialhome/meta/schema.yml arrived as a new file, so ef5ea7cb5's
rename never saw it. Its credential would not have resolved and the role
would have deployed with an unusable administrator account.
The incoming specs also needed this branch's conventions: raw timeouts wrapped,
four page.goto calls routed through gotoOnion, a request.get given an
onion-aware timeout, and 21 roles marked playwright-service-flag over the tor
block this branch added to them.
make quality green on the merge result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every conflicted hunk resolves onto the swarm-nfs side, because on all six
files it is the strict superset: it carries the display-name codec
(utils/roles/display.py) the CI role selections now encode through, and the
sequencing glyph move from 910748ec4. svc-net-tor holds no unique content on
any of them -- runs.py even had non-conflicted hunks already calling `codec`,
so keeping its side would have left a NameError.
The lint fallout is settled in the same commit, as 4afc9b770 did for the
previous merge: the shell-or-true lint lives on svc-net-tor (c505ce746) and
swarm-nfs brings two unmarked `|| true` sites with it. Both get a nocheck
reason rather than a rewrite, because neither swallows anything -- the
slaptest call is the measured exception of c4bbbba93 and has_cn_config decides
right below it, and the dockerd restart is followed by an is-active poll that
fatals. In both the exit code was never the authority.
make quality green end to end.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Blocking a persona removes the only end-to-end proof that a role's auth chain
works. docs/contributing/artefact/files/role/playwright.specs.js.md:146 has
always required each flag to carry a one-line nocheck stating the role contract
that justifies it, plus a matching paragraph in the role's README or TODO.
Nothing enforced it, so 89 flags across 46 roles carried neither.
The lint mirrors test_has_spec's shape and uses is_suppressed_at with
mode="line-above", so the marker must sit directly above its flag -- a blank
line between them breaks the association, which is what the persona helper's
reader will look at. A literal =false blocks nothing and is exempt; the
sso-conditional ternary is not, because it blocks in every variant that pins
services.sso.enabled false.
Writing 89 reasons turned out to be the point. Five roles audited in one pass
blocked for five entirely different mechanisms, all of which read as the same
shrug behind a bare =true: pgAdmin's login id is the account e-mail rather than
the Keycloak username; phpMyAdmin authenticates as the MariaDB root account;
GitLab's seeded password belongs to the built-in root, not the administrator;
Mastodon's tootctl accounts create never sets a password; phpLDAPadmin binds
with an LDAP DN and a different secret. Further out: MediaWiki removes the form
that would accept the Keycloak secret ($wgPluggableAuth_EnableLocalLogin=false)
and bounces anonymous requests back into the IdP, so the unauthenticated
landing the contract asserts can never be observed; Mattermost Team Edition has
no native OIDC, so Keycloak rides the GitLab OAuth slot behind a relabelled
link; MinIO's console returns redirectRules: null and renders no SSO button at
all; XWiki's administrator is an XWikiUsers object with no password property.
Two flags turned out to be unjustifiable and are gone rather than annotated.
web-app-mailu gated PERSONA_GUEST_BLOCKED on services.dashboard.enabled, but
runGuestFlow does one anonymous goto and asserts no signed-in marker -- it
never touches the dashboard. A leftover from a helper version that entered
through the dashboard account menu. And web-app-mig's rationale claimed the
role is an in-app module of web-app-dashboard; it has its own compose stack,
healthcheck and canonical domain, and dashboard is merely a tile consumer.
Nine of the reasons say plainly that the block is not a role property but a gap
in the shared persona helpers -- decidim's sign-out is a data-method link the
generic click cannot activate, fider's SSO entry is a two-step modal, taiga,
wordpress, xwiki, odoo and penpot bind OIDC on a path the personas never visit,
jenkins blocks unconditionally although its OIDC variant looks runnable, and
opentalk's own spec proves both personas work. That list did not exist before;
it is the actual yield here, not the 89 comment lines.
45 pre-existing comments came into the comment lint's scope because this sweep
touched their files -- it only inspects files that differ from HEAD. 28 of them
were the banner "Service gate flags rendered for closure.", every one sitting
above a line that is not a service gate flag at all, orphaned by the
alphabetical re-sort these templates once got.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The backend has never carried a healthcheck, in neither the include-template
era nor after the probes moved into services.yml. That is why it stayed
invisible when its containerd shim was orphaned: docker had dropped the task
record while the process lived on, every exec into it returned "task not
found", and swarm kept reporting the replica as running because nothing ever
declared it unhealthy. Frontend, postgres, redis and openresty recovered in
that same run precisely because they are probed.
The tcp flavor asks for nothing the image does not already provide: it opens
a bash socket against the port declared under ports.internal and greps the
response for a status line, so it needs neither curl nor wget. Sixteen roles
across node, ruby, python and php images already probe this way. The five
minute start period covers the JVM boot.
The one assumption left is that the server binds a loopback-reachable
address. Penpot documents no host setting and its upstream compose does not
probe the backend either, so this is inferred from the frontend reaching
backend:6060 from another container. A probe that cannot pass would turn the
next CI run red rather than damage anything.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven include templates under sys-svc-container decided how a container was
probed, and they read their inputs from template-local `{% set %}` scope.
That scope leaks: a value set for one service silently reached every service
below it, which is how taiga's front and gateway ended up probing a port
declared for a different service, and how bluesky's frontend probed a PDS
endpoint.
Probes now live in utils/docker/healthcheck.py, one class per flavor behind a
PROBES registry, and each service declares its own:
healthcheck:
flavor: curl
path: health/ready
start_period: 20m
`port_key` picks another entry under ports.internal for the services that
expose their probe elsewhere (keycloak on management, nextcloud talk on
signaling). Without a flavor the entry must carry an explicit test argv, so a
missing declaration fails loudly instead of rendering a probe against nothing.
Rendering HEAD's templates against the new output confirms 49 of 56 call
sites are byte-identical; the remaining seven are covered by an Ansible run.
Three probes change on purpose: erpnext's backend gains a check it never had,
bluesky's frontend drops the leaked PDS path, and msmtp_curl moves from a
hardcoded localhost:80 to the service's own port on 127.0.0.1 like every
other flavor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
swarm's docker stack deploy drops the compose depends_on, and the penpot frontend nginx resolves its proxy_pass upstreams (backend, exporter) at config load, so when a peer service's VIP is not yet registered the frontend exits with [emerg] host not found in upstream and crash-loops under restart_policy any; the reschedules pile all three replicas onto the manager and the seed step finds no worker task. Override the frontend command to poll getent for both peers before exec-ing nginx, so all replicas start on the first pass and swarm spreads them one per node. The image entrypoint ends in exec "$@" and ships getent, so this is behaviour-preserving in both modes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New roles/sys-svc-container/templates/env.j2 sets TZ={{ HOST_TIMEZONE }} and LANG=C.UTF-8. Every role env.j2 includes it at the top; it renders via stack_host_template, the same engine that resolves the includes in compose.yml.j2, so it works in both compose and swarm. TZ/LANG are removed from the individual role env.j2 so each value stays single-source.
New lint tests/lint/ansible/test_env_base_include.py verifies every role env.j2 includes the base and never re-sets a base-owned key (keys derived from the base file).
mattermost and gitea meta/volumes.yml drop the /etc/timezone host-path bind: swarm hard-rejects a task whose bind source is missing on the node (Debian-trixie nodes have no /etc/timezone). TZ now comes from the base env. /etc/localtime bind also dropped; the container timezone follows TZ.
Non-directive comments were removed from the touched env.j2 (banners, restatements, reference URLs, narration) so the changed files pass the comment-lint; only tool directives remain.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
penpot's HikariCP pool fills eagerly and upstream defaults to 60
connections; with 3 swarm replicas that exceeds the non-superuser slots
(max_connections 170 minus reserved) and the post-deploy backup's
pg_dump is rejected with 'remaining connection slots are reserved for
roles with the SUPERUSER attribute'. Cap the pool per replica from the
topology group size. Deliberately not using
POSTGRES_ALLOWED_AVG_CONNECTIONS: its consumer count resolves to 1 and
it is replica-blind; repairing that SPOT is tracked in
https://github.com/infinito-nexus/core/issues/416.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lookup('canonical_url', app_id[, key][, consumer=app]) resolves the web base
URL for an app or one of its named canonicals (element/synapse/filer),
following the onion-injected merged-domain map: .onion targets resolve to
http (Tor provides transport security), everything else follows TLS_ENABLED.
The consumer= kwarg returns '' unless the consuming app has the target's
service binding enabled, replacing the recurring
"(lookup(...) if lookup('config', app, 'services.X.enabled') else '')"
template branching.
- utils/cache/domains.py: inject the node onion into dict-form
(named-canonical) domain declarations, not just list/str forms
- keycloak redirect_uris: onion redirect URIs are http
- 13 playwright.env.j2: manual scheme branching -> canonical_url(consumer=)
- lint: forbid manual '://' concatenation and to_onion_url in role templates
- unit tests: 10 cases (primary/named x clearnet/onion, consumer gating)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The sys-svc-container healthcheck templates (curl/wget/nc/tcp/http) now
derive container_port from services.<service_name>.ports.internal.http,
so a role no longer needs to set container_port only to feed the shared
healthcheck.
Add the container-port-set-unused lint that fails a role setting
container_port without using it elsewhere, then resolve every flagged
role: drop the redundant assignment where the derivation reproduces the
value, or mark it with nocheck where the port is non-http, a constant, or
keyed on a service the derivation cannot reach.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every compose.yml.j2 now declares services as '{{ service_name }}:' with
its '{% set %}' knobs on two-space lines above the key, includes at four
spaces and if/for tags aligned to the content they wrap. The layout
renders correctly because stack_host_template enables lstrip_blocks for
compose.yml.j2; indent(4, True) in the container base and '{%+' guards
in whitespace-sensitive scalars keep the shared fragments identical
under both render modes. A new lint test enforces the skeleton.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
One entrypoint wrapper imports the project CA into the JVM truststore for penpot and jenkins: env-driven (CA_TRUST_CERT from sys-svc-compose-ca, set in both modes), fail-open without a cert (production), fail-loud with one; writable cacerts are patched in place, read-only layouts fall back to /tmp + JAVA_TOOL_OPTIONS. Replaces penpot's inline compose command and jenkins' role-local entrypoint, which silently skipped the import when cacerts were not writable. ca_cert_container becomes a plain constant since its only Jinja consumer moved into the wrapper.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The version pin sat on the app-level penpot entry, which container_image never
reads, so the frontend/backend/exporter deployed :latest. Since Penpot 2.15 the
backend SSRF-blocks outbound HTTP to hosts resolving to private IPs, so the OIDC
token exchange to the privately-resolving Keycloak failed with unable-to-auth
and login never completed. Pin the three image-bearing services to 2.16.2 (the
release CI actually exercised) and set PENPOT_SSRF_ALLOWED_HOSTS to the Keycloak
host (https? strips either scheme). env.j2 stays shared by compose and swarm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extract the csp, domains, and networks blocks out of roles/*/meta/server.yml
into dedicated meta files, each mapping to its own top-level applications key
via the recursive meta-walk:
csp: -> meta/csp.yml -> applications.<app>.csp
domains: -> meta/domains.yml -> applications.<app>.domains
networks: -> meta/networks.yml -> applications.<app>.networks
Residual nginx/proxy keys (status_codes, client_max_body_size, locations,
ip6_enabled, config_upstream_url) stay in server.yml; an emptied server.yml is
deleted. 98 server.yml -> 249 new files, 74 deleted.
- Consumers repointed applications.<app>.server.{csp,domains,networks} ->
applications.<app>.{csp,domains,networks} across templates, vars, tasks,
lookups and filters.
- Infra readers migrated: canonical_domains_map, application_domain_index,
build_ldap_role_entries, service-registry overlay discovery,
roles_with_service, and the file-based readers utils/domains/list.py +
utils/meta/scan.py (read the now-deleted server.yml directly).
- Detection + mapping: _META_TOPICS gains csp/domains/networks;
ROLE_FILE_META_{CSP,DOMAINS,NETWORKS} constants + ROLE_FILES entries.
- New self-expiring lint guard (test_server_meta_no_split_fields) fails if a
server.yml still carries a split field; auto-fails from 2026-12-28 as a
delete reminder.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Take the host IP as an optional third element of each term
([service, protocol, host_ip]) instead of an ip= kwarg, and build the whole
ports: block from declared ports.local/public (host) + ports.internal
(container). Convert every compose template across 84 roles, folding the
remaining raw var/numeric/specific-ip mappings (including mailu's per-interface
mail ports bound to MAILU_IP4_PUBLIC). Multi-term calls render one term per line.
Add the compose-ports-must-use-container-ports lint: it forbids any raw
"<host>:<published>:<container>" publish item under a ports: block and requires
every term to declare ports.internal. Remove
tests/lint/roles/test_swarm_internal_ports.py, now subsumed.
Fix a container_depends_on rendering regression. The lookup self-indented
(default indent=4), so the added 4-space placement + | indent(4) triple-indented
the block. Flip the lookup default to column 0 (consistent with
container_networks), make every call use the 4-space + | indent(4) form, and add
a lint enforcing it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
About 38 web-app/web-svc roles now consume the central svc-db-* and svc-dns engines via lookup('engine', ...) instead of embedded sidecars: meta/services flags, variant matrices, and env/conf hosts point at the shared services. Lets these apps run unpinned and NFS-shareable in swarm.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reconcile main's object-store feature and pre-swarm deploy tasks with the branch's compose/networks/volumes SPOT and swarm lints.
objstore (seaweedfs/minio): standalone compose + embedded service templates migrated to the container_image/container_volumes/container_networks/compose_volumes/compose_networks/depends_on lookups; meta/volumes.yml on the canonical dict schema with s3.json and nginx.conf as type:config; embedded services made swarm-compatible via compose_only/compose_restart.
deploy tasks (erpnext, opentalk, seaweedfs, shopware, snipe-it, wordpress, yourls): container exec routed through the container_address resolver and gated on IS_STACK_HOST, shell module plus stack_host_template, no chdir or compose-verb forms.
Also removed orphaned vars, restored MAILU_INSTANCE_DIR, freed the seaweedfs subnet (192.168.206.0/24), stripped literal quotes from the nextcloud/bookwyrm env templates, and routed main's objstore compose_volumes tests through dir_var_lib.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Convert the two registry-driven networks templates into pure-Python
rendering functions plus thin Ansible lookup plugins:
roles/sys-svc-compose/templates/networks.yml.j2 -> {{ lookup('compose_networks') }}
roles/sys-svc-container/templates/networks.yml.j2 -> {{ lookup('container_networks') | indent(4) }}
The rendering logic lives in utils/networks/render.py as pure-Python
functions that take a registry + closure-style config/database lookups,
making the logic directly unit-testable without a Jinja stub harness.
The lookup plugins are thin wrappers that resolve application_id,
DEPLOYMENT_MODE, and swarm.network.encryption from the templar, build
the service_registry, and forward to the render functions.
Migrate 233 include lines across 85 production templates. Convert
web-app-matrix's hand-coded MDAD networks block (default + conditional
postgres + conditional openldap) to the generic lookup; this adds an
openresty attachment when services.sso.enabled (harmless extra
interface, pinned in test_matrix_mdad_consumer_attaches_postgres_ldap_sso).
Mark local-DB / redis sidecars (sys-svc-rdbms/templates/services/,
svc-db-redis/templates/service.yml.j2) with `# nocheck: networks-literal`
because they intentionally attach to default-only.
Extend the lint to scan ALL roles/*/templates/**/*.yml.j2 (not only
compose.yml.j2) and to forbid the legacy include form; the marker
above is the documented escape hatch.
Coverage:
- tests/unit/utils/networks/test_render.py (30 cases incl. encrypted=false literal pin)
- tests/unit/plugins/lookup/test_compose_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/unit/plugins/lookup/test_container_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/integration/infrastructure/compose/test_networks_render.py (10 snapshot scenarios)
- tests/integration/infrastructure/compose/test_network_includes.py (3 lint rules)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add plugins/lookup/container_image.py: returns the full `image: "<ref>"`
line so templates drop the redundant image: prefix and surrounding
quotes. Supports `custom=True` to auto-construct `<entity_name>_custom`
for locally-built images, plus explicit `image=`/`version=` overrides
for exotic cases (taiga back-auth/front-oidc variants, decidim infinito
suffix, etc.).
* 25 unit tests for the lookup cover compose vs swarm prefix, custom
flag, overrides, tag_only=True, missing version warnings.
* New strict lint (tests/lint/.../test_no_raw_image_in_compose_templates)
fails on any raw `image:` line in roles/*/templates/*.j2 unless
nocheck-suppressed. Legacy `image: "{{ lookup(...) }}"` form no
longer accepted.
* Migrate 159 image refs across ~80 roles to the standalone
`{{ lookup('container_image', application_id, '<service_key>') }}`
form. 26 roles use `custom=True` (mariadb, postgres, openldap,
jenkins, jira, gitlab, nextcloud, mediawiki, shopware, wordpress,
odoo, joomla, openproject, bookwyrm, espocrm, hugo, littlejs,
fediwall, roulette-wheel, openresty, confluence and more).
* Fix svc-db-redis application_id leak: service.yml.j2 is included
into consuming roles where application_id evaluates to the outer
role, so the image lookup needs the hardcoded 'svc-db-redis'.
* Integration tests (test_templates_require_image,
test_build_requires_image) recognise the standalone lookup as an
implicit `image:` at the same indent.
* Mediawiki swarm tasks: add explicit `:{{ MEDIAWIKI_VERSION }}` to
the registry-prefixed image string because the custom var is now
bare.
Only 3 legitimate exotic nocheck-suppressed image lines remain:
matrix-mdad runner (nested services.matrix.runner.image path),
matrix mautrix bridges (per-iteration dynamic) and svc-registry-docker
(chicken-and-egg self-reference).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add files/playwright/test-seaweedfs.js (+ SEAWEEDFS_* playwright.env vars, spec require wiring, and removal of the playwright-service-flag nocheck above the seaweedfs service) for baserow, bookwyrm, mattermost, openwebui, peertube, penpot, pixelfed. Each test reuses the role's real login flow and triggers an upload that writes to the app's S3 bucket (file-field, avatar, chat attachment, composer photo, video), verified via the shared runSeaweedfsStorageCheck (bucket grew through the Filer UI).
openwebui: also export dismissAllOpenModals from _shared.js (used by the new test) and match the uploaded file by basename (not the full filename) to avoid the extension-split false-negative.
Exempt gitea and opentalk via meta/services.yml nocheck markers (seaweedfs-playwright + playwright-service-flag): gitea has no automatable UI login in this stack (oauth2-proxy + OIDC account-link / native login both fail; personas blocked); opentalk only writes S3 for recordings/whiteboard exports, which need the recorder service + a live multi-party meeting.
Builds on 9b6c6415f (shared runSeaweedfsStorageCheck helper + consumer lint).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Switch penpot, nextcloud, gitea, mattermost, openwebui, funkwhale, baserow, openproject, pixelfed, mastodon, bookwyrm, listmonk, peertube and opentalk to the engine-agnostic objstore lookup: the minio service entries are pinned off, a seaweedfs service (gated on the provider being in group_names, public for the media apps) replaces them, and the matrix variants pin the new flag. OpenTalk keeps object storage mandatory through an embedded fallback and provisions its bucket via weed shell.
Add objstore lookup SPOT keys consumed by these roles: endpoint (host:port), public_domain, public_bucket (public S3 host plus bucket) and public_url (scheme plus public_bucket, scheme derived from the provider TLS state). This removes the repeated lookup('config', lookup('objstore', ...), 'server.domains.canonical.api') / host:port concatenations from the consumer templates. Listmonk settings render their JSON values via the to_json filter.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- tasks/main.yml: remove `| default('')` from changed_when on command
results (stdout always set when the conditional fires) per
test_no_redundant_default_on_module_io.
- env.j2: drop `| default('')` on LDAP.SERVER.SECURITY (defined
unconditionally in group_vars/all/12_ldap.yml) per
test_no_jinja_default_on_spot_path.
- spec: split the monolithic Playwright spec into per-surface companions
(_shared.js + test-login-{native,oidc,ldap}.js) required by playwright.spec.js;
OIDC and LDAP cover both administrator and biber; native registers a local
account (skips when OIDC is the login path / registration disabled).
- meta/main.yml: description now describes the application, not the role.
- meta/server.yml: canonical domain -> penpot.design.{{ DOMAIN_PRIMARY }}.
- meta/info.yml: add upstream intro video.
- meta/services.yml + vars: registration_enabled default in services.yml
(off when OIDC enabled, operator-overridable); PENPOT_FLAGS toggles
enable/disable-registration accordingly.
- templates/env.j2: bind PENPOT_SMTP_ENABLED to email.enabled and
PENPOT_LDAP_SSL/STARTTLS to LDAP.SERVER.SECURITY (constants, not literals).
- templates/compose.yml.j2: trim the backend comment to the exception and gate
the JVM CA-trust import behind self-signed TLS (no-op in production); readable
YAML block.
- templates/playwright.env.j2: dotenv_quote APP_BASE_URL; drop redundant blank lines.
Verified live: 5 passed, 6 skipped, 0 failed (OIDC/project/asset skip on this
sso-off deploy; native + LDAP + baseline + guest pass).
Deployed Penpot + Keycloak + OpenLDAP end to end and drove the per-role
Playwright suite green against the live stack (8 passed, 2 skipped):
TLS baseline, OIDC (administrator + biber), LDAP (administrator + biber),
project creation, and image asset upload. The two skips are the generic
biber/administrator persona scenarios, declared blocked via
PERSONA_*_BLOCKED (mirroring web-app-taiga): Penpot's in-app "OpenID"
login entry and SPA user-menu logout are not driveable by the generic
persona helper, so both users' real auth is exercised by the dedicated
OIDC + LDAP scenarios.
Fixes found during bring-up:
- OIDC JVM CA trust: the backend command imports the internal CA into a
writable cacerts copy and points the JVM at it via JAVA_TOOL_OPTIONS.
with-ca-trust.sh covers OS/NSS/env trust but not the JVM truststore, so
server-side token/userinfo calls to Keycloak failed with "PKIX path
building failed". CA path hardcoded (the CA-override double-escapes a $).
- disable-onboarding added to PENPOT_FLAGS so users land on the dashboard.
- Playwright env: ADMIN_EMAIL/BIBER_EMAIL for the LDAP form; alphabetised.
Requirement 235 is fully checked off and archived out of docs/requirements/.
Add a new `web-app-penpot` role deploying Penpot's frontend + backend +
exporter containers behind the standard reverse proxy, with shared
svc-db-postgres and a svc-db-redis sidecar. Wires native OIDC (Keycloak)
and LDAP (OpenLDAP) login via PENPOT_FLAGS, CSP/vhost via sys-stk-full,
backup-ready asset volume, marketplace tags, matrix variants
(all-on/all-off/LDAP-only), and a service-gated Playwright spec with
guest/biber/administrator personas plus separate OIDC and LDAP scenarios.
Implements docs/requirements/235-setup-penpot.md. Live-deploy DoD items
(fresh-host deploy, domain reachability, full in-container `make test`)
remain unverified in the sandbox and are left unchecked.
Remove the entire web-app-penpot role, all associated docs, and all other
changes accumulated on this branch. Working tree now matches origin/main
exactly. Commit history preserved for reference.
- Move playwright spec to roles/web-app-penpot/files/playwright/ (standard location)
- Fix APP_BASE_URL: use normalizeUrl() to strip dotenv_quote double-quotes
- Gate biber/admin tests with safeSkipUnlessEnabled('oidc') (penpot requires OIDC)
- Add meta/variants.yml with all-true, all-false, LDAP-only variants
- Fix templates: use container_depends_on lookup, correct LDAP group vars
- Align services.yml: javascript service static false with nocheck comment
- Update playwright.env.j2: OIDC_SERVICE_ENABLED env var for service gate
- All make test targets pass (lint, test-unit, test-integration, test-lint, test-external)
Backend/exporter/frontend services were listing environment variables
without values (e.g. - PENPOT_DATABASE_PASSWORD) which causes Docker
Compose to pass through from host environment. Since these aren't set
on the host, they become null and override the env_file values.
Fixed by removing environment sections - env_file already contains all
needed variables with correct values.
Changed from single-line array format to multiline YAML block style
for better readability and to avoid potential parsing issues with
complex shell commands containing comments and special characters.
The container-based SSL certificate import approach is correct since
keytool only exists inside the Java container, not on the host.
- Removed backend healthcheck that was blocking stack startup
- Backend depends_on redis (healthy), frontend waits for backend (started)
- Playwright test.beforeAll() already polls backend API for up to 16 minutes
- This matches pattern from commit 332ef7833 (other roles without healthchecks)
- Avoids Docker Compose blocking on backend health determination
- Simpler, more reliable: let application-level tests handle readiness
- Changed from wget/curl (not available in penpotapp/backend image) to bash TCP socket
- Uses: exec 3<>/dev/tcp/localhost/6060 with HTTP GET request
- Pure bash built-in, guaranteed to work since backend command uses /bin/bash
- Matches pattern used by other roles (e.g., sys-svc-container/templates/healthcheck/tcp.yml.j2)
- Add healthcheck to backend service (curl http://localhost:6060/)
* interval: 30s, timeout: 10s, retries: 10, start_period: 120s
* This ensures backend is truly ready before tests run
- Update depends_on to use health conditions instead of just service_started
* frontend waits for backend to be healthy
* backend and exporter wait for redis to be healthy
- Add unsafe-eval to script-src-elem and script-src-attr in CSP config
* Ensures CSP2 fallback merge includes unsafe-eval for Penpot's eval() usage
Fixes#204 - Resolves 502 Bad Gateway errors during Playwright tests by ensuring proper container startup ordering
