meta/volumes.yml says where a volume lands inside its container. Wherever a
second file spelled the same path out, the two drifted independently: the mount
moves, the consumer keeps pointing at the old place, and nothing fails until the
process looks for content that is not there. e031b7f61 fixed exactly this for
web-app-moodle; this generalises it and makes the rule hold.
The lookup gains what a caller needs. `targets` maps each mounting service to
its container path, and `target` is the one path an entry mounts at -- present
only when every mount agrees. An entry that lands at different paths per service
omits `target` rather than picking one, and the caller names the service:
"{{ lookup('volume', application_id, 'prometheus_config').target }}"
"{{ lookup('volume', 'web-app-prometheus', 'host_sys').targets['node-exporter'] }}"
66 literals across 48 role files now derive. Replacement only happened where the
path stood alone -- /srv/app inside /srv/app/data is a different path -- and
every rewritten file was re-parsed, Jinja and YAML alike. Three sites the tool
refused rather than guess were done by hand: zammad's path sat in a Jinja list
literal, seaweedfs' inside a quoted string feeding a filter, and openresty's
/var/run/docker.sock is not a mount target at all but the socket on the host,
which only happens to spell the same string. That one carries the single
nocheck.
tests/lint/repository/spot/test_volumes_path_single_spot.py keeps it that way.
roles/*/files/ is out of scope by construction: it is copied verbatim into an
image and executed there, so a Dockerfile or entrypoint has no Ansible to
evaluate a lookup with.
One defect this found, and it was in the tooling rather than the roles: deciding
ambiguity AFTER filtering short paths made web-app-prometheus' host_sys look
single-valued -- it mounts /sys for cadvisor and /host/sys for node-exporter,
and /sys is below the length filter. The template would have rendered an
undefined `.target`. Both the converter and the rule now decide ambiguity over
every target and filter afterwards, and that call reads targets['node-exporter'].
Verified by resolving rather than by reading: all 70 accesses in the diff were
run against the real registry and every one yields a non-empty path equal to the
literal it replaced. make test-lint: 328 passed.
Not verified: no Ansible render, so a call site whose surrounding context cannot
build the applications registry would still fail at deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/schema.yml became meta/secrets.yml, but its content still landed at
applications.<app>.credentials — the one meta file whose payload did not sit
under a root named after the file. Every other topic obeys that rule:
meta/volumes.yml goes to volumes, meta/rbac.yml to rbac. Credentials now go to
secrets.credentials.
The path exists exactly once, in utils/manager/credential_key.py, and
OVERRIDE_SECTION is derived from its two segments rather than spelled beside
them, so the dotted form used by --set and the nested form used by the
inventory cannot drift. Six consumers read it: the loader, the generator, the
CLI override parser, the two emit helpers, and the four lookup validators.
--set now accepts exactly one key shape,
applications.<app>.secrets.credentials.<key>. The three short forms are
rejected instead of guessed at: the same short key names different credentials
on different apps. That also retired primary_app_id, which only existed to
disambiguate them.
Three defects surfaced while wiring this up. _apply_one_role_schema still wrote
to the old root, so an app's own credentials would have moved while its
providers' stayed behind. web-app-pgadmin reads through
lookup('applications', ...).credentials.<key>, an access path no regex over
lookup('config', ...) could see, and would have deployed with an empty postgres
password. And _META_NON_GENERIC_TOPICS still listed 'schema' — a bare topic
name, invisible to any path search — so the generic file-root loop assigned the
raw schema under secrets and the follow-up write landed inside the cached
document, truncating keycloak's four credentials to one for every later
reader.
tests/integration/ansible/lookups/config/test_credentials_roundtrip.py pins both
directions of the seam and the cache invariant that last defect broke.
Existing inventories still carry the pre-rename root; migrating them is
deliberately out of scope here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven include templates under sys-svc-container decided how a container was
probed, and they read their inputs from template-local `{% set %}` scope.
That scope leaks: a value set for one service silently reached every service
below it, which is how taiga's front and gateway ended up probing a port
declared for a different service, and how bluesky's frontend probed a PDS
endpoint.
Probes now live in utils/docker/healthcheck.py, one class per flavor behind a
PROBES registry, and each service declares its own:
healthcheck:
flavor: curl
path: health/ready
start_period: 20m
`port_key` picks another entry under ports.internal for the services that
expose their probe elsewhere (keycloak on management, nextcloud talk on
signaling). Without a flavor the entry must carry an explicit test argv, so a
missing declaration fails loudly instead of rendering a probe against nothing.
Rendering HEAD's templates against the new output confirms 49 of 56 call
sites are byte-identical; the remaining seven are covered by an Ansible run.
Three probes change on purpose: erpnext's backend gains a check it never had,
bluesky's frontend drops the leaked PDS path, and msmtp_curl moves from a
hardcoded localhost:80 to the service's own port on 127.0.0.1 like every
other flavor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
penpot's HikariCP pool fills eagerly and upstream defaults to 60
connections; with 3 swarm replicas that exceeds the non-superuser slots
(max_connections 170 minus reserved) and the post-deploy backup's
pg_dump is rejected with 'remaining connection slots are reserved for
roles with the SUPERUSER attribute'. Cap the pool per replica from the
topology group size. Deliberately not using
POSTGRES_ALLOWED_AVG_CONNECTIONS: its consumer count resolves to 1 and
it is replica-blind; repairing that SPOT is tracked in
https://github.com/infinito-nexus/core/issues/416.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New lint rejects literal port numbers in roles/*/vars/main.yml: the port
belongs in meta/services.yml (services.<svc>.ports.<scope>.<name>) where
band governance and stack rendering read it; vars resolve it via
lookup('config', ...). Migrated the 16 remaining literals across
elasticsearch, registry-docker, erpnext, flowise, funkwhale, magento,
moodle, nextcloud, penpot and shopware.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The version pin sat on the app-level penpot entry, which container_image never
reads, so the frontend/backend/exporter deployed :latest. Since Penpot 2.15 the
backend SSRF-blocks outbound HTTP to hosts resolving to private IPs, so the OIDC
token exchange to the privately-resolving Keycloak failed with unable-to-auth
and login never completed. Pin the three image-bearing services to 2.16.2 (the
release CI actually exercised) and set PENPOT_SSRF_ALLOWED_HOSTS to the Keycloak
host (https? strips either scheme). env.j2 stays shared by compose and swarm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
About 38 web-app/web-svc roles now consume the central svc-db-* and svc-dns engines via lookup('engine', ...) instead of embedded sidecars: meta/services flags, variant matrices, and env/conf hosts point at the shared services. Lets these apps run unpinned and NFS-shareable in swarm.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add plugins/lookup/container_image.py: returns the full `image: "<ref>"`
line so templates drop the redundant image: prefix and surrounding
quotes. Supports `custom=True` to auto-construct `<entity_name>_custom`
for locally-built images, plus explicit `image=`/`version=` overrides
for exotic cases (taiga back-auth/front-oidc variants, decidim infinito
suffix, etc.).
* 25 unit tests for the lookup cover compose vs swarm prefix, custom
flag, overrides, tag_only=True, missing version warnings.
* New strict lint (tests/lint/.../test_no_raw_image_in_compose_templates)
fails on any raw `image:` line in roles/*/templates/*.j2 unless
nocheck-suppressed. Legacy `image: "{{ lookup(...) }}"` form no
longer accepted.
* Migrate 159 image refs across ~80 roles to the standalone
`{{ lookup('container_image', application_id, '<service_key>') }}`
form. 26 roles use `custom=True` (mariadb, postgres, openldap,
jenkins, jira, gitlab, nextcloud, mediawiki, shopware, wordpress,
odoo, joomla, openproject, bookwyrm, espocrm, hugo, littlejs,
fediwall, roulette-wheel, openresty, confluence and more).
* Fix svc-db-redis application_id leak: service.yml.j2 is included
into consuming roles where application_id evaluates to the outer
role, so the image lookup needs the hardcoded 'svc-db-redis'.
* Integration tests (test_templates_require_image,
test_build_requires_image) recognise the standalone lookup as an
implicit `image:` at the same indent.
* Mediawiki swarm tasks: add explicit `:{{ MEDIAWIKI_VERSION }}` to
the registry-prefixed image string because the custom var is now
bare.
Only 3 legitimate exotic nocheck-suppressed image lines remain:
matrix-mdad runner (nested services.matrix.runner.image path),
matrix mautrix bridges (per-iteration dynamic) and svc-registry-docker
(chicken-and-egg self-reference).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The all-off matrix variant pins services.sso.enabled=false while
web-app-keycloak stays in group_names, so the previous group_names-based
defaults yielded disable-login-with-password AND no OIDC — leaving no usable
login method, and the native Playwright test failed (no password form).
Derive both toggles from PENPOT_SSO_ENABLED (the variant-aware sso flag),
matching how the oidc/ldap flags are already computed:
- native_login = not sso (off under OIDC, on otherwise)
- registration = sso (on under OIDC for JIT provisioning)
Drops the services.penpot.native_login / registration_enabled meta keys
(a group_names default can't see variant overrides). README + req-023 updated.
- meta/services.yml: new `penpot.native_login` toggle, default off when
web-app-keycloak is present (forces SSO), on otherwise.
- vars: PENPOT_FLAGS renders enable-/disable-login-with-password from it.
- tasks: administrator local-password bootstrap gated on native_login.
- playwright: test-login-native.js skips when `sso` is enabled.
- meta/main.yml: drop the Figma parenthetical from the description.
- README + req-023 updated.
Biber OIDC login failed the deploy with `?error=registration-disabled`:
Penpot routes first-time OIDC logins through the registration path, so a
disabled flag blocks JIT-provisioning of not-yet-seen federated users
(admin already existed, biber did not). LDAP self-provisions without it.
- meta/services.yml: registration_enabled now follows OIDC (on when
web-app-keycloak is present), off otherwise; native admin is bootstrapped
and LDAP needs no registration. vars/README/req-023 updated to match.
- playwright: split the combined OIDC and LDAP login specs into one file
per persona — test-login-{oidc,ldap}-{admin,biber}.js — wired through the
orchestrator; README Testing section updated.
- spec: split the monolithic Playwright spec into per-surface companions
(_shared.js + test-login-{native,oidc,ldap}.js) required by playwright.spec.js;
OIDC and LDAP cover both administrator and biber; native registers a local
account (skips when OIDC is the login path / registration disabled).
- meta/main.yml: description now describes the application, not the role.
- meta/server.yml: canonical domain -> penpot.design.{{ DOMAIN_PRIMARY }}.
- meta/info.yml: add upstream intro video.
- meta/services.yml + vars: registration_enabled default in services.yml
(off when OIDC enabled, operator-overridable); PENPOT_FLAGS toggles
enable/disable-registration accordingly.
- templates/env.j2: bind PENPOT_SMTP_ENABLED to email.enabled and
PENPOT_LDAP_SSL/STARTTLS to LDAP.SERVER.SECURITY (constants, not literals).
- templates/compose.yml.j2: trim the backend comment to the exception and gate
the JVM CA-trust import behind self-signed TLS (no-op in production); readable
YAML block.
- templates/playwright.env.j2: dotenv_quote APP_BASE_URL; drop redundant blank lines.
Verified live: 5 passed, 6 skipped, 0 failed (OIDC/project/asset skip on this
sso-off deploy; native + LDAP + baseline + guest pass).
Deployed Penpot + Keycloak + OpenLDAP end to end and drove the per-role
Playwright suite green against the live stack (8 passed, 2 skipped):
TLS baseline, OIDC (administrator + biber), LDAP (administrator + biber),
project creation, and image asset upload. The two skips are the generic
biber/administrator persona scenarios, declared blocked via
PERSONA_*_BLOCKED (mirroring web-app-taiga): Penpot's in-app "OpenID"
login entry and SPA user-menu logout are not driveable by the generic
persona helper, so both users' real auth is exercised by the dedicated
OIDC + LDAP scenarios.
Fixes found during bring-up:
- OIDC JVM CA trust: the backend command imports the internal CA into a
writable cacerts copy and points the JVM at it via JAVA_TOOL_OPTIONS.
with-ca-trust.sh covers OS/NSS/env trust but not the JVM truststore, so
server-side token/userinfo calls to Keycloak failed with "PKIX path
building failed". CA path hardcoded (the CA-override double-escapes a $).
- disable-onboarding added to PENPOT_FLAGS so users land on the dashboard.
- Playwright env: ADMIN_EMAIL/BIBER_EMAIL for the LDAP form; alphabetised.
Requirement 235 is fully checked off and archived out of docs/requirements/.
Add a new `web-app-penpot` role deploying Penpot's frontend + backend +
exporter containers behind the standard reverse proxy, with shared
svc-db-postgres and a svc-db-redis sidecar. Wires native OIDC (Keycloak)
and LDAP (OpenLDAP) login via PENPOT_FLAGS, CSP/vhost via sys-stk-full,
backup-ready asset volume, marketplace tags, matrix variants
(all-on/all-off/LDAP-only), and a service-gated Playwright spec with
guest/biber/administrator personas plus separate OIDC and LDAP scenarios.
Implements docs/requirements/235-setup-penpot.md. Live-deploy DoD items
(fresh-host deploy, domain reachability, full in-container `make test`)
remain unverified in the sandbox and are left unchecked.
Remove the entire web-app-penpot role, all associated docs, and all other
changes accumulated on this branch. Working tree now matches origin/main
exactly. Commit history preserved for reference.
- Move playwright spec to roles/web-app-penpot/files/playwright/ (standard location)
- Fix APP_BASE_URL: use normalizeUrl() to strip dotenv_quote double-quotes
- Gate biber/admin tests with safeSkipUnlessEnabled('oidc') (penpot requires OIDC)
- Add meta/variants.yml with all-true, all-false, LDAP-only variants
- Fix templates: use container_depends_on lookup, correct LDAP group vars
- Align services.yml: javascript service static false with nocheck comment
- Update playwright.env.j2: OIDC_SERVICE_ENABLED env var for service gate
- All make test targets pass (lint, test-unit, test-integration, test-lint, test-external)
