Extend test_compose_resource_limits.py to enforce resource limits on every
container a role runs itself (not provided by another role via the service
registry): the primary entity needs min_storage/cpus/mem_reservation/mem_limit/
pids_limit; uncovered sidecar containers need the four resource keys. A service
is exempt only when another role covers it (e.g. discourse's postgres ->
svc-db-postgres, whose entity declares the limits), replacing the previous
blanket shared:True skip.
Declare the previously-missing limits across 18 roles: mostly self-run redis
cache sidecars (cpus "0.5" / 128m / 512m / 512), plus the minio mc client,
the coturn gateway, and the web-svc-* css/html/asset/cdn/file provider entities.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reconcile main's object-store feature and pre-swarm deploy tasks with the branch's compose/networks/volumes SPOT and swarm lints.
objstore (seaweedfs/minio): standalone compose + embedded service templates migrated to the container_image/container_volumes/container_networks/compose_volumes/compose_networks/depends_on lookups; meta/volumes.yml on the canonical dict schema with s3.json and nginx.conf as type:config; embedded services made swarm-compatible via compose_only/compose_restart.
deploy tasks (erpnext, opentalk, seaweedfs, shopware, snipe-it, wordpress, yourls): container exec routed through the container_address resolver and gated on IS_STACK_HOST, shell module plus stack_host_template, no chdir or compose-verb forms.
Also removed orphaned vars, restored MAILU_INSTANCE_DIR, freed the seaweedfs subnet (192.168.206.0/24), stripped literal quotes from the nextcloud/bookwyrm env templates, and routed main's objstore compose_volumes tests through dir_var_lib.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Give each reverse-proxied app a swarm-resolvable upstream: add services.<entity>.ports.internal.http to the role metadata and rename the main compose service from the generic 'application' to the entity name, so its short DNS alias on the shared overlay is unique. Thread the service name through compose, vars (*_SERVICE_NAME), volume mounts and the oauth2 origin host. Consolidate the split image/ports blocks in bridgy-fed and postmarks into the entity block.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Penpot validates raster uploads with ImageMagick `identify` and rejected the
1x1 transparent PNG as :invalid-image ("Seems that this is not a valid image"),
so no asset layer was created and the marker assertion timed out at 60s.
Add a real 32x32 opaque RGB PNG (validImagePng) in _shared.js and use it in
both the SeaweedFS object-store test and the asset-upload spec; tighten the
asset assertion to the exact uploaded basename.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add files/playwright/test-seaweedfs.js (+ SEAWEEDFS_* playwright.env vars, spec require wiring, and removal of the playwright-service-flag nocheck above the seaweedfs service) for baserow, bookwyrm, mattermost, openwebui, peertube, penpot, pixelfed. Each test reuses the role's real login flow and triggers an upload that writes to the app's S3 bucket (file-field, avatar, chat attachment, composer photo, video), verified via the shared runSeaweedfsStorageCheck (bucket grew through the Filer UI).
openwebui: also export dismissAllOpenModals from _shared.js (used by the new test) and match the uploaded file by basename (not the full filename) to avoid the extension-split false-negative.
Exempt gitea and opentalk via meta/services.yml nocheck markers (seaweedfs-playwright + playwright-service-flag): gitea has no automatable UI login in this stack (oauth2-proxy + OIDC account-link / native login both fail; personas blocked); opentalk only writes S3 for recordings/whiteboard exports, which need the recorder service + a live multi-party meeting.
Builds on 9b6c6415f (shared runSeaweedfsStorageCheck helper + consumer lint).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Switch penpot, nextcloud, gitea, mattermost, openwebui, funkwhale, baserow, openproject, pixelfed, mastodon, bookwyrm, listmonk, peertube and opentalk to the engine-agnostic objstore lookup: the minio service entries are pinned off, a seaweedfs service (gated on the provider being in group_names, public for the media apps) replaces them, and the matrix variants pin the new flag. OpenTalk keeps object storage mandatory through an embedded fallback and provisions its bucket via weed shell.
Add objstore lookup SPOT keys consumed by these roles: endpoint (host:port), public_domain, public_bucket (public S3 host plus bucket) and public_url (scheme plus public_bucket, scheme derived from the provider TLS state). This removes the repeated lookup('config', lookup('objstore', ...), 'server.domains.canonical.api') / host:port concatenations from the consumer templates. Listmonk settings render their JSON values via the to_json filter.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Convert the two registry-driven networks templates into pure-Python
rendering functions plus thin Ansible lookup plugins:
roles/sys-svc-compose/templates/networks.yml.j2 -> {{ lookup('compose_networks') }}
roles/sys-svc-container/templates/networks.yml.j2 -> {{ lookup('container_networks') | indent(4) }}
The rendering logic lives in utils/networks/render.py as pure-Python
functions that take a registry + closure-style config/database lookups,
making the logic directly unit-testable without a Jinja stub harness.
The lookup plugins are thin wrappers that resolve application_id,
DEPLOYMENT_MODE, and swarm.network.encryption from the templar, build
the service_registry, and forward to the render functions.
Migrate 233 include lines across 85 production templates. Convert
web-app-matrix's hand-coded MDAD networks block (default + conditional
postgres + conditional openldap) to the generic lookup; this adds an
openresty attachment when services.sso.enabled (harmless extra
interface, pinned in test_matrix_mdad_consumer_attaches_postgres_ldap_sso).
Mark local-DB / redis sidecars (sys-svc-rdbms/templates/services/,
svc-db-redis/templates/service.yml.j2) with `# nocheck: networks-literal`
because they intentionally attach to default-only.
Extend the lint to scan ALL roles/*/templates/**/*.yml.j2 (not only
compose.yml.j2) and to forbid the legacy include form; the marker
above is the documented escape hatch.
Coverage:
- tests/unit/utils/networks/test_render.py (30 cases incl. encrypted=false literal pin)
- tests/unit/plugins/lookup/test_compose_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/unit/plugins/lookup/test_container_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/integration/infrastructure/compose/test_networks_render.py (10 snapshot scenarios)
- tests/integration/infrastructure/compose/test_network_includes.py (3 lint rules)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add plugins/lookup/container_image.py: returns the full `image: "<ref>"`
line so templates drop the redundant image: prefix and surrounding
quotes. Supports `custom=True` to auto-construct `<entity_name>_custom`
for locally-built images, plus explicit `image=`/`version=` overrides
for exotic cases (taiga back-auth/front-oidc variants, decidim infinito
suffix, etc.).
* 25 unit tests for the lookup cover compose vs swarm prefix, custom
flag, overrides, tag_only=True, missing version warnings.
* New strict lint (tests/lint/.../test_no_raw_image_in_compose_templates)
fails on any raw `image:` line in roles/*/templates/*.j2 unless
nocheck-suppressed. Legacy `image: "{{ lookup(...) }}"` form no
longer accepted.
* Migrate 159 image refs across ~80 roles to the standalone
`{{ lookup('container_image', application_id, '<service_key>') }}`
form. 26 roles use `custom=True` (mariadb, postgres, openldap,
jenkins, jira, gitlab, nextcloud, mediawiki, shopware, wordpress,
odoo, joomla, openproject, bookwyrm, espocrm, hugo, littlejs,
fediwall, roulette-wheel, openresty, confluence and more).
* Fix svc-db-redis application_id leak: service.yml.j2 is included
into consuming roles where application_id evaluates to the outer
role, so the image lookup needs the hardcoded 'svc-db-redis'.
* Integration tests (test_templates_require_image,
test_build_requires_image) recognise the standalone lookup as an
implicit `image:` at the same indent.
* Mediawiki swarm tasks: add explicit `:{{ MEDIAWIKI_VERSION }}` to
the registry-prefixed image string because the custom var is now
bare.
Only 3 legitimate exotic nocheck-suppressed image lines remain:
matrix-mdad runner (nested services.matrix.runner.image path),
matrix mautrix bridges (per-iteration dynamic) and svc-registry-docker
(chicken-and-egg self-reference).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Penpot validates raster uploads with ImageMagick `identify` and rejected the
1x1 transparent PNG as :invalid-image ("Seems that this is not a valid image"),
so no asset layer was created and the marker assertion timed out at 60s.
Add a real 32x32 opaque RGB PNG (validImagePng) in _shared.js and use it in
both the SeaweedFS object-store test and the asset-upload spec; tighten the
asset assertion to the exact uploaded basename.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add files/playwright/test-seaweedfs.js (+ SEAWEEDFS_* playwright.env vars, spec require wiring, and removal of the playwright-service-flag nocheck above the seaweedfs service) for baserow, bookwyrm, mattermost, openwebui, peertube, penpot, pixelfed. Each test reuses the role's real login flow and triggers an upload that writes to the app's S3 bucket (file-field, avatar, chat attachment, composer photo, video), verified via the shared runSeaweedfsStorageCheck (bucket grew through the Filer UI).
openwebui: also export dismissAllOpenModals from _shared.js (used by the new test) and match the uploaded file by basename (not the full filename) to avoid the extension-split false-negative.
Exempt gitea and opentalk via meta/services.yml nocheck markers (seaweedfs-playwright + playwright-service-flag): gitea has no automatable UI login in this stack (oauth2-proxy + OIDC account-link / native login both fail; personas blocked); opentalk only writes S3 for recordings/whiteboard exports, which need the recorder service + a live multi-party meeting.
Builds on 9b6c6415f (shared runSeaweedfsStorageCheck helper + consumer lint).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Switch penpot, nextcloud, gitea, mattermost, openwebui, funkwhale, baserow, openproject, pixelfed, mastodon, bookwyrm, listmonk, peertube and opentalk to the engine-agnostic objstore lookup: the minio service entries are pinned off, a seaweedfs service (gated on the provider being in group_names, public for the media apps) replaces them, and the matrix variants pin the new flag. OpenTalk keeps object storage mandatory through an embedded fallback and provisions its bucket via weed shell.
Add objstore lookup SPOT keys consumed by these roles: endpoint (host:port), public_domain, public_bucket (public S3 host plus bucket) and public_url (scheme plus public_bucket, scheme derived from the provider TLS state). This removes the repeated lookup('config', lookup('objstore', ...), 'server.domains.canonical.api') / host:port concatenations from the consumer templates. Listmonk settings render their JSON values via the to_json filter.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The all-off matrix variant pins services.sso.enabled=false while
web-app-keycloak stays in group_names, so the previous group_names-based
defaults yielded disable-login-with-password AND no OIDC — leaving no usable
login method, and the native Playwright test failed (no password form).
Derive both toggles from PENPOT_SSO_ENABLED (the variant-aware sso flag),
matching how the oidc/ldap flags are already computed:
- native_login = not sso (off under OIDC, on otherwise)
- registration = sso (on under OIDC for JIT provisioning)
Drops the services.penpot.native_login / registration_enabled meta keys
(a group_names default can't see variant overrides). README + req-023 updated.
- meta/services.yml: new `penpot.native_login` toggle, default off when
web-app-keycloak is present (forces SSO), on otherwise.
- vars: PENPOT_FLAGS renders enable-/disable-login-with-password from it.
- tasks: administrator local-password bootstrap gated on native_login.
- playwright: test-login-native.js skips when `sso` is enabled.
- meta/main.yml: drop the Figma parenthetical from the description.
- README + req-023 updated.
Biber OIDC login failed the deploy with `?error=registration-disabled`:
Penpot routes first-time OIDC logins through the registration path, so a
disabled flag blocks JIT-provisioning of not-yet-seen federated users
(admin already existed, biber did not). LDAP self-provisions without it.
- meta/services.yml: registration_enabled now follows OIDC (on when
web-app-keycloak is present), off otherwise; native admin is bootstrapped
and LDAP needs no registration. vars/README/req-023 updated to match.
- playwright: split the combined OIDC and LDAP login specs into one file
per persona — test-login-{oidc,ldap}-{admin,biber}.js — wired through the
orchestrator; README Testing section updated.
- tasks/main.yml: remove `| default('')` from changed_when on command
results (stdout always set when the conditional fires) per
test_no_redundant_default_on_module_io.
- env.j2: drop `| default('')` on LDAP.SERVER.SECURITY (defined
unconditionally in group_vars/all/12_ldap.yml) per
test_no_jinja_default_on_spot_path.
- spec: split the monolithic Playwright spec into per-surface companions
(_shared.js + test-login-{native,oidc,ldap}.js) required by playwright.spec.js;
OIDC and LDAP cover both administrator and biber; native registers a local
account (skips when OIDC is the login path / registration disabled).
- meta/main.yml: description now describes the application, not the role.
- meta/server.yml: canonical domain -> penpot.design.{{ DOMAIN_PRIMARY }}.
- meta/info.yml: add upstream intro video.
- meta/services.yml + vars: registration_enabled default in services.yml
(off when OIDC enabled, operator-overridable); PENPOT_FLAGS toggles
enable/disable-registration accordingly.
- templates/env.j2: bind PENPOT_SMTP_ENABLED to email.enabled and
PENPOT_LDAP_SSL/STARTTLS to LDAP.SERVER.SECURITY (constants, not literals).
- templates/compose.yml.j2: trim the backend comment to the exception and gate
the JVM CA-trust import behind self-signed TLS (no-op in production); readable
YAML block.
- templates/playwright.env.j2: dotenv_quote APP_BASE_URL; drop redundant blank lines.
Verified live: 5 passed, 6 skipped, 0 failed (OIDC/project/asset skip on this
sso-off deploy; native + LDAP + baseline + guest pass).
Upstream main added web-app-erpnext on 192.168.105.176/28, which now
collides with web-app-penpot after the rebase. test_unique_subnets /
test_no_overlapping_subnets failed in manual CI (run 27055475680).
Reassign penpot to the next free /28 (192.168.105.192/28).
Manual CI (run 27030584621) failed two branch-caused jobs:
- lint-javascript: no-unused-vars on `expectedBase` in the spec's
penpotLdapLogin (the LDAP path asserts on /auth/login, not the base
URL). Remove the dead binding (eslint now 0 errors).
- images-mirror-missing: the `penpot` metadata holder carried
`image: penpotapp`, so the mirror tried docker.io/library/penpotapp:2.5.4
(access denied — not a real image). The real images live on the
frontend/backend/exporter entities; the holder only needs `version`
(the shared tag). Drop the holder image (matches taiga/openproject).
Deployed Penpot + Keycloak + OpenLDAP end to end and drove the per-role
Playwright suite green against the live stack (8 passed, 2 skipped):
TLS baseline, OIDC (administrator + biber), LDAP (administrator + biber),
project creation, and image asset upload. The two skips are the generic
biber/administrator persona scenarios, declared blocked via
PERSONA_*_BLOCKED (mirroring web-app-taiga): Penpot's in-app "OpenID"
login entry and SPA user-menu logout are not driveable by the generic
persona helper, so both users' real auth is exercised by the dedicated
OIDC + LDAP scenarios.
Fixes found during bring-up:
- OIDC JVM CA trust: the backend command imports the internal CA into a
writable cacerts copy and points the JVM at it via JAVA_TOOL_OPTIONS.
with-ca-trust.sh covers OS/NSS/env trust but not the JVM truststore, so
server-side token/userinfo calls to Keycloak failed with "PKIX path
building failed". CA path hardcoded (the CA-override double-escapes a $).
- disable-onboarding added to PENPOT_FLAGS so users land on the dashboard.
- Playwright env: ADMIN_EMAIL/BIBER_EMAIL for the LDAP form; alphabetised.
Requirement 235 is fully checked off and archived out of docs/requirements/.
Add a new `web-app-penpot` role deploying Penpot's frontend + backend +
exporter containers behind the standard reverse proxy, with shared
svc-db-postgres and a svc-db-redis sidecar. Wires native OIDC (Keycloak)
and LDAP (OpenLDAP) login via PENPOT_FLAGS, CSP/vhost via sys-stk-full,
backup-ready asset volume, marketplace tags, matrix variants
(all-on/all-off/LDAP-only), and a service-gated Playwright spec with
guest/biber/administrator personas plus separate OIDC and LDAP scenarios.
Implements docs/requirements/235-setup-penpot.md. Live-deploy DoD items
(fresh-host deploy, domain reachability, full in-container `make test`)
remain unverified in the sandbox and are left unchecked.
Remove the entire web-app-penpot role, all associated docs, and all other
changes accumulated on this branch. Working tree now matches origin/main
exactly. Commit history preserved for reference.
- Move playwright spec to roles/web-app-penpot/files/playwright/ (standard location)
- Fix APP_BASE_URL: use normalizeUrl() to strip dotenv_quote double-quotes
- Gate biber/admin tests with safeSkipUnlessEnabled('oidc') (penpot requires OIDC)
- Add meta/variants.yml with all-true, all-false, LDAP-only variants
- Fix templates: use container_depends_on lookup, correct LDAP group vars
- Align services.yml: javascript service static false with nocheck comment
- Update playwright.env.j2: OIDC_SERVICE_ENABLED env var for service gate
- All make test targets pass (lint, test-unit, test-integration, test-lint, test-external)
kevinveenbirkenbachandClaude Opus 4.8
evangelostsak