The baserow (infinito_sso.py) and bookwyrm (header_auth.py) trusted-header bridges now read identity only from the X-Forwarded-* headers nginx overwrites from the oauth2-proxy auth response. The client-injectable X-Auth-Request-*/Remote-User/X-Forwarded-Name variants are dropped, closing an authenticated-user header-injection escalation (including a groups->superuser path) in the claim-less edge case where X-Forwarded-* would be empty.
Add SSO security e2e tests (test-oidc-security.js) to both roles: forged-header gate bounce, trusted-header bridge endpoint gating, SSO-disabled inertness, and injected-header re-identification of an authenticated session. Wire the new spec into baserow's playwright.spec.js.
Update the bookwyrm header_auth unit test: precedence now uses two listed headers, plus a regression guard asserting proxy-uncontrolled headers are ignored.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
OpenTalk's images (web-frontend, controller, recorder) live on
registry.opencode.de, which was absent from both _MIRRORABLE_REGISTRIES and
_ALL_REGISTRY_PREFIXES in utils/docker/image/discovery.py. So is_mirrorable_image
returned False, iter_role_images skipped them, the GHCR mirror never copied
them, and the deploy pulled direct from registry.opencode.de -> TLS handshake
timeout (deploy failed).
Add the registry to both sets so the single iter_role_images gate enrolls the
3 images: the mirror-sync copies them to GHCR and the resolver rewrites the
deploy to pull from the GHCR mirror. The prefix entry strips the host so the
mirror destination is well-formed (no doubled registry). Add a regression test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Baserow's native SSO is Enterprise-only, so gate the community image with the shared Keycloak oauth2-proxy and patch urls.py (via a custom Dockerfile) with a trusted-header bridge. The bridge provisions/repairs a local Baserow user from the oauth2-proxy identity headers and mints native refresh/access/user_session tokens, mirroring Baserow's own SSO redirect contract (?token=&user_session=).
proxy.conf.j2 performs the nginx handoff: an authenticated browser without a jwt_token cookie is 302'd to /api/infinito/sso/login/, which redirects back to /?token=&user_session= for the frontend middleware to consume. bootstrap_admin marks the superuser email_verified + completed_onboarding to skip onboarding.
Replace the LDAP service/variant with proxy-gating and rewrite the seaweedfs + persona Playwright tests to drive Baserow through its API using the SSO-minted JWT.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BookWyrm core has no OIDC/LDAP login, so behind the oauth2-proxy a gated visitor only reaches BookWyrm's own login form. Add a trusted-header SSO bridge (files/sso/header_auth.py: RemoteUser middleware + backend) that turns the oauth2-proxy identity (X-Forwarded-Preferred-Username, set by Keycloak after OIDC or LDAP-federated login) into a real BookWyrm session, auto-provisioning the local user. Active only when PROXY_HEADER_SSO is truthy (= sso is_proxy_gated, centralised in BOOKWYRM_SSO_PROXY_GATED).
Supporting fixes: USE_S3_FOR_EXPORTS=false (exports stay local; SeaweedFS has no per-bucket ACL so private exports must not share the public media bucket) fixes the fresh-build settings crash / preferences 502; proxy_pass_request_body off on the keycloak sso_proxy /oauth2/ block so the auth_request subrequest no longer hangs 60s on gated POST bodies (504->500); split playwright.spec.js into per-test files; add positive (oidc/ldap -> authenticated session) + negative (forged header -> gated) SSO e2e tests; fix the avatar test (form-scoped Save locator + valid PNG); remove dead BookWyrm OIDC_* env; unit tests for the bridge + the gated settings patch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Render the S3 objectstore config (templates/config/objectstore.config.php.j2) into NC's config dir before install, gated on objstore.enabled — NC cannot switch an already-installed instance to objectstore primary storage, so it must be present at install time.
Add the SeaweedFS consumer storage Playwright test (files/playwright/test-seaweedfs.js, playwright.env.j2 SEAWEEDFS_* vars, spec wiring; drop the playwright-service-flag nocheck for the seaweedfs service) plus the shared runSeaweedfsStorageCheck helper (admin logs into the Filer once, counts bucket objects via the Filer JSON listing, asserts growth) and the seaweedfs-consumer integration lint.
web-app-seaweedfs: bind weed to 0.0.0.0 (-ip.bind) so the multi-homed gateway is reachable by consumers that resolve the service name to its IP on the shared network.
nextcloud_install: classify 'was not found on the appstore' as unavailable (skippable, non-fatal) so a plugin without a release for the pinned NC version does not abort the whole deploy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The oauth2-proxy sidecar set cookie_domains to the app's primary domain only. A role served under several sibling subdomains (e.g. SeaweedFS filer/master/api under one base) therefore could not read its session cookie on the non-primary hosts, looping the auth chain. Add the common_dns_suffix filter and use it so cookie_domains becomes the longest DNS suffix shared by all of the app's domains: a single-domain app keeps its exact host (unchanged), a multi-domain app collapses to the shared parent so one session cookie spans every subdomain.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The auto-generated "Older Releases" index in CHANGELOG.md emitted dash
bullets while the rest of the file uses asterisks, failing markdownlint
MD004 (ul-style, consistent) with 49 errors. Switch the archive index
generator to asterisk bullets, convert the existing entries, and align
the unit test assertions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a meta lint asserting every web-app role with postgres or mariadb declares a seaweedfs object-store service, exemptable via the seaweedfs-required nocheck marker. Mark the 33 database-backed roles that do not use object storage, and register the rule in the suppression catalog.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Switch penpot, nextcloud, gitea, mattermost, openwebui, funkwhale, baserow, openproject, pixelfed, mastodon, bookwyrm, listmonk, peertube and opentalk to the engine-agnostic objstore lookup: the minio service entries are pinned off, a seaweedfs service (gated on the provider being in group_names, public for the media apps) replaces them, and the matrix variants pin the new flag. OpenTalk keeps object storage mandatory through an embedded fallback and provisions its bucket via weed shell.
Add objstore lookup SPOT keys consumed by these roles: endpoint (host:port), public_domain, public_bucket (public S3 host plus bucket) and public_url (scheme plus public_bucket, scheme derived from the provider TLS state). This removes the repeated lookup('config', lookup('objstore', ...), 'server.domains.canonical.api') / host:port concatenations from the consumer templates. Listmonk settings render their JSON values via the to_json filter.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a database-style object-store abstraction: the objstore lookup resolves the active engine (seaweedfs|minio) per consumer and yields a uniform S3 payload, objstore_secret_key is auto-generated like database_password, sys-svc-objstore delegates per-consumer provisioning to the resolved provider's objstore_init entrypoint (lazy-loading the provider when it was not preloaded), and compose/network/volume plumbing supports both a shared central instance and an embedded storage-only instance per consumer.
Add the web-app-seaweedfs provider role: Filer and Master UIs behind the admin-only oauth2-proxy via a role-local nginx and proxy.conf.j2 (S3 API domain stays ungated, admin UIs deny with 403 when SSO is disabled), per-consumer S3 identities from s3.json, idempotent bucket provisioning via weed shell, a /24 on the central-provider subnet range, Playwright coverage for admin/biber/guest plus consumer bucket existence, and unit tests for the objstore lookup and the per-engine compose volumes.
Mark web-app-minio lifecycle: unsupported, move its mc client image into a dedicated services.client entry, align it with the objstore credential contract, and make its consumer provisioning idempotent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add tests/lint/docker/dockerfile/test_from_parameterized.py to fail any
roles/*/{files,templates}/**/Dockerfile* whose FROM does not reference an
ARG (${VAR}) or Jinja variable ({{ var }}). FROM scratch and a
`# nocheck: from-parameterized` per-line marker are exempt.
Refactor web-app-bluesky login-broker Dockerfile to satisfy the rule:
surface the base image and version via services.broker.{base_image,base_version}
-> vars/main.yml -> docker-compose build args -> ARGs in the Dockerfile.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a detect-affected-roles job to entry-pull-request-change.yml that
runs the affected-roles resolver upfront on host python (via make
install) and emits whitelist + roles_only. ci-orchestrator skips
test-development when roles_only=true; done job tolerates the skip.
entry-manual derives roles_only from its whitelist input.
affected_roles.sh now invokes the resolver host-side only (compose-exec
fallback removed; it was a no-op in every existing caller because
compose was never up there, always emitting __ALL__ fail-safe). The
PR-flow gate consolidates what test-deploy-* used to recompute per
job; downstream effective_whitelist.sh now consumes the pre-resolved
whitelist verbatim.
Test fixture switched from a fake docker binary to a fake PYTHON
binary and clears BASH_ENV so the bash-startup env loader does not
reset PYTHON.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root cause of step-08 deploy failures across all 4 test-development
distros in run 26572559126: `make compose-deploy mode=reinstall
apps=web-app-dashboard INFINITO_SERVICES_DISABLED=matomo` (step 07)
made make export the cmdline var into the recipe's shell env, which
reached `runner.py` → `make setup` → `make dotenv`. The dotenv
generator's `setdefault` then baked the runtime-only value into the
on-disk `.env`. Step 08 sourced that .env, filtered `web-app-matomo`
out of its primary apps, and exited with "All primary apps disabled".
Fix:
- Drop the 7 routing knobs (apps, mode, purge, bundles, disable,
full_cycle, variant) from `passthrough.STATIC_KEYS` and from
`default.env`. They are per-invocation; .env is the persistent SPOT
and must not capture them.
- Switch every consumer from `os.environ["KEY"]` strict access to
`os.environ.get("KEY", "")` so the missing-key path stops at "no
filter" instead of a KeyError.
- Rename the env-var surface to match the make-alias surface 1:1
(lowercase): INFINITO_APPS → apps, INFINITO_SERVICES_DISABLED →
disable, INFINITO_BUNDLES → bundles, INFINITO_FULL_CYCLE →
full_cycle, INFINITO_VARIANT → variant, INFINITO_DEPLOY_MODE → mode,
INFINITO_PURGE_ENTITIES → purge. The make alias was the user-facing
name in every doc; routing-knob env vars now use the same name so
the cmdline form, the recipe env-prefix, and the consumer reads
all line up.
- Collapse the Makefile `compose-deploy` recipe to just the two
knobs whose alias differs from the env name (`type` →
INFINITO_DEPLOY_TYPE, `debug` → INFINITO_DEBUG). GNU Make
auto-exports cmdline variables into the recipe's shell env, so the
identical-name `$(if $(apps),apps="$(apps)")` lines were pure
noise.
- Drop `disable: ${disable}` from compose.yml. The dev CLI builds
`extra_env` per `docker exec` call and forwards `disable` via
`-e`, so the container env never needs the value frozen at
compose-up time.
- Update the 3 actual leak call sites
(07/08/09_deploy_*.sh) to the make-alias form
(`disable=matomo`, `full_cycle=true`, `variant=1`).
- Add `SIM112` to ruff's ignore set: the lowercase env names are
the deliberate make/env parity, not a typo.
Repro before fix: `INFINITO_SERVICES_DISABLED=matomo make dotenv`
wrote `INFINITO_SERVICES_DISABLED=matomo` to `.env`. After fix:
`disable=matomo make dotenv` leaves `.env` untouched.
Tests: `make test` and `make autoformat` green (2711 tests, 8 lint
sub-targets).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Installs kpmx and runs `pkgmgr archive docs/requirements` in one
shot. The requirements-completeness lint failure message now points
contributors at this target instead of asking them to invoke pkgmgr
directly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The lint test only used two tiny helpers from kpmx
(iter_archivable_files, count_unchecked_items). Inlining them is ~15
lines and lets us drop kpmx from dev dependencies entirely, so neither
host nor container has to install the package just for a lint that
walks docs/requirements/ and counts task-list markers. The
pkgmgr archive CLI remains the canonical archival tool; the failure
message still points there.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The fully-checked-Markdown archive helper used to live under
cli/contributing/requirements/archive/ in this repo. It is a generic
spec-folder-management primitive (numbered NNN-topic.md files,
task-list completeness check, README ## Archive section append) and
now ships in pkgmgr.actions.archive so every kpmx-managed repository
can rely on the same convention.
Changes here:
* tests/lint/repository/documentation/test_requirements_completeness.py
imports the archive primitives from pkgmgr.actions.archive instead
of the vendored cli.contributing.requirements.archive, and points
contributors at `pkgmgr archive docs/requirements` in the failure
message.
* cli/contributing/requirements/README.md becomes a stub that
redirects readers to the kpmx archive subcommand.
* The vendored cli/contributing/requirements/archive/ package and its
unit tests under tests/unit/ are deleted.
* The pkgmgr role pin bumps to v1.15.0 to pick up the new archive
subcommand on deployed hosts.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`web-opt-rdr-www` generates a redirect vhost
`<NGINX_DIR>/conf.d/servers/{http,https}/www.<domain>.conf` for every
deployed app's canonical and alias domains. Those redirect vhosts are
not part of any app's `server.domains.{canonical,aliases}` list, so the
entity-keyed purge primitive in `utils/cleanup/nginx_vhosts.py` did
not yield them and `iter_vhost_files_for_entity` left them on disk.
Between matrix-deploy rounds this leaked the redirect vhosts of the
previous round's broader app set into the next round's narrower one.
On run 26451828677 job 77879571696 (web-opt-rdr-www on debian) the
round 1 -> round 2 transition
round 1/2 inv=github-0 apps=[15 apps incl. web-svc-cdn,
web-svc-simpleicons, web-svc-css, ...]
round 2/2 inv=github-1 apps=[web-opt-rdr-www]
dropped the bare `cdn.infinito.example.conf` etc. via the existing
purge but kept the matching `www.cdn.infinito.example.conf`,
`www.icon.infinito.example.conf`, ... behind. Round 2 only redeploys
web-opt-rdr-www, whose `www_redirect_domains` lookup now only emits
its own `www.w3redirect.<primary>` redirect, so nothing reissues the
stale www-prefixed vhosts.
`sys-ctl-hlth-csp` discovers domains by scanning .conf filenames in
the nginx servers dir, so it still probed the stale `www.<x>.conf`
hosts. nginx served them with a cert whose SAN no longer covered the
hostname (the per-domain cert path baked into the conf had been
removed alongside the bare vhost), the TLS handshake returned
`net::ERR_SSL_UNRECOGNIZED_NAME_ALERT`, the CSP probe exited 11, and
`sys-service-terminator : Apply state` failed after 10 retries.
Extend `iter_vhost_files_for_entity` to also yield the
`www.<domain>.conf` variant for every domain it already yields,
skipping domains that already begin with `www.` so a role that
declares its canonical with a leading `www.` (web-opt-rdr-www's own
`www.w3redirect.<primary>`) does not produce `www.www.<x>.conf`. The
existing two- and three-step purge primitives (db / compose / dir /
nginx) and the orchestrator in `scripts/container/purge/apps.sh` keep
their entity-keyed contract; only the per-entity yield set widens.
Coverage:
- test_yields_www_redirect_variant: per-app www-prefixed file is
yielded alongside the bare vhost.
- test_www_prefixed_domain_not_double_prefixed: a www-prefixed
canonical does not synthesize `www.www.<x>` even when such a file
exists on disk (defense in depth against a future regression that
prepends unconditionally).
- test_removes_www_redirect_variant: end-to-end purge removes both
bare and www variant, leaves an unrelated www-prefixed file alone.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* TC003: move `pathlib.Path` into `TYPE_CHECKING` blocks across the
new `requirements/archive` package and its completeness lint test.
* TC001: same treatment for `VaultHandler` in
`credentials/emit.py` (annotation-only).
* SIM105/S110: replace `try/except/pass` in
`current_play_redirect_domains.py` with
`contextlib.suppress(Exception)`.
* `archive/__main__.py`: mark `README_PATH.read_text(...)` with
`# nocheck: cache-read` — CLI rewrites the README in the same flow,
mirroring the `changelog/archive` pattern.
* Drive-by ruff-format condensations in the archive unit tests.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* `python -m cli.contributing.requirements.archive` moves the H1 of
every fully-checked `docs/requirements/NNN-*.md` into the
`## Archive` block of the directory README and deletes the file.
Files with any remaining `- [ ]` are skipped. Split SRP-wise into
discovery / inspect / readme / __main__.
* New lint `test_requirements_completeness` warns per unchecked
acceptance criterion (GitHub annotation) and fails when any
requirement is ready for archival, pointing at the CLI. Keeps the
requirements directory small so AI agents process less context.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The plugin's previous
domain_primary = str(vars_.get("DOMAIN_PRIMARY") or "")
domain_homepage = str(vars_.get("DOMAIN_HOMEPAGE") or "")
read inventory values that may carry Ansible 2.18+'s
``_AnsibleTaggedStr`` with the ``TrustedAsTemplate`` tag — exactly
the shape ``inventories/development/default.yml:20`` produces with
DOMAIN_PRIMARY: "{{ lookup('env', 'INFINITO_DOMAIN') }}"
A bare ``str(value)`` casts the tag away and leaves the literal Jinja
text in the plugin output. The output then flows into
``CURRENT_PLAY_REDIRECT_DOMAINS`` via the set_fact in
``tasks/stages/01_constructor.yml`` and lands in
``roles/web-opt-rdr-domains/tasks/main.yml`` loop items as
item={'source': "{{ lookup('env','INFINITO_DOMAIN') }}", 'target': "{{ … }}"}
The ``lookup('tls', domain, 'enabled')`` call in
``roles/web-opt-rdr-domains/tasks/redirect-domain.yml:5`` then
receives the literal Jinja string and raises
The lookup plugin 'tls' failed: tls: application_id
'{{ lookup('env', 'INFINITO_DOMAIN') }}' not found in domains mapping
(job 77797371504 on run 26428080957). That CI ran on SHA d1410c0be,
which already shipped this plugin — so the upstream replacement of
the inline two-stage set_fact merges with the lookup plugin did not
fix the underlying trust-tag-strip bug; it just relocated it from
Jinja to Python.
Add ``_resolve_str`` that runs the value through ``self._templar``
before string-coercion. Idempotent for already-resolved strings,
exception-safe (falls back to ``str(raw)`` if the templar is
unavailable, e.g. on hosts without rendered vars). Both
``DOMAIN_PRIMARY`` and ``DOMAIN_HOMEPAGE`` route through it.
Defense-in-depth with the existing ``plugins/lookup/tls.py``
defensive templating (commit f32266d53): even if a future code path
hands ``lookup('tls', …)`` an untemplated string, the tls plugin
itself now templates the first term. Either layer alone suffices for
the rdr-domains failure; both together guarantee the symptom cannot
re-surface from a single regression elsewhere.
Coverage: two regression tests in
``tests/unit/plugins/lookup/test_current_play_redirect_domains.py``
— ``test_jinja_tagged_domain_primary_is_templated`` proves the
templar call happens and produces the resolved domain, and
``test_templar_failure_falls_back_to_raw_value`` proves a raising
templar does not crash the plugin. The 10 existing tests stay green.
``make test`` green across all 5 targets.
Class follow-up: 11 further lookup plugins read inventory values
without ``templar.template`` first (see git log for ``vars_.get``
call sites). Not in scope for this fix; logged for a separate
audit pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Six ``uri_retry`` tasks under ``roles/web-app-xwiki/tasks/`` carried
their XML payload as inline ``body: |`` blocks (CheckExtension Groovy
page, InstallExtensions Groovy page, the administrator user-page
PUT/POST pair, and the post-Fix-#8 XWikiAdminGroup PUT/POST pair).
Inline XML mixes Ansible YAML indentation with Jinja control flow
(``{% raw %}``-escaping for ``{{groovy}}`` delimiters), defeats
editor XML support, and silently rots when a future contributor
copy/edits the snippet without realising the surrounding ``body: |``
context.
Split by concern:
- ``templates/xml/*.xml.j2`` — six XML files that embed Jinja:
``user_page``, ``user_xwikiusers_object``,
``xwiki_admin_group_page``, ``xwiki_admin_group_member_object``,
``install_extensions_page``, ``main_webhome_page`` (the last one
was already a template; moved here for symmetry with the new five).
- ``files/xml/check_extension_page.xml`` — fully static; rewritten
without ``{% raw %}{{groovy}}{% endraw %}`` because the file is
served via ``lookup('file', …)`` (no Jinja pass), so ``{{groovy}}``
reaches XWiki literally.
Task references migrate to ``body: "{{ lookup('template',
'xml/<name>.xml.j2') }}"`` (or ``lookup('file', 'xml/<name>.xml')``
for the static case), normalising the previously bespoke long-form
``roles/web-app-xwiki/templates/main_webhome_page.xml.j2`` lookup in
the bargain.
SPOT-extract the magic string ``XWikiAdminGroup``:
- ``XWIKI_ADMIN_GROUP_PAGE: "XWikiAdminGroup"`` in vars/main.yml,
documented as the page-name part used by REST URLs, the
XAR-shipped page, the LDAP ``group_mapping`` argument, and the
OIDC ``groups.mapping`` argument.
- Five call sites updated to ``{{ XWIKI_ADMIN_GROUP_PAGE }}`` or
bare ``XWIKI_ADMIN_GROUP_PAGE`` inside list contexts:
``tasks/03_administrator.yml`` (×4 — two task names, the PUT URL,
the POST URL), ``templates/xml/xwiki_admin_group_page.xml.j2``,
``templates/compose.yml.j2`` (LDAP group_mapping), and
``templates/xwiki.properties.j2`` (OIDC groups.mapping, in a
commented example).
Lint guard: new ``tests/lint/ansible/tasks/test_no_inline_xml_body.py``
scans every ``roles/`` and ``tasks/`` YAML for ``body: |`` (or
``body: >``) folded scalars whose first content line starts with
``<`` and fails with a pointer at the matching ``templates/xml/...``
or ``files/xml/...`` alternative. JSON / shell / SQL bodies stay
allowed. Per-task opt-out: ``# nocheck: inline-xml-body`` on the
``body:`` line or the line above. Five unit tests cover the
scanner: matches XML body, ignores JSON body, honours same-line
nocheck, honours line-above nocheck, repo-wide sweep clean after
this refactor.
XWiki v17.10.8 distribution-merge behaviour was cross-checked
against upstream (``AbstractXarEntryType.upgradeType =
THREEWAYS``); the empty ``<member/>`` placeholder in the
XAR-shipped ``XWikiAdminGroup`` cannot overwrite our explicitly-added
member (different GUID / number).
``make test`` green across all 5 targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The inventory validator at cli/administration/inventory/validate
walked `recursive_keys(host_vars) ⊆ recursive_keys(application_defaults)`
and flagged any extra key with "Missing default for <app>: <key>".
After commit 10cb02461 ("bake only variants.yml overrides into
matrix-deploy --vars") host_vars carry every key the active
variants.yml overlay declares. Since 29377e1bd
(`_build_application_defaults` reads only `meta/<topic>.yml`, not
variants.yml — to keep the variant-0 service-key leak closed), the
defaults' allow-set is artificially narrower than the role contract:
roles/web-opt-rdr-www declares `services.dashboard.{enabled,shared}`
exclusively in `meta/variants.yml`, so the validator rejects the
baked host_vars with three "Missing default" entries before
ansible-playbook even starts (job 77797115291 on run 26428083128,
distro=debian; distro-agnostic per the matrix).
Widen the validator's legal-key set per app to
recursive_keys(defaults[app]) ∪ ⋃ recursive_keys(variants[app][i])
variants.yml IS part of the role contract; the validator's previous
"defaults only" view was incomplete. Runtime
`_build_application_defaults` stays narrow (so 29377e1bd's
variant-0-leak fix stays in place) — only the validator widens. Typos
not declared by any variant remain caught: a new
`test_key_typo_in_variants_still_rejected` unit test pins that.
Package split (KISS/DRY/SPOT), mirroring the credentials/ refactor:
`validate/__main__.py` carried six orthogonal concerns plus
orchestration in 175 LoC. Split:
- `validate/keys.py`: `recursive_keys` (pure helper).
- `validate/loaders.py`: `load_yaml_file` (with vault-block stripping)
and `load_inventory_files` (directory scan).
- `validate/applications.py`: `compare_application_keys` (the widened
variants-aware logic).
- `validate/users.py`: `compare_user_keys`.
- `validate/hosts.py`: `validate_host_keys`.
- `validate/__main__.py`: thin orchestrator — argparse + load defaults/
variants/users + invoke compare_* + exit code. 75 LoC.
`validate/__init__.py` re-exports the public API
(`main`, `compare_application_keys`, `compare_user_keys`,
`load_inventory_files`, `load_yaml_file`, `recursive_keys`,
`validate_host_keys`, `PROJECT_ROOT`). External callers
(`tests/unit/cli/administration/inventory/validate/test_main.py`,
`tests/unit/cli/administration/inventory/validate/test_hosts.py`)
keep working unchanged via this re-export.
`__main__.py` uses ABSOLUTE imports (`from cli.administration.inventory.
validate.applications import ...`) instead of relative because it's
invoked as a subprocess by tests and by the deploy entrypoint; the
sys.path bootstrap at the top of the file makes `cli.…` resolvable
in that mode.
Test layout follows the split — one test file per module:
- `test_keys.py` (6 tests: empty/non-dict/flat/nested/deep/prefix).
- `test_loaders.py` (8 tests: vault stripping, missing file, broken
YAML, top-level + recursive directory scan, empty-block skip).
- `test_applications.py` (9 tests: all-keys-match, unknown app, extra
key, credentials skip, variants legal, variants union, no-variants
fallback, source path in error, typo still rejected).
- `test_users.py` (5 tests: match, unknown user warn, missing default,
password skip, credentials skip).
- `test_hosts.py` (renamed from test_inventory_hosts.py — unit tests
for `validate_host_keys`).
- `test_main.py` (renamed from test_inventory.py — end-to-end
subprocess tests for `main()`, including the
`test_variants_only_key_is_accepted` regression guard for this fix).
Coverage: 36 tests, all green. `make test` green across all 5 targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The credentials baker at cli/administration/inventory/credentials/__main__.py
walked `schema_creds.items()` one level deep and called `str(value)` on
every leaf. For nested credential groups like
``credentials.recaptcha = {key, secret}`` (declared in
roles/web-app-espocrm/meta/schema.yml and
roles/web-app-listmonk/meta/schema.yml) this collapsed the inner dict
into its Python repr, vault-encrypted that blob, and wrote it as a
single scalar at ``credentials.recaptcha``. The runtime ``config``
lookup then choked on the descent into ``recaptcha.key``:
The lookup plugin 'config' failed: Expected dict for 'key', got
_AnsibleTaggedStr at 'key'
Full path so far: applications['web-app-espocrm'].credentials.recaptcha.key
Current object: "{'key': '{{ CAPTCHA.RECAPTCHA.KEY }}', 'secret':
'{{ CAPTCHA.RECAPTCHA.SECRET }}'}"
(jobs 77797371397 espocrm and 77797371442 listmonk on run 26428080957.)
Introduce ``emit_credentials`` (new module: ``credentials/emit.py``)
that walks the schema recursively. Scalar leaves still vault-encrypt
via ``to_vault_block``; dict nodes recurse into a nested CommentedMap
via ``ensure_map``. The walker also tracks dotted full-key paths so
the existing ``--force``-mode override-apply logic sees the same dotted
keys it always saw.
Auto-healing for previously-corrupted host_vars: ``ensure_map``
overwrites non-CommentedMap placeholders, so the next bake replaces a
stringified ``credentials.recaptcha`` blob with the correct nested
structure. Operators with corrupted entries simply re-run the
credentials generator; no manual cleanup needed.
Package refactor (KISS/DRY/SPOT):
The 450+ LoC ``__main__.py`` was carrying four orthogonal concerns
(vault, --set parsing, recursive emission, prompts) plus orchestration.
Split by concern:
- ``credentials/vault.py``: ``is_vault_encrypted``,
``is_ruamel_vault``, ``to_vault_block`` (+ private helpers
``_vault_body``, ``_make_vault_scalar_from_text``).
- ``credentials/overrides.py``: ``parse_overrides``, ``override_for``.
- ``credentials/emit.py``: ``ensure_map``, ``emit_credentials``.
- ``credentials/prompts.py``: ``ask_for_confirmation``.
- ``credentials/__main__.py``: argparse + ``main`` + two thin mode
handlers (``_run_snippet_mode``, ``_run_default_mode``) + the
force-override helper.
``credentials/__init__.py`` re-exports the public API
(``main``, ``ask_for_confirmation``, ``parse_overrides``, etc.) so
external callers (notably ``tests/unit/cli/administration/inventory/
test_credentials.py``) keep working unchanged.
Test layout follows the split:
- ``tests/unit/cli/administration/inventory/credentials/test_vault.py``
(20 tests: vault detection, body extraction, scalar construction,
to_vault_block paths, plus the VaultHandler subprocess coverage).
- ``tests/unit/cli/administration/inventory/credentials/test_overrides.py``
(13 tests: parse_overrides + override_for incl. dotted nested keys).
- ``tests/unit/cli/administration/inventory/credentials/test_emit.py``
(10 tests: ensure_map placeholder replacement, recursive emission,
none → empty, skip_existing semantics, override precedence,
dotted-path tracking, the on-disk corruption recovery case).
- ``tests/unit/cli/administration/inventory/credentials/test_prompts.py``
(5 tests: ask_for_confirmation yes/no/garbage/case/whitespace).
``tests/unit/cli/administration/inventory/test_credentials.py`` keeps
only the end-to-end ``main()`` integration coverage (including
``test_nested_credentials_dict_recurses_instead_of_stringifying`` —
the explicit regression guard for the original failure mode).
Coverage: 47 new per-module unit tests + the 5 existing main()
integration tests, all green. ``make test`` green across all five
targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CI job 77797371504 on run 26428080957 crashed with:
tls lookup: application_id '{{ lookup('env', 'INFINITO_DOMAIN') }}'
not found in domains mapping
`DOMAIN_PRIMARY` is defined in inventories/development/default.yml:20
as `"{{ lookup('env', 'INFINITO_DOMAIN') }}"`. On Ansible 2.18+ that
string carries a `TrustedAsTemplate` tag (it is an
`_AnsibleTaggedStr`). The constructor at tasks/stages/01_constructor.yml:64
wrapped it in `[ DOMAIN_PRIMARY ] | flatten | map('string') | first`
— the explicit `str()` cast strips the trust tag. The resulting plain
string flowed into `CURRENT_PLAY_REDIRECT_DOMAINS.source` as the
literal text `{{ lookup('env', 'INFINITO_DOMAIN') }}`, was passed
through `domain: "{{ item.source }}"` in
roles/web-opt-rdr-domains/tasks/main.yml:14 (one render pass that
produces untrusted content), and finally reached `as_str(terms[0])`
in plugins/lookup/tls.py:39 unchanged. The tls plugin never templated
its arg, so `resolve_term` looked up the literal Jinja string in the
domains mapping and raised.
Two layers, both necessary:
1. plugins/lookup/tls.py:39 — defensively template `terms[0]` through
`self._templar` before string-coercion. Mirrors the idiom in
plugins/lookup/unit_name.py:33 and plugins/lookup/email.py:61. Any
future caller that hands the plugin an unresolved expression now
gets it resolved before the domains-map lookup.
2. tasks/stages/01_constructor.yml:64-65 — drop the
`[ X ] | flatten | map('string') | first` ritual for
`DOMAIN_PRIMARY` and `DOMAIN_HOMEPAGE`. Both are scalar strings in
every known inventory (group_vars/all/00_general.yml:41,43 +
inventories/development/default.yml:20); the over-defensive
wrapper was the trust-tag stripper. Direct reference preserves the
tag through `CURRENT_PLAY_REDIRECT_DOMAINS` and every downstream
consumer.
Downstream consumers audited; all transparent to `_AnsibleTaggedStr`
(it inherits from `str`):
- merge_mapping filter: content-based dict-key compare
- domain_mappings filter: takes `DOMAIN_PRIMARY` directly, no cast
- items2dict(key_name='source', value_name='source'): hashable
- generate_all_domains filter: `isinstance(item, str)` matches
- sys-stk-front-base `when: ... | length > 0`: only counts
- web-opt-rdr-domains -> tls lookup (the failing path): now templates
Coverage: new `test_term_with_unresolved_jinja_is_templated` in
tests/unit/plugins/lookup/test_tls.py verifies the lookup feeds its
first term through the templar. The 12 existing tls tests stay
green. `make test` green across all 5 targets.
Class follow-up (not addressed here): the
`| string` / `| map('string')` cast still strips `TrustedAsTemplate`
elsewhere if a future commit reintroduces it; 12 other lookup
plugins do not yet defensively template their args. A lint guard and
plugin-wide audit will come in separate PRs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pruned per project convention "no unnecessary code comments":
- module docstring: 23 lines → 1 line (the file name + symbol already
state intent; full incident write-up belongs in the commit log)
- wipe_database_entries docstring: 11 lines → 2 lines
- six inline rationale comments in tests/__main__ removed (test
names already encode the intent)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
After 10cb02461 switched `payload._resolve_variant_payloads` from
`get_variants` to `get_variant_overrides_only`, matrix-deploy bakes
only the raw `meta/variants.yml` slice into host_vars (typically just
`services:`). Lookups that read `variables['applications']` raw no
longer see content from `meta/rbac.yml`, `meta/server.yml`, etc.
`rbac_group_path` was the last lookup still going through the raw
host_vars read. For joomla (whose `meta/rbac.yml` declares the
`editor` role) the post-10cb02461 host_vars carried no `rbac` block
at all, the lookup walked an empty `rbac.roles` map, and the deploy
crashed with "rbac_group_path: role 'editor' is not declared under
applications[web-app-joomla].rbac.roles. Declared roles:
['administrator']" (job 77797371433 on run 26428080957).
Route `_get_application` through `get_merged_applications` so the
meta-merged view (services + server + rbac + volumes + schema
credentials + variant overrides) is restored. This mirrors the
existing pattern in plugins/lookup/{config,tls,nginx,domain}.py and
keeps the mirror-override intent of 10cb02461 intact: the bake step
still emits only the variant overrides; the merged cache assembles
the full view at lookup time.
Test updates:
- TestRbacGroupPathLookup.setUp patches get_merged_applications to
read from variables['applications'] so the existing 14 tests stay
hermetic without disk-reading roles/*/meta/*.
- New TestRbacGroupPathReadsMergedApplications regression case
exercises the exact 10cb02461 scenario: host_vars only carry a
variants.yml slice, the merged cache supplies the rbac block, the
lookup returns the correct path.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Matrix-deploy variant transitions wipe a consumer's compose stack and
its named DB volume (`compose down -v` in
scripts/container/purge/entity/compose.sh) but `databases.csv` at
$FILE_DATABASE_SECRETS survives the purge. When the next round drops
the consumer from its inventory, the stale baudolo-seed row remains
behind. The subsequent backup run reads that row, attempts
`mariadb-dump -u<entity> -p<old_pw>` against the freshly initialised
RDBMS user that no longer exists, and the systemd unit fails with
"Access denied for user 'matomo'@'127.0.0.1' (using password: YES)"
(job 77797356000 on run 26428080957: matomo dump in
svc-bkp-container-2-local variant 1, where round 1 seeded matomo but
round 2 only deploys postgres + mariadb + svc-bkp itself).
Add `utils.cleanup.databases_csv` mirroring the existing
`utils.cleanup.tokens` pattern: parse the baudolo-seed CSV format
(';'-separated, four columns instance/database/username/password, with
header — verified against baudolo.seed.__main__:check_and_add_entry)
and strip every row whose `database` or `username` column matches an
entity derived from the supplied app ids via
utils.roles.entity_name.get_entity_name. Header and blank lines are
preserved verbatim; the file is rewritten only when at least one row
was removed.
Wire the helper into scripts/container/purge/apps.sh right after the
token-store wipe so every matrix-deploy inter-round purge invokes it
alongside the other cleanups. Operator-triggered purges
(`INFINITO_APPS=<app> bash scripts/container/purge/apps.sh`) also get
the cleanup — that is a behaviour change relative to before, where
the row would survive an operator-triggered entity teardown.
Coverage: 12 unit tests in tests/unit/utils/cleanup/test_databases_csv.py
covering the happy path, the multi-app case, the
`database='*'`/username-match case, the unknown-app no-op, header and
blank-line preservation, the all-rows-removed case, malformed short
rows, and the legacy-header-less file. `make test` green across all 5
targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- web-opt-rdr-www canonical: rdr-www.{DOMAIN_PRIMARY} -> w3redirect.{DOMAIN_PRIMARY}.
- New lookup `www_redirect_domains` (role-scoped): returns full {source, target}
mapping list, resolves DOMAIN_HOMEPAGE target for the role's own canonical via
lookup('domain', 'web-opt-rdr-www'). REDIRECT_WWW_DOMAINS now carries dicts;
tasks/main.yml and _01_cloudflare_edge_redirect.yml updated to consume dicts.
- New lookup `current_play_redirect_domains`: replaces the two inline
set_fact merges in tasks/stages/01_constructor.yml. Merges user-supplied
redirect_domain_mappings, primary-redirect (conditional on web-opt-rdr-domains
in deployed), and per-app domain mappings (inlined from the now-removed
domain_mappings filter).
- domain_redirect_mappings filter removed; web-app-keycloak RedirectUrisTest
pollution of utils.roles.applications.config in sys.modules fixed via
tearDownClass cleanup.
- csp_skip_domains lookup: csp.health.skip opt-out path removed (architecture
no longer needs it).
- Lint test_only_web_app_roles_integrate_dashboard expanded from web-app-* to
web-*; test_role_and_group_vars_used now scans .py files too so vars consumed
only inside plugins (DOMAIN_HOMEPAGE, AUTO_BUILD_ALIASES) are seen.
- New role-scoped variants.yml on web-opt-rdr-www with dashboard on/off.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Since 576fad84f (2026-05-10) moved role images from
`defaults/main.yml` into `meta/services.yml`, matrix-deploy's variant
baker has been writing the FULL effective config (services.yml +
variants.yml deep-merged via `get_variants`) into host_vars through
the `--vars` JSON consumed by `apply_vars_overrides`. That populates
`applications.{role}.services.{svc}.image` (and `.version`, .cpus,
etc.) with the upstream value, which then silently disables
`apply_mirror_overrides` for every role with an explicit image
declaration: the default `if_missing` policy short-circuits when the
field is already non-blank, so the rendered compose / Dockerfile FROM
keeps pulling from the upstream registry rather than the GHCR mirror.
The regression went unnoticed because Docker Hub usually serves the
pull, but it eventually surfaced as a BuildKit 429 on
`docker.io/library/php:8.2-apache` during the suitecrm Dockerfile
build (job 77602110531 on run 26362676803). Nextcloud, moodle,
mariadb consumers, postgres consumers, simpleicons, phpmyadmin,
phpldapadmin, jenkins, baserow, fider, ... all share the same latent
defect — only suitecrm happened to hit the rate limit first.
Fix the root cause instead of papering over per-role with
`mirror_policy: force`: introduce `get_variant_overrides_only` next
to `get_variants` and switch payload.py to it. The new function
returns the raw `meta/variants.yml` entries per role — no merge with
`meta/services.yml`. matrix-deploy now bakes only the variant-
specific overrides into host_vars; the role's services.yml fields
flow through the normal Ansible lookup chain at deploy time, leaving
`image`/`version` blank in host_vars so `apply_mirror_overrides`
populates them from `mirrors.yml` under its default `if_missing`
policy.
Behavior preserved for the other `get_variants` consumers:
* utils/manager/inventory.py:104 (schema-driven credentials) keeps
the full merged variant payload.
* planner.py:58 (variant-count only) is unaffected.
* legacy_resolver.py:42 (CombinedResolver runtime services_overrides)
keeps the full merged view.
Coverage:
* 3 new tests in test_applications.py for the new helper (single
empty variant when no meta/variants.yml, raw override without base
merge, cache invariance).
* test_payload.py + test_builder.py mocks repointed from `get_variants`
to `get_variant_overrides_only`.
* `make test` green across all 5 targets (lint, unit, integration,
external, test-lint) — 188 tests total.
Side-effects assessed:
* Repo-wide grep confirms no template / task accesses
`applications.X.services.Y.image` directly via hostvars — every
consumer goes through `lookup('config', …)` which still deep-merges.
* Roles previously pulling upstream now resolve via the GHCR mirror
whenever the image is already synced (the `images-mirror-missing.yml`
workflow keeps the destination populated).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`_apply_one_role_special_rules()` in utils/manager/inventory.py used
`bool(resolve_database_service_key(...))` to decide whether to
auto-generate a `database_password` credential for a role. The
resolver raises when multiple RDBMS keys are enabled — which is the
correct contract for compose_volumes / container_depends_on /
plugins.lookup.database (they need an unambiguous single key) but the
wrong contract for inventory's credential-generation pass: variant 1
of svc-bkp-container-2-local intentionally enables both mariadb and
postgres to exercise the engine-side dump path against both engines
at once, and the deploy aborted with `ValueError: multiple direct
database services are enabled: mariadb, postgres` during inventory
generation (job 77602095377 on run 26362676803).
Add a non-raising sibling helper `has_single_database_service()` that
returns True iff exactly one RDBMS service is enabled, and switch
inventory's call site to it. Multi-DB roles now skip the auto-
generated `database_password` (which was a phantom credential for
svc-bkp anyway — no schema.yml, no cross-role lookup; the role's own
`04_seed-database-to-backup.yml` receives `database_password` as a
caller-passed task var, not as a self-credential lookup).
`resolve_database_service_key` keeps its strict raise semantics for
the compose_volumes / container_depends_on / database-lookup callers,
which still need a single unambiguous key.
Add unit coverage for both functions (3 + 4 tests) in
tests/unit/utils/roles/applications/services/test_database.py.
Static check only (per operator request).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`generate_all_domains(include_www=True)` synthesizes `www.X` for every
canonical domain X and pushes them into CURRENT_PLAY_DOMAINS_ALL, but
they are never registered in the global canonical_domains_map or in
applications[*].server.domains. The strict `tls`/`cert` lookups
therefore raised `domain 'www.X' not found (domains/applications)`
when web-opt-rdr-www iterated REDIRECT_WWW_DOMAINS and called
`lookup('tls', 'www.X', 'enabled')` in the redirect-domain.yml TLS
gate — failing job 77602110568 on run 26362676803 (and cascading
through sys-front-tls/vars/main.yml, sys-front-tls/tasks/02_http.yml
and sys-front-tls-selfsigned/tasks/01_core.yml, all of which call
`lookup('tls', application_id, …)` with the same leaked www.X).
Add a single fallback branch to `resolve_term`: if the strict lookup
fails AND the term starts with `www.`, retry both resolver paths
(legacy reverse + applications) against the bare domain. The fix is
strictly additive — every previously successful term and every non-www
miss preserves its current behavior.
Coverage:
* test_www_prefix_falls_back_to_bare_domain
* test_www_prefix_with_unknown_bare_still_raises
* test_non_www_unregistered_still_raises (regression guard)
Verified statically via the in-container `resolve_term`:
www.infinito.example -> (svc-prx-openresty, infinito.example)
www.rdr-www.infinito.example -> (web-opt-rdr-www, rdr-www.infinito.example)
The global self-signed cert SAN list (cert.py:193) is built from
CURRENT_PLAY_DOMAINS_ALL, so it already covers the www.X variants —
nginx serves the redirect vhost with that cert and the TLS handshake
on the hlth-webserver check matches.
Static check only (per operator request).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Flatten records from all summaries and sort by ISO-Z timestamp so the
step-summary table is chronological across apps/variants/passes instead
of grouped per file.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`_build_application_defaults` returned variant 0 of every role with a
meta/variants.yml, which then deep-merged with the variant-N override
that build_dev_inventory bakes into the inventory. Any service flag
that variant N did not explicitly disable carried through from
variant 0 into the runtime applications map. The most visible casualty
was svc-bkp-container-2-local variant 1 (postgres + mariadb only),
where gitea + keycloak leaked in via variant 0, sys-service-loader
pulled in svc-db-openldap via keycloak.services.ldap, and the deploy
crashed in sys-svc-compose/04_files.yml on openldap's env.j2 with
"credentials is defined in schema at roles/svc-db-openldap/meta/schema.yml
but not set in application 'svc-db-openldap'".
Defaults now come from each role's variant-free base config
(meta/<topic>.yml plus schema-defaults, no variants overlay). Variants
stay accessible via get_variants for any caller that explicitly needs
a specific variant. Inventory-level applications.<app> overrides win
exactly as before.
Supporting changes that keep the variant matrix consistent under the
new defaults model:
- New lint guard tests/integration/roles/meta/variants/
test_non_baseline_explicit_disables.py: for every dynamic
service-key declared in a role's meta/services.yml that variant 0
pins to literal true, every non-baseline variant MUST explicitly
pin the same key to either literal true or to literal
enabled: false + shared: false. Statically-enabled keys are exempt
(same rule as test_auth_isolation.py).
- roles/web-app-{fusiondirectory,lam,nextcloud,phpldapadmin}/meta/
variants.yml: explicit `false` pins for dashboard / email / logout /
matomo / prometheus / talk in restrictive variants so deep-merge no
longer hands them the deleted variant-0 inheritance.
- roles/web-app-moodle/templates/playwright.env.j2: gate the LAM
block on `lookup('config', application_id, 'services.lam.enabled',
false)`. LAM is a transitive helper, not a moodle-owned service, so
the default-bearing lookup tolerates the missing path when the
variant does not deploy LAM.
- tests/unit/utils/cache/test_variants.py: rename the legacy
"always variant zero" coverage to TestMergedApplicationsUsesBaseConfig
and add test_get_application_defaults_omits_variant_zero_only_keys
as the regression guard for the leak this commit fixes.
- docs/contributing/artefact/files/role/variants.md +
docs/contributing/design/variants.md: variant 0 now manually
enumerates every dynamic services.yml key with
enabled: true + shared: true; `{}` is only permitted when the role
has no dynamic-enabled service-key.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
InventoryManager.resolve_schema_includes_recursive previously read the
root role's base meta/services.yml when discovering shared providers,
regardless of which matrix variant the planner had pinned for the
round. For a role whose variants.yml enables additional services per
variant (e.g. svc-bkp-container-2-local variant 2 enabling
services.ldap), the base config had no `enabled+shared` flag for those
services, so the recursive resolver never visited the variant's
shared providers — their credentials were never generated, and the
deploy crashed in the provider's env.j2 with "credentials is defined
in schema at roles/<provider>/meta/schema.yml but not set in
application '<provider>'".
Wires the active variant through the existing inventory pipeline:
- utils/manager/inventory.py: InventoryManager(variant=None|int). When
set, load_role_config_by_path overlays variants.yml[N] onto the root
role's meta config (only for the root — transitive providers keep
their own base lookup). variant=None preserves the pre-refactor base
behavior for every standalone caller.
- cli/administration/inventory/credentials: new --variant N flag,
forwarded to InventoryManager.
- cli/administration/inventory/provision: new --app-variants <JSON>
flag ({app_id: variant_index}), forwarded per-app to the credentials
subprocess.
- cli/administration/deploy/development/inventory/builder.py: emits
--app-variants from spec.variant_selectors() when the round pins any
variant. Apps without variants.yml collapse to a single empty
overlay in get_variants so this is a no-op for them.
10 unit tests cover: variant overlay for root role, variant=None
fallback to base, out-of-range fallback, no overlay on non-root
paths, --variant CLI forwarding (set + default), --app-variants
generator forwarding (set + omit), builder --app-variants emission
(set + omit).
Caveat: bkp-c2l CI variant 1 still loads svc-db-openldap transitively
at runtime (outside the planner's variant closure) — a separate
sys-service-loader / role-meta bug, not addressed here.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
keepass.info responds with 415 Unsupported Media Type to the probe's
GET; the previous 25a77bc7e widening landed 418/429/451 but missed 415.
Treat 415 like 401/403/405 (server alive, request format rejected).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Commit f63118f34 ("feat(lint): enforce auth-isolation in
variants.yml") added `enabled: false, shared: false` overrides for
mariadb / postgres to every auth-isolated LDAP-only variant. That
disables the role's hard-required DB at deploy time, so e.g. the
moodle baseline blows up with `Required env var MOODLE_DB_HOST is
not set` as soon as round 2 of the matrix hits variant 1.
The auth-isolation rule's premise — "auth-isolated rounds must trim
non-auth services from the include closure" — only buys closure
savings for services whose `meta/services.yml` base declares
`enabled` as a dynamic Jinja expression. DB-engine entries are
already a static `enabled: true`, so the resolver pulls them in
regardless of the variant pin; the variant `false` only breaks the
deploy without trimming anything.
Two changes:
- `test_auth_isolation`: exempt service-keys whose `services.yml`
base has a static literal `enabled: true` from the per-variant
pin obligation. Such keys should not be declared in auth-isolated
variants at all; the inherited base value is correct.
- 10 web-app roles in the active deploy scope (akaunting, baserow,
bookwyrm, espocrm, friendica, gitea, moodle, nextcloud, odoo,
openproject): drop the entire mariadb/postgres stanza from the
LDAP-only variant — both `enabled` and `shared` inherit from
`meta/services.yml`. Matches the pre-f63118f34 state.
- 6 more roles out of the current iteration scope (funkwhale,
opentalk, shopware, snipe-it, suitecrm, taiga) carry the same
broken `enabled: false` line: removed it, kept the pre-existing
`shared: false` to preserve the variant's own-DB-container
topology intent.
`make test` is green; `variants-auth-isolation` and
`variants-static-override` both pass with no per-variant nochecks
required.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The compose templates appended additional depends_on entries (e.g. taiga
rabbitmq services, openproject cache/seeder, mailu resolver/front) on
column 6 directly after the lookup call. When the lookup returned ""
(no DB and no redis), those entries dangled without a depends_on header
and broke YAML parsing.
Move every such trailing block into the lookup's `extra` kwarg so the
header is always emitted as soon as any entry exists. Adds a regression
test for the extras-only case.
Affected: taiga, mailu, magento, pixelfed, openproject, espocrm.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Eliminate include-style depends_on assembly. The lookup resolves DB/redis
inductively from applications config and returns indented YAML (or "" when
no entries), removing the empty `depends_on:` block bug in libretranslate.
Migrates 43 compose templates from `{% include … dmbs_*.yml.j2 %}` to
`{{ lookup('container_depends_on', application_id) }}`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`status` was too vague for what the lookup returns (apps in current
play). Rename to `deployment` and add two new fields:
* `runtime` — RUNTIME variable value
* `deployed` — runtime-aware authoritative set: equals `running`
(whitelist or groups) for ephemeral CI runtimes (act, github),
falls back to `groups` for persistent runtimes (host, dev). This
matches reality: in CI the container is fresh so only whitelisted
apps exist; in production a partial deploy still leaves prior
apps live, so groups is the right set.
Switch the two webserver/csp healthchecks to `lookup('deployment').deployed`
so they only probe domains for apps that actually exist in the current
deploy. In production behaviour is unchanged.
Update remaining consumers (cli/build/include, web-app-keycloak,
test-e2e-playwright) for the rename. Existing `status` semantics
(`whitelist`, `running`, `groups`, `all`) preserved under the new
name.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cluster 2a (hlth-msmtp failed=65 in universal single-app deploys):
the health-check script sent "To: administrator" but the administrator
Unix user is only provisioned in server-pass, so postfix rejected the
recipient. Replace the inline Jinja conditional with a role-local
lookup plugin that picks administrator.email (external), "administrator"
(local + user exists) or "root" (local + missing).
Side change: SIM105 cleanup in utils/install/lint/__main__.py.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Fixes Cluster 3a ca-inject YAML parse: when a role variant disables its
DB (enabled=false), the depends_on template emitted ` :` because
the existing `not shared` guard fired for a non-existing DB host.
The new derived flag mirrors `service.required` (= enabled AND shared)
and is consumed in:
- depends_on/dbms_base.yml.j2, dmbs_excl.yml.j2 (the actual fix)
- sys-svc-rdbms postgres/mariadb service templates
- web-app-discourse, web-app-snipe-it (latent same-pattern defensives)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements docs/requirements/021-sso-flavor-migration.md atomically:
* Schema: every role's meta/services.yml replaces the parallel
`oauth2:` / `oidc:` top-level blocks with a single `sso:` block
carrying `enabled`, `shared`, and a `flavor` discriminator
(`oidc` | `oauth2` | `saml`; default `oidc`). Flavor-specific
sub-keys live under `sso.<flavor>.*` (`sso.oauth2.{origin,acl,
allowed_groups}`, `sso.oidc.plugin`).
* Provider role: roles/web-app-oauth2-proxy/ is deleted in its
entirety; its 5 sidecar templates move under
roles/web-app-keycloak/templates/sso_proxy/, and the per-consumer
rendering task lives at roles/web-app-keycloak/tasks/sso_proxy.yml.
web-app-keycloak now declares `provides: sso`.
* Runtime: new resolver utils/roles/applications/services/sso.py
centralises the SSO predicates (`is_enabled`, `is_proxy_gated`,
`is_oidc_native`) and the oauth2 sub-values
(`oauth2_origin_{host,port}`, `oauth2_acl`, `oauth2_allowed_groups`).
Three lookup plugins consume the resolver: `sso` (single-app
predicate access), `sso_proxy_consumers` (enumerates oauth2-flavored
consumers), `sso_oidc_plugin` (renamed from `oidc_flavor`, Nextcloud
plugin selector). sys-stk-backend, sys-svc-compose, sys-svc-proxy
and plugins/filter/compose_volumes.py all gate on
`lookup('sso', ..., 'is_proxy_gated')` instead of manually combining
`enabled` and `flavor`.
* Identifier renames repo-wide: `services.<entity>.ports.local.oauth2`
→ `ports.local.sso`; `PORT_BANDS.local.oauth2` → `PORT_BANDS.local.sso`;
`OAUTH2_PROXY_*` env/Jinja vars → `SSO_PROXY_*`;
`oauth2_proxy_cookie_secret` → `sso_proxy_cookie_secret`;
`OAUTH2_SERVICE_ENABLED` + `OIDC_SERVICE_ENABLED` collapse into
`SSO_SERVICE_ENABLED`; Playwright gating helpers
(isServiceEnabled/skipUnlessServiceEnabled/requireService/safeIsEnabled/
safeSkipUnlessEnabled) move from `oidc`/`oauth2` to `sso`.
* Dual-block resolution: per req's Background-Table, web-app-bookwyrm
is `flavor: oauth2`; web-app-gitea and web-app-friendica keep
`flavor: oauth2` with their existing `oauth2.acl.blacklist`
(/user/login) and `oauth2.acl.whitelist` (federation + discovery
endpoints) — preserving the operative oauth2-proxy gating.
* Tests: the two mutual-exclusion guards
(tests/integration/iam/oauth2_oidc/test_mutual_exclusive.py and
test_acl_mutual_exclusive.py) are deleted per Decision #10;
test_oauth2_contract.py is renamed and rewritten to
test_sso_contract.py, asserting that `flavor: oauth2` roles carry
`oauth2.origin.{host,port}` plus `ports.local.sso`; test_sso.py
(web-app integration) now requires the unified `sso:` block with an
explicit flavor or a `# nocheck: sso` opt-out; new
tests/lint/repository/no_legacy_sso_paths/ guards the source tree
against re-introducing the legacy strings.
* Docs: req 21 itself has every AC checked off, and reqs 006, 008,
009, 013, 015, 016, 019, 020 plus 10+ role READMEs and
docs/contributing/actions/testing/suppression.md are updated to the
unified SSO naming.
Empirically validated end-to-end against the validation app set
(web-app-keycloak, web-app-nextcloud, web-app-bookwyrm, web-app-gitea,
web-app-friendica, web-app-dashboard) in matrix variants 0 and 1; v2
also passes the 4 target roles' Playwright after a documented
pre-existing KC_HOSTNAME-drift workaround (separate concern, not
caused by this refactor).
BREAKING CHANGE: every consumer of `services.oauth2.*`,
`services.oidc.*`, `OAUTH2_PROXY_*`, `OAUTH2_SERVICE_ENABLED`,
`OIDC_SERVICE_ENABLED`, `oauth2_proxy_cookie_secret`,
`oauth2_proxy_application_id`, `ports.local.oauth2`, the
`web-app-oauth2-proxy` role, or the legacy gating helper service names
(`oidc`/`oauth2`) must migrate to the unified `sso` shape documented
in docs/requirements/021-sso-flavor-migration.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Renames the shared-services loader role from sys-utils-service-loader to
sys-service-loader (shorter, no orphaned "utils" prefix; entity name in
meta/services.yml realigns to service-loader per get_entity_name).
Introduces an explicit two-phase contract in the loader: after the
ordered preload pass, flush a `_service_loader_post_run` queue that
service roles register into during preload via the new helper
`list_or_shoot.yml` (list during preload, shoot directly outside it).
Replaces the `when: not _service_loader_preloading` guards in
sys-svc-compose and sys-svc-webserver-core with helper calls. Closes
the gap where non-web deploys would never load these post-load
dependencies; in normal web deploys behavior stays the same thanks to
each sub-role's `run_once_<role>` gate.
Docs, comments, test docstring, and constructor-stage invocation
follow the rename. Historical changelogs (CHANGELOG.md, packaging/*)
intentionally left at the original name.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
