The web-origins scope mapper listed every registered origin in each access token. openresty forwards that token to SSO-gated upstreams, so on a Tor deployment the claim reached 5457 of the token's 8650 characters and LAM's Apache refused the request with 400 (Moodle#0, run 33689701895).
The realm template carries the change, so a fresh import and every CI round get it directly. The reconcile task exists because 04_realm_import.yml only imports when the realm is missing, which would leave the old mapper in place on running installations.
Keycloak resolves CORS from client.webOrigins in DefaultCors.checkAllowedOrigins whenever the token carries no origins, so enforcement is unchanged; that fallback exists since 26.7 and the role pins 26.7.3. Introspection keeps the claim, and nothing in this repository reads it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The realm carried exactly one client, the shared OIDC one, and no SAML client
existed anywhere in the repo - `flavor: saml` was in the resolver's enum with
zero consumers. Apps that cannot speak OIDC therefore had no way to
authenticate against Keycloak at all; they could only be fronted by the
oauth2-proxy, which authenticates at the edge and leaves the app itself without
a session.
KEYCLOAK_SAML_APPS collects every deployed app whose services.sso.flavor is
saml. It filters stepwise - value.services, then .sso, then .flavor, each
guarded by `defined` - because Ansible's undefined is a Marker(StrictUndefined,
Tripwire): resolving that chain in one selectattr raises for every app without
an sso block instead of yielding undefined, which would have broken the realm
render for every deployment in the repo.
The include is unconditional and the loop emits a leading comma per client, so
the realm needs no branch on the list being empty. That matters: an Ansible
`>-` value that failed to convert would be a non-empty string and a truthiness
guard would then splice a broken clients array.
The client's shape is not guesswork - every attribute comes from a failure
reproduced against a live Keycloak 26.7.1 until a SuiteCRM SAML login completed:
defaultClientScopes is emptied because the built-in role_list scope emits one
Role attribute per realm role and the SP rejects the assertion with "Found an
Attribute element with duplicated Name"; saml.client.signature is false because
the SP does not sign its AuthnRequests; the username mapper's attribute name
comes from OIDC.ATTRIBUTES.USERNAME rather than a hardcoded claim.
The new test renders the complete realm - every include, the role's own
ldap_groups_filter, StrictUndefined - and parses it as JSON for zero, one and
several SAML apps. The file had no test before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The provisioned become password doubles as the Keycloak administrator realm password (user-administrator/meta/users.yml) and the guide flow generated it 64-char alphanumeric, so kcadm set-password failed the realm passwordPolicy specialChars(1) on every fresh single-app deploy. Delegate generate_random_password to ValueGenerator.generate_strong_password, the same class-guaranteeing generator every other realm user already gets via the strong_password filter; the realm policy stays strict. Also drop the commented-out Trusted Hosts block from the realm template.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces inline `applications`/`users` merging with dedicated lookup
plugins and runtime hydration. Introduces `add_csp_hash` filter,
integrates sort into `find_dock_val_by_bkp_entr`, splits the Keycloak
one-shot realm import into `05a_realm_import_run.yml`, and fixes the
Mastodon administrator task YAML. Removes obsolete `csp_hashes` filter,
`sys-token-store` Jinja filter, and empty CLI setup modules.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace usages of the get_app_conf filter with the config lookup plugin
across roles and templates. Adjust quoting where required to keep YAML
and Jinja syntax valid. No functional changes intended.
https://chatgpt.com/share/69860678-cef8-800f-8402-7fe6231fdc2d
The get_app_conf() helper no longer requires an explicit third
parameter to enable application-specific lookups. All templates,
tasks, and vars have been updated to rely on the new default behavior
by removing the redundant `True` argument.
This simplifies usage, improves readability, and enforces a consistent
API across Jinja2 templates and Ansible logic.
https://chatgpt.com/share/6985f5f6-c084-800f-887a-4a70d62e6dcf
The TLS lookup has been renamed from `tls_resolve` to `tls` to better
reflect its responsibility as a strict TLS state resolver.
This refactor updates all usages across the codebase, including:
- lookup plugins and internal helpers
- roles, vars, tasks, and templates
- tests, stubs, and error messages
- documentation and changelog references
Behavior, lookup API, and semantics remain unchanged.
https://chatgpt.com/share/6985e635-5a94-800f-b794-937e566ee308
Migrate all lookup() usages from the deprecated want= keyword argument to the
new strict positional want-path API. Update call order where required
(e.g. nginx_paths) and ensure consistency across vars, tasks, and templates.
https://chatgpt.com/share/6985d642-b40c-800f-9cdf-ed54fc484a92
- move realm import toggle from actions.import_realm to docker.services.keycloak.realm.import_enabled
- add docker.services.keycloak.realm.totp_enabled flag
- conditionally include CONFIGURE_TOTP for users and realm required actions
- prepare realm import for optional OTP enforcement
https://chatgpt.com/share/6978cf00-196c-800f-abd6-5d0a11f7fcf8
- Remove deprecated filter_plugins/tls.py and all dependent filters
- Introduce strict lookup plugin lookup_plugins/tls_resolve.py as single TLS authority
- Migrate all roles, templates, vars, and tasks to use tls_resolve (protocols, ports, URLs, cert paths)
- Enforce explicit TLS context (domains, applications, TLS_ENABLED, TLS_MODE)
- Support letsencrypt and self_signed with SAN handling and per-app overrides
- Update TLS provider roles to assert resolver state instead of setting facts
- Add comprehensive unit tests for tls_resolve lookup
https://chatgpt.com/share/6972b49b-b074-800f-a323-dd9fe69d02eb
with explicit per-application TLS resolution.
Key changes:
- Introduce sys-front-tls-layer as the single TLS abstraction
- Centralize certificate paths via TLS_CERT_FILE / TLS_KEY_FILE / TLS_CA_FILE
- Replace sys-svc-letsencrypt SSL templates with generic TLS layer includes
- Add tls_* Jinja filters (enabled, mode, protocol, port, websocket)
- Resolve TLS mode per application (letsencrypt | self_signed | off)
- Remove implicit defaults and WEB_* globals to fail fast on misconfiguration
- Support self-signed certificates with SAN awareness
- Make TLS behavior explicit and consistent across all roles
This decouples application logic from certificate providers and
allows switching TLS flavor without touching app roles.
https://chatgpt.com/share/69727c61-8e68-800f-a267-473a7e627978
- Introduce shared CAPTCHA settings and RECAPTCHA_ENABLED/HCAPTCHA_ENABLED flags in group_vars
- Wire reCAPTCHA/hCaptcha credentials into EspoCRM, Keycloak, Listmonk and Nextcloud
- Replace EspoCRM set_flags.php with generic seed_config.php and hook it into entrypoint/env
- Fix run_once handling in sys-ctl-cln-disc-space and minor CSS Jinja spacing issue
https://chatgpt.com/share/692a1d4f-1154-800f-a4ae-bb068aa24a53
- Fixed incorrect handling of RBAC group configuration (moved from OIDC claims into dedicated RBAC variable set).
- Unified RBAC group usage across applications (LAM, pgAdmin, phpLDAPadmin, phpMyAdmin, YOURLS).
- Replaced old 'KEYCLOAK_OIDC_RBAC_SCOPE_NAME' with dedicated 'KEYCLOAK_RBAC_GROUP_*' variables.
- Updated OAuth2 Proxy configuration to use 'RBAC.GROUP.CLAIM'.
- Refactored Keycloak role task structure:
* Renamed and reorganized task files for clarity ('_update.yml', '02_cleanup.yml', etc.).
* Introduced meta and dependency handling separation.
- Cleaned up Keycloak config defaults and recaptcha placeholders.
Replace 01_import.yml with 01_initialize.yml (KEYCLOAK_HOST_IMPORT_DIR)
Add generic 02_update.yml (kcadm updater for clients/components)
- Resolve ID → read current → merge (kc_merge_path optional)
- Preserve immutable fields; support kc_force_attrs
Update tasks/main.yml:
- Readiness via KEYCLOAK_MASTER_REALM_URL; kcadm login
- Merge LDAP component config from Realm when KEYCLOAK_LDAP_ENABLED
- Update client settings incl. frontchannel.logout.url
realm.json.j2: include ldap.json in UserStorageProvider
ldap.json.j2: use KEYCLOAK_LDAP_* vars for bindDn/credential/connectionUrl
vars/main.yml: add KEYCLOAK_* URLs/dirs and KEYCLOAK_DICTIONARY_REALM(_RAW)
docker-compose.yml.j2: mount KEYCLOAK_HOST_IMPORT_DIR
Cleanup: remove 02_update_client_redirects.yml, 03_update-ldap-bind.yml, 04_ssh_public_key.yml; drop obsolete config flag; formatting
Note: redirectUris/webOrigins ordering may still cause changed=true; consider sorting for stability in a follow-up.
- Removed all static 'id' fields from realm.json.j2, ldap.json.j2, and client.json.j2
- Replaced 'desktop-secret' with correct 'client-secret' authenticator type
- Standardized Jinja filters to use 'to_json' consistently
- Corrected defaultClientScopes entry from 'web-app-origins' to built-in 'web-origins'
- Verified LDAP mapper definitions and optional realm role mapping
- Ensured realm.json.j2 contains only required scopes
References: Chat with ChatGPT (2025-08-17)
https://chatgpt.com/share/68a1aaae-1b04-800f-aa8d-8a0ef6d33cba
- Replace KEYCLOAK_KCADM_PATH with KEYCLOAK_EXEC_KCADM consistently
- Externalize client.json to separate Jinja2 template and include it in realm.json
- Simplify LDAP bind update to use explicit KEYCLOAK_LDAP_* vars
- Add async/poll support for long-running kcadm updates
- Restructure vars/main.yml: clearer grouping (General, Docker, Server, Update, LDAP, API)
- Compute redirectUris/webOrigins centrally in vars
- Align post.logout.redirect.uris handling with playbook
Conversation: https://chatgpt.com/share/68a1a11f-f8ac-800f-bada-cdc99a4fa1bf
