Four independent bugs were each blocking the autonomous
fresh-purged-apps iteration of web-app-{openwebui,wordpress,nextcloud,
phpmyadmin,yourls}; bundling them so the pipeline keeps running.
- cli/core/discovery.py: stop walking argv prefixes at the first flag
or path-shaped arg and ignore OSError so the matrix-init's
`infinito create inventory <abs-path> --vars '<huge JSON>'` no longer
hits ENAMETOOLONG inside `is_dir()`. New regression tests pin both.
- utils/cache/data.py: when this module moved to utils/cache/data.py,
PROJECT_ROOT silently slid one level deep, pointing ROLES_DIR at a
non-existent `<repo>/utils/roles`. Implicit lookups returned empty
defaults so `lookup('users','contact')` failed at deploy time as if
the user weren't defined. Use parents[2] and pin the invariant in a
new regression test.
- roles/web-app-keycloak/vars/main.yml: KEYCLOAK_PER_APP_LDAP_ROLES_MAPPER
inherited `subComponents`/`id` from the realm-import dictionary's base
`ldap-roles` mapper. kcadm rejects those keys on `create components`
("Unrecognized field 'subComponents'"), with stderr swallowed by
`2>/dev/null` in upsert_app_mapper.sh. Strip both keys exactly like
update/05_ldap.yml already does for the shared mapper.
- scripts/tests/deploy/local/deploy/fresh-purged-app.sh: pass
`RUNTIME=dev` through `--vars`, otherwise the host process running
init falls back to `detect_runtime() == "host"` and the Playwright
E2E gate (`RUNTIME in [dev, act, github]`) never fires.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both in-process cache layers — the cached YAML loader and the cached
runtime-data getters — now live next to each other under
`utils/cache/`. The package gains a README that documents the
"orthogonal layers, no third cache" rule so future contributors hit
the right primitive on the first look.
Moves:
- `utils/yaml_cache.py` -> `utils/cache/yaml.py`
- `utils/runtime_data.py` -> `utils/cache/data.py`
- `tests/unit/utils/test_yaml_cache.py` -> `tests/unit/utils/cache/test_yaml.py`
- `tests/unit/utils/test_runtime_data.py` -> `tests/unit/utils/cache/test_data.py`
- `tests/unit/utils/test_variants.py` -> `tests/unit/utils/cache/test_variants.py`
New:
- `utils/cache/__init__.py` (empty per project convention)
- `utils/cache/README.md` describing the two layers, when to reach for
which, lifetime, test-only helpers, and design guidelines (RFC 2119
keywords, emoji-after-headline, no self-SPOT declaration, no
slash-in-link-text).
- `tests/unit/utils/cache/__init__.py`
Mechanical updates:
- All imports across cli/, plugins/lookup/, tests/, and roles/ rewritten
from `utils.yaml_cache` -> `utils.cache.yaml` and
`utils.runtime_data` -> `utils.cache.data` (40+ files).
- `REPO_ROOT` parents-index in the moved tests bumped from `parents[3]`
to `parents[4]` to reflect the deeper nesting.
- Six broken markdown links in docs/ updated to the new module paths
(test_markdown_links lint test stays green).
- Two test files that used the `from utils import runtime_data` form
now use `from utils.cache import data as runtime_data` so the local
alias remains.
Tests: 848 passed across unit + lint suites (1 pre-existing collection
error in test_host_vars.py unrelated to this change).
Phase 2 of the YAML-cache migration. The cli/meta/applications/resolution/*
and cli/build/* modules walk the same `roles/<role>/<config|vars|meta>/...yml`
files repeatedly during a single invocation. Direct `yaml.safe_load` calls
re-parsed each file every time. This commit routes those reads through the
in-process cached loader added in the previous commit so each absolute
path is parsed at most once per Python process.
utils/yaml_cache:
- New `load_yaml_any(path, *, default_if_missing=...)` for files whose
root is not a mapping (Ansible task lists, etc.). Shares the cache
with `load_yaml`, so a hot path that mixes both still pays parse
cost only once per file.
- `load_yaml` keeps its strict dict-root validation; non-mapping roots
now point callers at `load_yaml_any` in the error message.
- Internal `_load_raw` shared between both entry points.
- Tests: +5 cases covering list/scalar roots, empty-file → {} coercion,
cache sharing across the two entry points, and strict rejection.
Migrated call sites (each preserves its existing public surface and
domain-error wrapper):
- cli/meta/applications/in_group_deps/__main__.py: per-role vars/main.yml
in `find_role_dirs_by_app_id` + the one-shot applications file load.
- cli/meta/applications/role_name/__main__.py: per-role vars/main.yml
scan in `get_role`.
- cli/meta/applications/resolution/run_after/__main__.py: per-role
meta/main.yml read.
- cli/meta/applications/resolution/dependencies/__main__.py: per-role
vars/main.yml AND meta/main.yml reads inside the transitive
dependency walk.
- cli/meta/applications/resolution/services/resolver.py: per-role
config/main.yml read in `_load_role_config` (the inner-most loop of
service-dependency resolution).
- cli/build/role_include/__main__.py: per-role meta/main.yml +
vars/main.yml reads inside the topological-sort build.
- cli/build/graph/__main__.py: per-role meta/main.yml read; the
tasks/main.yml reader uses `load_yaml_any` (root is a list).
`yaml` import is preserved in two files (cli/build/graph/__main__.py and
cli/meta/applications/in_group_deps/__main__.py) because they still call
`yaml.safe_dump` on output; only the read paths moved.
Tests: full unit + lint suites green (594 passed, 1 pre-existing
unrelated collection error in test_host_vars.py).
Out of scope for this commit (one-off readers where the cache adds
nothing): cli/validate/inventory, cli/fix/vars_main_files,
cli/create/inventory/{host_vars,services_disabler,mirror_overrides},
cli/create/role, cli/meta/callorder, cli/meta/roles/lifecycle_filter.
Those each read their target file at most once per invocation, so
caching would just add an indirection without saving disk I/O.
Many CLI tools walk the same `roles/<role>/<config|vars|meta>/...yml`
files multiple times during a single invocation (dependency
resolution, service resolution, role introspection, ...). Each direct
`yaml.safe_load(path.read_text())` re-parses YAML and re-touches disk;
for `infinito create inventory` over the ~600-role tree this dominates
wall-clock time.
This change introduces `utils/yaml_cache.py`, a small in-process,
per-absolute-path memoising loader, and migrates the three existing
ad-hoc loaders to delegate to it.
`utils/yaml_cache.py` exposes:
- `load_yaml(path, *, default_if_missing=_MISSING)`: cached read,
dict only at the YAML root, raises `FileNotFoundError` by default
for missing files (callers can opt into a default value).
- `dump_yaml(path, data)`: write-through that evicts the cached entry,
so a CLI tool that edits a file mid-run sees the new content on the
next read.
- `invalidate(path)`: explicit per-path drop for the rare cases where
an external process touches the file.
- `_reset_cache_for_tests()` for unit-test isolation.
Cache semantics are documented in the module docstring; the key point
is that the same dict instance is returned on cache hit (no deep
copy), so callers that want to mutate the result MUST copy first.
This trades that constraint for ~5x fewer disk reads during the hot
resolution loops without measurably increasing memory.
Migrated wrappers (each keeps its existing public surface + domain
error type so call sites stay untouched):
- `cli/create/inventory/yaml_io.py` — `load_yaml(path)` /
`dump_yaml(path, data)`. The legacy "missing file -> {}" contract
stays via `default_if_missing={}`. Non-mapping roots still surface
as `SystemExit`.
- `cli/meta/applications/resolution/combined/yaml_utils.py` —
`load_yaml_file(path)` re-raises errors as `CombinedResolutionError`.
- `cli/meta/applications/resolution/services/yaml_utils.py` —
`load_yaml_file(path)` re-raises errors as `ServicesResolutionError`.
Tests: `tests/unit/utils/test_yaml_cache.py` covers cache hit,
path-resolution sharing, missing-file behaviour with and without the
default, non-mapping rejection, write-through eviction, and explicit
invalidation. All existing unit + lint test suites stay green.
Phase 2 (auditing the remaining direct `yaml.safe_load` call sites in
`cli/` and migrating the hot ones) is intentionally out of scope here
to keep the diff focused; this commit is the prerequisite SPOT.
Introduces per-role `meta/variants.yml` for declaring multi-variant
deploys (e.g. WordPress Single-Site vs. Multisite) and reshapes the
development deploy CLI to materialise one inventory folder per variant
and iterate them with co-located sync+async passes.
Why:
- The previous single-inventory + Pass-1-then-Pass-2 split deployed each
variant twice and ran the async pass against a host left in a
different variant's state, masking real bugs.
- Inventory and runtime concerns were tangled: variant selection lived
as a runtime extra-var and required cross-stage coordination.
How:
- `roles/<role>/meta/variants.yml` (file format docs at
docs/contributing/artefact/files/role/variants.md): YAML list of
deep-merge overrides on top of `config/main.yml`. Empty entry `{}` is
variant 0 (= legacy payload). 6 roles get their dev overrides moved
here from inventories/dev.yml; WordPress declares 2 variants.
- `utils/runtime_data.py` gains `_build_variants` / `get_variants` and
drops the retired `_active_variants` runtime selector. Loader always
returns variant 0 as the default; non-zero variants reach
`get_merged_applications` as inventory-level `applications.<app>`
overrides baked at init time.
- New `cli/deploy/development/inventory.py`: `DevInventorySpec`,
`plan_dev_inventory_matrix` (pure planner shared between init and
deploy), `build_dev_inventory_matrix` (materialises one folder per
round at `<dir>-<R>`, or `<dir>` when single-variant). Single-folder
paths preserve full backward compat.
- `cli/deploy/development/deploy.py`: matrix loop with per-round
cleanup (only apps whose variant changed AND that are about to
re-deploy), `--variant <idx>` (or `VARIANT=` env) to pin a single
round, and `--full-cycle` (or `FULL_CYCLE=true`) which interleaves
sync + async per round so the async re-deploy targets exactly the
host state the matching sync deploy just produced.
- `cli/deploy/development/exec.py` gains repeatable `--env KEY=VAL` so
bash callers can inject per-run context into in-container helpers
without inlining heredocs.
- `scripts/meta/env/inventory.sh` exports `INVENTORY_VARS_FILE`
(Python mirror in `cli.deploy.development.common.DEV_INVENTORY_VARS_FILE`;
test_inventory_vars_file_spot.py locks them against drift).
- `make deploy-fresh-purged-apps` and `scripts/tests/deploy/ci/dedicated.sh`
switch to a single init + single deploy `--full-cycle`. The
`make deploy-reuse-*` bash scripts honour `VARIANT=` by suffixing
`INVENTORY_DIR`/`INVENTORY_FILE`.
- The 5 `scripts/tests/deploy/local/deploy/*.sh` scripts now delegate
their in-container heredocs to dedicated helpers in
`scripts/tests/deploy/local/utils/`, removing the bash-in-bash
quoting hell.
Tests:
- `tests/unit/utils/test_variants.py` (renamed from
`test_application_variants.py`): loader edge cases, variant-zero
default, retired-`_active_variants` guard.
- `tests/unit/cli/deploy/development/test_inventory.py` (new): planner,
variant baking, builder, `filter_plan_to_variant`, `DevInventorySpec`.
- `tests/unit/cli/deploy/development/test_matrix_deploy.py` (new):
per-round deploy filter, cleanup decisions, `--variant` pinning,
`--full-cycle` per-round interleave.
- `tests/unit/cli/deploy/development/test_inventory_vars_file_spot.py`
(new): drift guard for the bash <-> Python literal pair.
- `tests/unit/cli/deploy/development/test_exec.py` extended for `--env`.
Docs:
- `docs/contributing/artefact/files/role/variants.md` (new): file format.
- `docs/contributing/design/matrix-deploy.md` (new): runtime mechanism
(folder layout, --variant, --full-cycle, cleanup logic).
- `docs/contributing/actions/deploy.md` (new): contributor dev-deploy
flow (make targets, --variant pinning).
- `docs/administration/deploy.md` slimmed to admin-only (was mixed
admin+dev), cross-links to the new contributor doc.
- `docs/agents/action/iteration/role.md` keeps only agent-iteration
recipes; mechanism explanation moved out and linked.
- `docs/contributing/documentation.md` adds the "no self-SPOT
declaration" rule; `applications.md` renamed its SoT header for
consistency.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add `roles/web-app-keycloak/filter_plugins/rbac_per_app.py` with two
filters that centralise the strings that vary per application:
kc_per_app_mapper_name(app_id) -> "ldap-roles-<app_id>"
kc_per_app_ldap_filter(app_id) -> "(&(objectClass=groupOfNames)(cn=<app_id>-*))"
Consume them from:
* `vars/main.yml::KEYCLOAK_PER_APP_LDAP_ROLES_MAPPER` for the mapper
name and the LDAP filter, instead of inlining the string templates.
* `tasks/update/ldap/per_app_mapper_one.yml` as the `MAPPER_NAME` env
variable handed to the upsert script.
* `files/scripts/per_app_mapper/upsert_app_mapper.sh`, which now reads
`MAPPER_NAME` from the environment instead of building it from
`APP_ID`.
A future rename of the per-app mapper convention is now a one-line
change in the filter plugin and stays consistent across vars, tasks,
and shell scripts. Unit tests cover both filters and the
`FilterModule` registration.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements requirements 005, 006, and 007.
005: per-application Keycloak `group-ldap-mapper`s anchor each app's
role groups under `/roles/<application_id>/<role_name>` (and, for
tenant-aware apps, `/roles/<application_id>/<tenant>/<role_name>`).
The flat-CN LDAP layout keeps each CN globally unique so Keycloak's
group-tree resolver never collides; per-app mappers use the LDAP
`description` as the visible Keycloak group name to keep the path
free of redundant prefixes. Migration purges legacy pre-005 entries
and forces a Keycloak `logout-all` so stale tokens drop pre-005
group paths.
006: per-service `<SERVICE>_SERVICE_ENABLED` env flags and the
`service-gating.js` helper let role specs `skipUnlessServiceEnabled`
cleanly when a shared service is missing or `SERVICES_DISABLED` is
set; `MODE_CI` markers (`INFINITO_MAKE_DEPLOY`, `GITHUB_ACTIONS`,
`ACT`) propagate through the deploy CLI into the test stage.
007: WordPress->Discourse round-trip Playwright scenario gated on the
`web-app-discourse` deployment.
WordPress Playwright runs 5/5 green on the single-site flow; the
multisite + Discourse scenarios are correctly skipped when their
service prerequisites are absent. The `rbac_group_path` lookup is the
single sanctioned producer for OIDC group paths and is wired into
every consumer (yourls, lam, minio, openwebui, pgadmin, phpldapadmin,
phpmyadmin, prometheus, xwiki).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Docker Official Image tags of the form `<semver>-<flavor>` (e.g. the
joomla:5.4.5-php8.3-apache pin) were previously skipped by the
`web-app` image version checker because its strict regex only matched
pure `x.y.z` shapes. As a result the auto-update workflow and the
external warning test never proposed a newer 5.4.x-php8.3-apache tag,
and flavored pins silently drifted out of date.
Widen the regex in `utils/docker/version_updater.py` to accept the
`<semver>-<flavor>` shape, with the flavor suffix treated as an opaque
discriminator: a candidate tag is only considered an upgrade when it
shares the *same* flavor. That guarantees a 5.4.5-php8.3-apache pin
never gets silently switched to 5.4.6-php8.4-apache or 5.4.6-fpm.
Refactor `tests/external/docker/test_image_versions.py` to import
directly from the util: ~130 lines of duplicated semver / Docker Hub /
GHCR helper code deleted, and the file now has exactly one code path
to keep in sync with the one used by the auto-update CLI
(`cli/fix/docker_image_versions`).
Chain impact:
* `.github/workflows/update.yml` -> `scripts/github/update_docker_image_versions.sh`
-> `python3 -m cli.fix.docker_image_versions` -> same util. No
workflow change needed; the next scheduled update run will pick
up flavored upgrades automatically.
* Live external run now reports 5 previously-silent upgrades:
web-app-joomla/joomla, web-app-magento/php, web-app-bridgy-fed,
web-app-suitecrm/php, web-app-shopware/redis.
Tests: existing `test_latest_semver_respects_depth` still covers the
pure-semver case; three new unit tests pin the flavor behaviour
(`is_semver` accepts flavored tags, `version_key/depth/flavor` strip
the flavor, `latest_semver` refuses cross-flavor upgrades).
Also drop an unused `autocreate_users: true` marker at the top of
`roles/web-app-joomla/config/main.yml` that was left over from an
earlier LDAP-autocreate experiment.
Auto-provision one OpenLDAP group per role declared under each application's
`rbac.roles` in `config/main.yml`, gated on `group_names` so only apps
deployed on the host contribute. Keycloak syncs the groups from LDAP, emits
the `groups` claim via the shared RBAC client scope, and WordPress maps that
claim to a local role in a mu-plugin (auto-loaded, cannot be deactivated
from the admin UI). No direct LDAP bind from WordPress.
Non-obvious design choice: the `groups` client scope MUST be a *default*
client scope, not optional. When optional + `include.in.token.scope=false`,
Keycloak 26 drops the group-membership mapper from userinfo even when the
scope is explicitly requested, so the mu-plugin receives no `groups` and
falls back to `subscriber` for every user. ID/access tokens still carry
the claim, but WP consumes userinfo.
Playwright verifies the three-role cycle (subscriber/editor/administrator)
serially and idempotently per requirement 004: the add step runs via the
Keycloak admin UI (detects "already a member" via a disabled picker
checkbox and skips); teardown runs via the Keycloak Admin REST API. Only
memberships the test itself added are removed, so any pre-existing
membership is preserved. Each hop (super-admin, biber, WP-admin) runs in
its own browser context so WP session cookies and Keycloak SSO cookies
cannot leak.
Resolves: docs/requirements/004-generic-rbac-ldap-auto-provisioning.md
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Rename the third flag returned by lookup('service', ...) from 'needed'
to 'required' across code, tests and design docs. Same semantics: True
when some deployed app has the service with both enabled: true AND
shared: true, directly or transitively through its own enabled service
dependencies.
The new name reads as a contract ("this service is required by a real
consumer") rather than a suggestion ("something needs it"), which
matches the stricter-than-enabled semantic the flag already carries.
Touchpoints:
- plugins/lookup/service.py: _is_service_needed → _is_service_required,
any_needed var, dict key in _compute_flags, docstring rationale.
- roles/sys-utils-service-loader/tasks/main.yml: caller switched to
.required.
- roles/web-app-wordpress/config/main.yml + tasks/plugins/wp-discourse.yml:
wp-discourse plugin gate uses lookup('service', 'discourse').required;
compose.services.discourse declared as shared-service dependency so
the service loader auto-loads web-app-discourse when inventory sets
enabled: true. The legacy include_tasks: utils/load_app.yml in
plugins/wp-discourse.yml is removed — the service loader now handles
it.
- tests/unit/plugins/lookup/test_service.py: 25 assertions and 6 test
method names renamed (test_needed_* → test_required_*,
test_transitive_needed_resolved,
test_direct_provider_is_not_needed_without_shared_flag).
- tests/integration/services/test_transitive_dependencies.py: one
assertion renamed.
- docs/contributing/design/services/base.md: Lookup-Plugins section
replaces the 'needed' bullet with an expanded 'required' entry that
documents when consumers should gate on it and why plain enabled is
not enough.
Also moves wordpress-app-specific keys (title, max_upload_size,
plugins) under compose.services.wordpress to match the convention
the other role keys already follow, and updates vars/main.yml lookup
paths accordingly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Second-wave rollout of tests/utils/fs:
- 6 tests switched to the shared content cache via read_text() (replacing
raw open()/Path.read_text()): roles/includes/test_existence,
roles/dependencies/test_unnecessary, vars/test_facts_not_overridden,
services/test_sys_requires_id, lint/ansible/roles/
test_service_core_first_task_run_once, lint/ansible/roles/
test_compose_resource_limits.
- 4 unit-lookup tests moved their fixture root from <repo>/.tmp_test_* to
tempfile.TemporaryDirectory() so transient files land in /tmp (matching
the AGENTS.md rule). Manual recursive-delete teardowns removed —
TemporaryDirectory.cleanup() covers that path safely.
- filters/test_usage.py: algorithmic inversion. Old loop was
O(N_filters × M_files × 4_regex); new scan walks files once, applies two
master-alternation regexes (bare `| name` and `{% filter name %}`) plus
one combined regex over all Python callables, and records hits per filter
via the capture group. The two dropped Jinja patterns were strict
subsets of the bare pattern, so the hit set is identical — only the
complexity changes (3.6s → 0.2s, ~18x).
- docs/agents/action/testing.md: new "Filesystem Scans — Use the Shared
Cache" section codifies the rule so future agent-written tests route
through tests.utils.fs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
setUp/tearDown patches `runtime_data._load_store_users` to return `{}`
so `_hydrate_users_tokens` can't pull a real `no-reply` token out of
`/var/lib/infinito/secrets/tokens.yml` (or any DIR_SECRETS / DIR_VAR_LIB
candidate that happens to exist in the test container). Without this,
`test_returns_plugin_defaults_when_no_vars` failed inside the CI
container with `password='395d6871414ffcbfbc3253db32328001'` because
`email._compute('password')` reads `users['no-reply'].tokens['web-app-mailu']`
and a prior deploy had persisted that token to disk. Patches the loader
rather than env vars because `_resolve_tokens_file` iterates four
candidate sources and any one of them existing reintroduces the leak.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
email._compute('domain') returned variables['DOMAIN_PRIMARY'] verbatim. When
DOMAIN_PRIMARY held an unrendered template (e.g. the group_vars default
"{{ lookup('env','DOMAIN') | default('infinito.localhost', true) }}"), the raw
expression leaked into rendered templates. In discourse's containers/discourse.yml
the literal "{{ ..." opened a YAML flow mapping and made Ruby Psych fail with
"did not find expected ',' or '}' while parsing a flow mapping", aborting
`./launcher rebuild discourse` in CI.
Route every _compute() result through the same _render() helper that already
covered SYSTEM_EMAIL_* overrides, and add a regression test that exercises a
templated DOMAIN_PRIMARY default.
lookup('oidc_flavor') was reading applications[APP].compose.services
straight off variables["applications"], which in Ansible is the
pre-merge override slice — role defaults from config/main.yml (like
compose.services.ldap.enabled: true) are not yet visible there. As a
result ldap.enabled resolved to the lookup's default False on every
invocation and the Nextcloud flavor silently fell through to
"sociallogin" even when LDAP was enabled.
Switch the lookup to get_merged_applications(), matching the path
lookup('config') already uses, so role-level defaults and inventory
overrides are both visible. Tests now patch that merge function rather
than inject raw variables.
Verified live against the reuse-kept stack: lookup('oidc_flavor')
resolves to 'oidc_login', plugins.oidc_login.enabled=True and
plugins.sociallogin.enabled=False.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Playwright drives Decidim's first-time OIDC registration (TOS + newsletter
modal) so the biber user is created on first login and reused idempotently
on subsequent runs. Ruby snippets previously inlined in tasks/01_setup.yml
and patch.py are now separate .rb files under files/ruby/, copied into the
image alongside patch.py. patch.py now also fixes Decidim 0.31.3's broken
omniauth_registrations_path view helper (both new.html.erb and
new_tos_fields.html.erb).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Relocates the role-local test file under `tests/unit/roles/web-app-decidim/`
so the new `test_*.py`-location lint has no offenders. Required for the
pairing commit b67b39221; got unstaged by pre-commit hook churn the
first time around.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Shorten the module name now that it backs more than just lookup plugins
(validator CLI, tests, dashboard role). Updates all 26 import sites and
adds a unit test covering the pure helpers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Moves the SYSTEM_EMAIL_* resolution logic out of group_vars/all/03_email.yml
into a dedicated Ansible lookup plugin, so every role template reads outgoing
mail settings via lookup('email') (global) or lookup('email', application_id)
(per-app). Per-app compose.services.email is merged on top of the resolved
defaults; SYSTEM_EMAIL_* variables remain usable as an inventory escape hatch.
Also splits the email docs: administrator-facing configuration stays under
docs/administration/configuration/, plugin contract and extension rules move
to docs/contributing/design/services/email.md. services.md is relocated to
services/base.md so the services/ subtree can host per-service design pages.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Roles outside the web-*/svc-db-* auto-default families (notably
svc-prx-openresty) declare an explicit `server.domains.canonical` and
must register in the global domains map so `lookup('tls', ...)` can
resolve them. Previously the prefix filter silently dropped them,
causing `application_id 'svc-prx-openresty' not found in domains
mapping` once the legacy `domains` set_fact / `CURRENT_PLAY_DOMAINS`
injection was removed.
Auto-default domain generation (entity_name.DOMAIN_PRIMARY) still
applies only to web-*/svc-db-*. Defensive type checks keep malformed
non-auto-default configs from raising, matching prior behavior.
Tests updated to cover the svc-prx-openresty case and renamed to
reflect the new auto-default-prefix semantics.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
find_provider_roles now uses build_service_registry_from_roles_dir, the
same resolver the SERVICES_DISABLED pre-check uses. Previously it matched
only roles with a literal compose.services.<svc>.image entry, so services
exposed via compose.services.<entity>.provides (e.g. mailu → email) were
invisible to the init phase. The pre-check then flagged the unresolved
provider role as a conflict and aborted every fresh-purged deploy that
disabled "email".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Commit 9b5de4e47 removed the `domains` set_fact in 01_constructor.yml
and migrated only the `domain` lookup to get_merged_domains, leaving
`tls`, `cert`, `compose_file_args`, and the dashboard `docker_cards`
lookup still requiring variables['domains']. That caused
`tls_common: required variable 'domains' is missing` during the OIDC
merge task (first lookup('tls', ...) call at defaults_oidc render).
Also updates sys-svc-compose tasks/handlers so the CA-trust when-clauses
use `lookup('domains') | has_domain(application_id)` instead of the
removed top-level variable. Unit tests for the four lookups now patch
get_merged_domains to read from variables['domains'] so they stay
hermetic without touching the real roles/ tree.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Role-local lookup_plugins/ only autoloads when the role executes, but
oidc_flavor is called at config-merge time from
roles/web-app-nextcloud/config/main.yml during the 01_constructor stage.
Ansible could not resolve the lookup and failed the merge.
Relocate the plugin to plugins/lookup/ (the path declared in
ansible.cfg) so it is discovered before role execution, and move the
unit tests alongside it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Removes the placeholder interpreter (utils/dict_renderer.py) in favor of
standard {{ lookup('config', ...) }} / {{ lookup('domain', ...) }} calls
inside role configs, and eliminates the `domains` set_fact stage in
tasks/stages/01_constructor.yml by introducing a cached get_merged_domains
helper plus a new `domains` lookup plugin (full map + by-app subtree).
The `domain` lookup no longer depends on a pre-computed `domains` variable;
it delegates to get_merged_domains, which respects inventory-level overrides
from both `applications:` (authoritative canonical source) and `domains:`
(direct override map). Caching mirrors the applications pipeline — keyed on
(roles_dir, variables signature, override fingerprint) with a re-entry guard.
Replaced `{{ domains[app_id] }}` / `{{ domains }}` usages across web-app-matrix,
-bluesky, -mastodon, -peertube, -keycloak, test-e2e-playwright, and
group_vars. Added two performance tests for lookup('domains') mirroring the
applications cache assertions (≥50x warm-to-hot ratio, 120s cold budget).
- Resolve merge conflict in web-app-peertube/vars/main.yml; migrate
remaining `applications |` and `users.*` references to
`lookup('applications')` / `lookup('users', ...)` in web-app-odoo and
web-app-decidim (compose, env, playwright, setup tasks).
- web-app-peertube: split OIDC tasks into oidc/{enable,disable}.yml,
add install_mem_limit filter + PEERTUBE_MEM_LIMIT /
PEERTUBE_OIDC_PLUGIN_INSTALL_MEM_LIMIT vars to raise the cgroup ceiling
during the one-off `npm run plugin:install` exec (prevents OOM-SIGKILL
that was surfacing as "Container is restarting"); add
upsert-oidc-plugin.sql.j2 and diagnostics.sh.
- CI: test-deploy-{local,server,universal,workstation}.yml tweaks plus
scripts/github/{dump_runner_state,enlarge_swap}.sh for constrained
runners.
- cli/meta/applications/ressources: new CLI entry with unit tests.
- Misc config/docs updates across dashboard, mailu, mattermost, asset,
cdn, file roles and contributing docs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Resolve conflicts by migrating lookup plugins to utils.applications.config
(get_app_conf -> get) while preserving branch's runtime applications/users
lookups. Notable changes:
- plugins/lookup/{config,nginx,service}.py, roles/web-app-dashboard/
lookup_plugins/docker_cards.py: switch to utils.applications.config.get,
keep utils.runtime_lookup_data.get_merged_applications/_users.
- roles/web-app-matrix/docs/Installation.md: use lookup('config', ...)
syntax with markdown link formatting.
- scripts/tests/e2e/rerun-spec.sh: take main's single-source image tag
derivation from roles/test-e2e-playwright/defaults/main.yml.
- tests/integration/test_get_app_conf_paths.py -> test_config_lookup_paths.py:
migrate scan pattern to lookup('config', ...) and skip docs/ directory.
- tests/unit/plugins/lookup/test_config_lookup.py: union of new imports.
- tests/unit/plugins/lookup/test_cert.py: reset runtime caches in
setUp/tearDown to avoid id()-based fingerprint cross-test pollution.
- Add __init__.py in 7 test directories left behind after prior cleanup.
- Delete dead duplicate group_vars/all/{04_users,05_applications}.yml
(untracked on disk; content long migrated to roles/*/{config,users}/main.yml).
Reintroduce the conditional OIDC-plugin selection that was lost when commit
77a0e16ea removed the _applications_nextcloud_oidc_flavor helper. Ship a
role-local lookup plugin at roles/web-app-nextcloud/lookup_plugins/oidc_flavor.py
with resolution order: explicit compose.services.oidc.flavor > oidc_login if
LDAP enabled > sociallogin. Wire all consumers (plugin toggles + oidc.config.php)
through lookup('oidc_flavor') and default compose.services.oidc.flavor to "" so
the LDAP-aware default takes effect (None would trip the non-empty-config
integration guard). Add 12 unit tests covering override, LDAP on/off,
whitespace, non-string, missing-app, and input-validation paths; evict
stubbed utils.applications.config leaked by the keycloak redirect_uris test
before loading the plugin so discovery-order no longer matters. Seed missing
__init__.py in tests/unit/roles/web-app-peertube and
tests/unit/cli/meta/applications/ressources so the repo-wide init-file guard
passes again.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Plugin install via `container exec` shares the cgroup with the long-running
peertube node process. Combined RSS was tripping the 8g mem_limit and
SIGKILLing peertube, which surfaced as "Container is restarting" on
subsequent exec attempts.
- Temp raise mem_limit + unlimited swap for the install window, restore
the compose-defined value on the happy path only
- Derive the bumped ceiling from PEERTUBE_MEM_LIMIT + install heap cap
via a new role-local `install_mem_limit` filter (overhead=4 covers
native RSS and spawned npm/node-gyp workers)
- Extract the SQL upsert into templates/upsert-oidc-plugin.sql.j2
- Move enable/disable OIDC tasks under tasks/oidc/ and rename the
diagnostics script
- Document the extract-code convention in docs/contributing/tools/ansible.md
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a container-shaped heuristic so non-shared compose.services entries
that declare neither resource fields (mem_*/pids_limit/cpus) nor
container-identifying fields (image/name/version/container) are treated
as feature flags and excluded from the resource table. Keeps the
keycloak recaptcha toggle and similar booleans out of the per-role
footprint. Adds unit coverage for the helper and recursion behavior.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a CLI that reports per-container resource budgets (mem_reservation,
mem_limit, pids_limit, cpus) for an Ansible role, recursively following
enabled shared services via the service registry. Aggregation rules:
mem_* and pids_limit are summed; cpus is MAX (shared across containers,
not additive). Includes text + JSON output, cycle protection, and unit
tests covering parsing helpers, aggregation, recursion, rendering, and
end-to-end CLI invocation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Consolidates application-config helpers under utils/applications/ with a
concise public API (config.get). Updates all plugin, role, and test
imports; removes the legacy get_app_conf filter and its tests.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Rename 06..19 → 04..17 so the sequence is dense and gap-free.
- Update hard-coded references in cli/, tests/, and PR template.
- Add README.md describing scope, load order, and naming rules
per docs/contributing/documentation.md standards.
Speed up per-play lookups that previously re-read YAML and re-templated
against a cold Templar on every invocation:
- applications_current_play: cache the rendered applications map keyed
on the current play's identity so the 28+ plugin install calls reuse
one resolved result.
- runtime_lookup_data / load_app: memoize the raw applications load and
reuse it from the templar path instead of rereading group_vars.
- templating: merge caller-supplied variables onto templar's existing
available_variables (keeping ansible_facts / hostvars visible) and
tag the input with TrustedAsTemplate so Ansible 2.19+ actually
renders {{ ... }} instead of returning it unchanged.
Tests: add integration perf test + unit regression guarding the
available_variables merge restores prev state after the call.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Lookups for 'service', 'service_registry', and 'applications_current_play'
always re-derive the merged applications mapping from inventory+role defaults
instead of trusting the passed kwargs, which could be shadowed or left
unrendered by nested template evaluations.
Dashboard/keycloak tests now consume the administrator password via
lookup('users','administrator').password so the rendered Playwright .env
matches the realm user provisioned by Ansible. Remove the hardcoded
DASHBOARD_KEYCLOAK_TEST_LOGIN_PASSWORD / KEYCLOAK_LOCAL_REALM_ADMIN_PASSWORD
overrides that previously masked the mismatch.
run_one.yml: drop users/applications from the template vars block. Those
entries created a shadowing context that broke nested lookup('users', ...)
inside lookup('template', ...).
utils/runtime_lookup_data: decrypt Ansible 2.19 EncryptedString values at
the lookup boundary and try templar rendering before falling back to
aggregated defaults when override mappings arrive as placeholders.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
