These three sub-keys lived under `galaxy_info:` purely as a workaround
because Galaxy silently ignores unknown keys (same pattern as the
`run_after` / `lifecycle` migration in req-010). The new lint
`tests/lint/repository/test_role_meta_main_galaxy_schema.py` rejects
them, and no consumer in `plugins/`, `utils/`, `cli/`, `filter_plugins/`
or `roles/` reads any of them, so the strip is safe.
Round-trip via ruamel.yaml so the remaining `galaxy_info:` block keeps
its key order, quoting style and inline comments.
224 files touched. Remaining lint offenders after this commit are still
present (`logo`, `homepage`, `video`, plus a handful of roles missing
required Galaxy fields like `min_ansible_version` / `platforms` /
`license`); those are a separate decision and not folded into this
change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Validates every roles/<role>/meta/main.yml against the Ansible Galaxy
schema:
* top-level keys ⊆ {galaxy_info, dependencies, collections,
argument_specs, allow_duplicates};
* galaxy_info sub-keys ⊆ Galaxy spec set (role_name, namespace, author,
description, company, license, min_ansible_version,
min_ansible_container_version, platforms, galaxy_tags,
issue_tracker_url, github_branch, standalone);
* required galaxy_info fields present (author, description, license,
min_ansible_version, platforms);
* platforms is a non-empty list of mappings each with a string `name`
(and a list `versions` when present);
* galaxy_tags is a list of non-empty strings;
* stringly-typed galaxy_info fields are actually strings;
* dependencies (when present) is a list of strings or mappings with a
`role`/`name` key.
The test reuses the process-wide LRU cache from tests.utils.fs.read_text
for file contents and adds a parser-level @lru_cache so each meta/main.yml
is parsed once per pytest run.
When the lint fails it lists every offending file with the exact problem
per line, so the fix-up scope is visible at a glance.
Currently 226 of ~250 roles fail the lint because of project-internal
keys (logo, license_url, repository, documentation, homepage, video)
historically buried under galaxy_info — same anti-pattern req-010 fixed
for run_after / lifecycle. Migrating these is out of scope for this
commit and tracked as a follow-up requirement.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add CI_ENABLE_AUTO_UPDATES (default unset/false). Workflow update
jobs and Dependabot PRs are skipped/auto-closed unless the variable
is set to true, so update noise is opt-in.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
SuiteCRM is not yet compatible with PHP 8.5.\n\nKeep the older image pin, mark the exception locally, and\nsuppress the docker version warning until the role can be\nmade compatible.
Follow-up to d002796a5. The formula `infinito_nexus_${INFINITO_DISTRO}` is
now defined in exactly one place — `scripts/meta/env/defaults.sh:18`,
exported outside the load-once guard so it tracks INFINITO_DISTRO across
matrix iterations (caller re-sources defaults.sh after distro changes).
Every other site reads INFINITO_CONTAINER strictly from env; nothing
re-derives the formula, no lazy/colon-dash fallbacks, no per-call defaults.
Production code aligned:
- cli/deploy/development/common.py: add resolve_container() (strict env
reader, raises SystemExit on unset). make_compose() calls it as an
early-fail guard so a missing defaults.sh source surfaces here instead
of as a confusing compose.yml interpolation error later.
- cli/deploy/development/deploy.py: stop re-deriving from compose.distro;
use resolve_container().
- compose.yml: container_name reads ${INFINITO_CONTAINER:?...} strictly,
so callers that bypass defaults.sh fail fast with a clear hint.
- scripts/meta/resolve/image/local.sh: drop ${INFINITO_DISTRO:-debian}
fallback; require INFINITO_DISTRO via :? error.
- scripts/tests/deploy/ci/dedicated.sh: drop local _container fallback.
- scripts/tests/deploy/local/deploy/fresh-kept-app.sh: drop redundant
${INFINITO_DISTRO:-debian} reassignment after sourcing env/all.sh
(defaults.sh already defaults the value).
- scripts/tests/deploy/local/deploy/fresh-purged-app.sh: drop
target_container fallback; INFINITO_CONTAINER is already in env.
- scripts/tests/deploy/local/exec/container.sh: drop the lazy
${INFINITO_CONTAINER:=infinito_nexus_...} default; defaults.sh source
in env/all.sh already guarantees the value.
- scripts/tests/dns.sh: source defaults.sh so callers only need to set
INFINITO_DISTRO (the implicit dependency was previously a footgun).
- scripts/system/tls/trust/{wsl2,linux}.sh: same — source defaults.sh,
drop the ${INFINITO_CONTAINER:-infinito_nexus_debian} fallback in
linux.sh.
- .github/workflows/test-dns.yml: drop redundant
INFINITO_CONTAINER: infinito_nexus_${{ matrix.distro }} from per-step
env blocks (3 sites). The env now flows from INFINITO_DISTRO via the
defaults.sh sourced inside the scripts.
- tests/unit/cli/deploy/development/test_matrix_deploy.py: replace the
module-level os.environ.setdefault('INFINITO_CONTAINER', 'infinito_nexus_arch')
shim — which itself encoded the formula — with a module-level patch of
resolve_container that returns an opaque test-fixture string. Drop the
now-unused distro="arch" field from the _args() Namespace fixture
(deploy.handler stopped consuming it after the --distro arg removal).
Documentation aligned:
- docs/contributing/environment/variables.md: rewrite the INFINITO_CONTAINER
row to spell out the SPOT contract — set automatically by defaults.sh,
never re-derived elsewhere, callers re-source defaults.sh after
changing INFINITO_DISTRO mid-process.
- docs/contributing/artefact/files/compose.yml.md: replace the
INFINITO_DISTRO row (which described the now-removed
`infinito_nexus_${INFINITO_DISTRO:-debian}` interpolation) with an
INFINITO_CONTAINER row noting the strict ${INFINITO_CONTAINER:?...} read.
Verified:
- Formula derivation: 1 production hit (the SPOT in defaults.sh:18).
- INFINITO_CONTAINER fallback patterns (:-/:=): 0.
- --distro CLI arg refs: 0 (one comment explaining the removal).
- make test green: 1553 unit + 168 integration tests, no skipped suites.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related improvements anchored on the same principle: the env var
INFINITO_DISTRO is the single source of truth for the development distro,
and INFINITO_CONTAINER is purely derived from it.
1. Fix the matrix-deploy purge failure (CI run 24942172018, job 73038526513,
web-app-wordpress @ fedora):
scripts/meta/env/defaults.sh eager-exported INFINITO_CONTAINER once
inside its load-once guard; ci/all.sh's iteration loop re-exported
INFINITO_DISTRO per round but the cached INFINITO_CONTAINER stayed at
the initial-default infinito_nexus_debian, so _purge_app_entities()
between rounds tried to docker exec into a container from the wrong
distro and failed with "No such container".
Move the INFINITO_CONTAINER derivation OUTSIDE the load-once guard so
every re-source picks up a CONTAINER name in lock-step with the current
INFINITO_DISTRO. ci/all.sh now re-sources defaults.sh after each
export INFINITO_DISTRO=\$distro to trigger that re-derivation. defaults.sh
stays the single spot that owns the formula.
2. Retire the --distro CLI arg from cli.deploy.development {up,down,stop,
restart,init,deploy,exec,logs,build}. The arg duplicated INFINITO_DISTRO
(its default already read from the env), creating a hidden second source
of truth that drifted from the env across matrix iterations. Replace it
with cli.deploy.development.common.resolve_distro() which reads
INFINITO_DISTRO strictly (raises SystemExit on unset/invalid), and
simplify make_compose() to take no args. All bash callers
(scripts/tests/deploy/{ci,local}/**, Makefile restart target) drop the
redundant --distro \"\${INFINITO_DISTRO}\" pass-through.
Verified: make test passes (1553 unit + 168 integration). The mechanism of
the WP fix is verified by a synthetic distro-cycling test of defaults.sh
re-derivation; the original CI multi-distro repro can only be confirmed
end-to-end by the next CI push (local single-distro deploys do not exercise
the stale-env path).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the agreed-upon move of run_after: and lifecycle: from
meta/main.yml.galaxy_info into meta/services.yml.<primary_entity> alongside
the rest of each role's service metadata. Primary entity is derived via
get_entity_name(role_name); multi-entity roles (bluesky/matrix) get a
dedicated metadata-holder entry next to their compose entities. Lifecycle
allow-list anchored on values actually present in roles today (planned,
pre-alpha, alpha, beta) plus stable/deprecated forward-compat. A small
helper (utils/roles/meta_lookup.py) hides the entity derivation from
callers and degrades gracefully on missing meta/services.yml. Depends on
both req-008 and req-009.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the agreed-upon migration of group_vars/all/{08_networks,09_ports}.yml
into per-role meta/server.yml.networks and meta/services.yml.<entity>.ports
with the always-category-keyed shape, the inter/local/public split, and the
top-level port: -> ports.inter rename + int-cast. Defines PORT_BANDS in
group_vars/all/08_networks.yml as the single source for suggester and lint,
specifies cli meta {ports,networks} suggest helpers, and pins the lint rules
including the flat-set host-bound port collision check that excludes inter.
Depends on req-008.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the agreed-upon migration of every role's metadata to single-
responsibility files under meta/: meta/schema.yml (with new optional default:
field), meta/users.yml, meta/server.yml, meta/rbac.yml, meta/services.yml
(file-root keyed by entity_name; long-tail config inlined), and
meta/volumes.yml. Drops the compose. prefix from all consumer paths and
enumerates the lookup plugins to update.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 934-line `utils/cache/data.py` monolith mixed cross-cutting helpers
(constants, yaml-loaders, deep-merge, signatures, render machinery)
with three independent domain caches (applications, users, domains).
Per the KISS/SPOT/DRY guidelines, split it along domain boundaries so
each file has a single, well-scoped responsibility:
utils/cache/__init__.py — package; owns _reset_cache_for_tests()
orchestrator
utils/cache/base.py — cross-cutting helpers: constants,
yaml-loaders, _deep_merge, cache-key +
content-fingerprint signatures, the
cross-domain re-entry guard, and the
templar-render machinery (the only
ansible-coupled symbol, lazy-imports
its dependency)
utils/cache/applications.py — variants + defaults +
get_merged_applications. Strictly
ansible-free at import time so the
GitHub Actions runner-host CLI path
(cli.deploy.development.init →
plan_dev_inventory_matrix →
get_variants) keeps working without
ansible installed
utils/cache/users.py — user defs, token store hydration,
alias materialization,
get_user_defaults, get_merged_users
utils/cache/domains.py — get_merged_domains, derived from the
merged applications view
utils/cache/data.py — DELETED. Façade is gone; the package
exports nothing the domain modules
don't already
Backward-compat: the facade was deleted (per operator request) and all
36+ consumers — 18 plugins, 2 CLI entry points, 9 test files, 4 doc
files, plus a role-internal lookup_plugin — were rewired to import
directly from the owning domain module. `_reset_cache_for_tests` is
now exposed at `from utils.cache import _reset_cache_for_tests`.
Tests added (per-module, focused):
tests/unit/utils/cache/test_base.py (31 cases)
tests/unit/utils/cache/test_applications.py ( 9 cases)
tests/unit/utils/cache/test_users_module.py (21 cases; suffix
disambiguates from
the existing
tests/unit/plugins/
lookup/test_users.py)
tests/unit/utils/cache/test_domains.py ( 6 cases)
The ansible-block invariants from CI runs 24934007615 and 24935979190
are now pinned via subprocess isolation: each ansible-less import test
spawns a fresh `python3 -c '…'` with a meta_path filter denying every
`ansible*` import and asserts the import + invocation succeed in that
clean process. The earlier in-process sys.modules-manipulation shims
fragmented module identity and broke under Python 3.13 inside the
infinito container; subprocess isolation has neither problem.
Doc-style audit (per docs/contributing/documentation.md):
- Removed em-dash from utils/cache/README.md (split into two
sentences)
- Updated stale `utils.runtime_data` reference in
docs/contributing/design/matrix-deploy.md
- Removed two SPOT self-declarations
(docs/agents/files/group_vars/applications.md and
docs/contributing/artefact/files/plugins/lookup/users.md) per
the documentation rule that SPOT status is conferred by
cross-links, not by the page asserting it about itself
Empirical proof:
- All four domain modules import without ansible (subprocess
isolation tests)
- get_variants(roles_dir=ROLES_DIR) returns 99 apps in the
ansible-blocked subprocess
- 142 cache tests + 1085 wider-suite tests pass; the only
remaining failures (test_host_vars free-function fixture bug,
test_runtime_performance timing flake) are pre-existing and
unrelated to this refactor (verified via `git stash`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI run 24935979190 failed every test-deploy-server / test-deploy-universal
job (~70 jobs) at the same import:
cli/deploy/development/init.py:125
→ cli/deploy/development/inventory.py:228 plan_dev_inventory_matrix
→ utils/cache/data.py:735 get_variants
→ utils/cache/data.py:675 _build_variants
→ plugins/lookup/application_gid.py:7
→ ansible.plugins.lookup
ModuleNotFoundError: No module named 'ansible'
The previous fix (commit 8e4886d70 / "lazy ansible imports") was
incomplete: it deferred `from plugins.lookup.application_gid import
LookupModule` to function-call time, but `_build_variants()` then
instantiated `ApplicationGidLookup()` immediately, which still pulls
the LookupBase parent and dies the same way. Lazy importing only buys
us anything if the calling code actually skips the import — which it
doesn't here. The runner-host Python (3.12) ships without ansible,
so the import has to be avoided structurally.
Split the dependency at the source:
- `plugins/lookup/application_gid.py`: extract a pure-Python
`compute_application_gid(application_id, roles_dir, base_gid)`
helper. Same algorithm (sort role ids by `<roles_dir>/<role>/config/
main.yml` glob, return base_gid + index), but raises `ValueError`
instead of `AnsibleError` so it has zero ansible coupling. The
ansible-facing `LookupModule` is now defined under a `try: from
ansible.plugins.lookup import LookupBase` guard — when ansible IS
installed (Ansible playbook process), the original
`lookup('application_gid', ...)` API works exactly as before; when
ansible is NOT installed (the GitHub runner host running the dev CLI),
the module imports cleanly and the helper is the only consumer.
- `utils/cache/data.py`: `_build_role_base_config` now calls
`compute_application_gid()` directly instead of constructing an
`ApplicationGidLookup()` instance. The `_build_variants` lazy import
introduced by 8e4886d70 is removed (no consumer left). Type-hint
parameter for `gid_lookup` removed — the function no longer takes one.
Empirical proof of the runner-host scenario, with `ansible` blocked
via a `MetaPathFinder`:
- `from utils.cache.data import get_variants` succeeds
- `get_variants(roles_dir=ROLES_DIR)` returns 99 entries
- `python3 -m cli.deploy.development init --help` exits 0
- 50 cache/discovery/oauth2_oidc unit + integration tests stay green
- the existing `TestImportableWithoutAnsible` regression class catches
any future regression at unit-test time
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Multisite variant deploy (variant=1 in meta/variants.yml) failed at
06_multisite.yml's `wp config set` loop, then the next iter failed even
earlier at 02_install.yml's `wp core install`. Two distinct shapes of
the same root cause — Ansible's `command:` module pipes the rendered
argv string through shlex, which strips the outer single-quotes from
`'blog.infinito.example'` before exec — meaning wp-cli received a
bareword and, because the loop used `--raw`, wrote it verbatim:
define( 'DOMAIN_CURRENT_SITE', blog.infinito.example );
define( 'PATH_CURRENT_SITE', / );
…which PHP can't parse. From that moment on, every subsequent `wp` CLI
invocation died at parse-time before it could do anything. The state
also persists across reuse-kept iterations because it lives inside the
WordPress docker volume — `wp config delete` couldn't clean it up
either, since it loads wp-config through PHP itself.
Fix:
- Split the Multisite constant-write loop in 06_multisite.yml into two:
one with `--raw` for true PHP literals (booleans, ints — MULTISITE,
SUBDOMAIN_INSTALL, SITE_ID_CURRENT_SITE, BLOG_ID_CURRENT_SITE) and a
second one WITHOUT `--raw` for string constants (DOMAIN_CURRENT_SITE,
PATH_CURRENT_SITE). Without `--raw`, wp-cli detects the type and
quotes the value itself — which is the right SPOT for that decision.
- Add a defensive `sed -i` at the very top of 01_patch_config.yml that
removes any pre-existing
`define( 'DOMAIN_CURRENT_SITE'|'PATH_CURRENT_SITE', ... );` line
before any `wp` CLI call runs. Pure textual substitution survives an
unparseable wp-config.php, so this self-heals carryover state from
pre-fix runs. The canonical, properly quoted versions are then
rewritten by 06_multisite.yml's string-write loop on the same play.
Empirical proof: iter 5 (`make deploy-reuse-kept-apps APPS=
web-app-wordpress VARIANT=1`) finished `ok=1630 changed=158 failed=0`,
wp-config.php now contains all seven Multisite constants in valid PHP,
and the Playwright suite reports 3 passed / 4 correctly skipped (the
Multisite sentinel passes; the Single-Site RBAC scenarios skip via
`test.skip(multisiteEnabled)` as intended).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI run 24934007615 broke every per-app deploy job (test-deploy-server,
test-deploy-universal — ~70 jobs) with the same trace:
cli/deploy/development/init.py:10
→ cli/deploy/development/inventory.py:26
→ utils/cache/data.py:15
→ plugins/lookup/application_gid.py:7
→ ansible.plugins.lookup
ModuleNotFoundError: No module named 'ansible'
`scripts/tests/deploy/ci/dedicated.sh:167` runs `python3 -m
cli.deploy.development init` directly on the GitHub Actions runner host
to materialise the dev inventory. The runner Python (3.12) ships without
`ansible`, but `utils.cache.data` was eager-importing
`ApplicationGidLookup` (extends `ansible.plugins.lookup.LookupBase`) and
`_templar_render_best_effort` (uses `ansible.errors.AnsibleError`) at
module scope. The cheap `get_variants` entry point — the only thing the
init step actually needs — therefore refused to import.
Move both ansible-coupled imports into the call sites that genuinely
need them:
- `_templar_render_best_effort` is now a lazy import inside
`_materialize_builtin_user_aliases` and `_render_with_templar` (the
two functions whose closures use it).
- `ApplicationGidLookup` is now a lazy import inside `_build_variants`
(the only caller that instantiates it). The forward reference in
`_build_role_base_config(gid_lookup: ApplicationGidLookup)` keeps
working because `from __future__ import annotations` is already in
effect; a `TYPE_CHECKING` import keeps static type checkers happy.
`merge_with_defaults` is pure-Python and stays at module scope.
Add `TestImportableWithoutAnsible` (two tests, one for `utils.cache.
data.get_variants`, one for the full
`cli.deploy.development.inventory.plan_dev_inventory_matrix` chain) that
artificially blocks `ansible` via a `MetaPathFinder` and asserts the
imports still complete. Without this guard a future eager import of an
ansible-coupled symbol would silently break ~70 CI jobs again.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The yourls Playwright admin-SSO spec was failing at /oauth2/callback with
HTTP 403 + `[AuthFailure] Invalid authentication via OAuth2: unauthorized`
because oauth2-proxy literal-matches `allowed_groups` against the JWT
groups claim, but the two sides emit different shapes:
- Keycloak's group-membership-mapper emits `/roles/<app>/<role>` (leading
slash, `full.path=true` is the realm-import default).
- The `rbac_group_path` lookup, by historical contract pinned in
tests/unit/plugins/lookup/test_rbac_group_path.py, returns the path
WITHOUT a leading slash.
Bridge the contract gap inside the oauth2-proxy template (the only consumer
that does literal equality) so the rest of the codebase keeps the existing
no-leading-slash contract. A new static guard test pins the normalisation
filter chain so future refactors get a fast signal instead of an HTTP 403
discovered via a multi-hour Playwright run.
While the file was open, also tighten its SPOT layout:
- Cache the `compose.services.oauth2.allowed_groups` lookup in a Jinja
`set` so the same lookup isn't called twice in the same render.
- Lift the hardcoded internal port `4180` into
`compose.services.oauth2-proxy.port`. container.yml.j2 (right side of
the port mapping) and oauth2-proxy-keycloak.cfg.j2 (`http_address`)
now read it via `lookup('config')`; both MUST stay in sync, that is
exactly what a single SPOT delivers.
- Move `configuration_file` from the role config root into
`compose.services.oauth2-proxy.configuration_file` so it lives next to
the other oauth2-proxy service settings; both consumers (tasks/main.yml
template destination and container.yml.j2 volume mount) updated.
- Drop unused top-level `allowed_roles` — no consumer in the repo, only
effective restriction is `compose.services.oauth2.allowed_groups` in
the per-app configs.
Empirical proof of the slash-normalize fix:
- iter 2 (pre-fix): yourls spec `admin sso login to admin panel, then
logout` fails with 60s timeout polling for /admin/, oauth2-proxy logs
show `[AuthFailure] ... unauthorized` 403 at /oauth2/callback.
- iter 3 (post-fix): same spec passes in 890ms, sibling biber-deny spec
passes in 384ms, deploy PLAY RECAP ok=1530 failed=0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four independent bugs were each blocking the autonomous
fresh-purged-apps iteration of web-app-{openwebui,wordpress,nextcloud,
phpmyadmin,yourls}; bundling them so the pipeline keeps running.
- cli/core/discovery.py: stop walking argv prefixes at the first flag
or path-shaped arg and ignore OSError so the matrix-init's
`infinito create inventory <abs-path> --vars '<huge JSON>'` no longer
hits ENAMETOOLONG inside `is_dir()`. New regression tests pin both.
- utils/cache/data.py: when this module moved to utils/cache/data.py,
PROJECT_ROOT silently slid one level deep, pointing ROLES_DIR at a
non-existent `<repo>/utils/roles`. Implicit lookups returned empty
defaults so `lookup('users','contact')` failed at deploy time as if
the user weren't defined. Use parents[2] and pin the invariant in a
new regression test.
- roles/web-app-keycloak/vars/main.yml: KEYCLOAK_PER_APP_LDAP_ROLES_MAPPER
inherited `subComponents`/`id` from the realm-import dictionary's base
`ldap-roles` mapper. kcadm rejects those keys on `create components`
("Unrecognized field 'subComponents'"), with stderr swallowed by
`2>/dev/null` in upsert_app_mapper.sh. Strip both keys exactly like
update/05_ldap.yml already does for the shared mapper.
- scripts/tests/deploy/local/deploy/fresh-purged-app.sh: pass
`RUNTIME=dev` through `--vars`, otherwise the host process running
init falls back to `detect_runtime() == "host"` and the Playwright
E2E gate (`RUNTIME in [dev, act, github]`) never fires.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both in-process cache layers — the cached YAML loader and the cached
runtime-data getters — now live next to each other under
`utils/cache/`. The package gains a README that documents the
"orthogonal layers, no third cache" rule so future contributors hit
the right primitive on the first look.
Moves:
- `utils/yaml_cache.py` -> `utils/cache/yaml.py`
- `utils/runtime_data.py` -> `utils/cache/data.py`
- `tests/unit/utils/test_yaml_cache.py` -> `tests/unit/utils/cache/test_yaml.py`
- `tests/unit/utils/test_runtime_data.py` -> `tests/unit/utils/cache/test_data.py`
- `tests/unit/utils/test_variants.py` -> `tests/unit/utils/cache/test_variants.py`
New:
- `utils/cache/__init__.py` (empty per project convention)
- `utils/cache/README.md` describing the two layers, when to reach for
which, lifetime, test-only helpers, and design guidelines (RFC 2119
keywords, emoji-after-headline, no self-SPOT declaration, no
slash-in-link-text).
- `tests/unit/utils/cache/__init__.py`
Mechanical updates:
- All imports across cli/, plugins/lookup/, tests/, and roles/ rewritten
from `utils.yaml_cache` -> `utils.cache.yaml` and
`utils.runtime_data` -> `utils.cache.data` (40+ files).
- `REPO_ROOT` parents-index in the moved tests bumped from `parents[3]`
to `parents[4]` to reflect the deeper nesting.
- Six broken markdown links in docs/ updated to the new module paths
(test_markdown_links lint test stays green).
- Two test files that used the `from utils import runtime_data` form
now use `from utils.cache import data as runtime_data` so the local
alias remains.
Tests: 848 passed across unit + lint suites (1 pre-existing collection
error in test_host_vars.py unrelated to this change).
Phase 2 of the YAML-cache migration. The cli/meta/applications/resolution/*
and cli/build/* modules walk the same `roles/<role>/<config|vars|meta>/...yml`
files repeatedly during a single invocation. Direct `yaml.safe_load` calls
re-parsed each file every time. This commit routes those reads through the
in-process cached loader added in the previous commit so each absolute
path is parsed at most once per Python process.
utils/yaml_cache:
- New `load_yaml_any(path, *, default_if_missing=...)` for files whose
root is not a mapping (Ansible task lists, etc.). Shares the cache
with `load_yaml`, so a hot path that mixes both still pays parse
cost only once per file.
- `load_yaml` keeps its strict dict-root validation; non-mapping roots
now point callers at `load_yaml_any` in the error message.
- Internal `_load_raw` shared between both entry points.
- Tests: +5 cases covering list/scalar roots, empty-file → {} coercion,
cache sharing across the two entry points, and strict rejection.
Migrated call sites (each preserves its existing public surface and
domain-error wrapper):
- cli/meta/applications/in_group_deps/__main__.py: per-role vars/main.yml
in `find_role_dirs_by_app_id` + the one-shot applications file load.
- cli/meta/applications/role_name/__main__.py: per-role vars/main.yml
scan in `get_role`.
- cli/meta/applications/resolution/run_after/__main__.py: per-role
meta/main.yml read.
- cli/meta/applications/resolution/dependencies/__main__.py: per-role
vars/main.yml AND meta/main.yml reads inside the transitive
dependency walk.
- cli/meta/applications/resolution/services/resolver.py: per-role
config/main.yml read in `_load_role_config` (the inner-most loop of
service-dependency resolution).
- cli/build/role_include/__main__.py: per-role meta/main.yml +
vars/main.yml reads inside the topological-sort build.
- cli/build/graph/__main__.py: per-role meta/main.yml read; the
tasks/main.yml reader uses `load_yaml_any` (root is a list).
`yaml` import is preserved in two files (cli/build/graph/__main__.py and
cli/meta/applications/in_group_deps/__main__.py) because they still call
`yaml.safe_dump` on output; only the read paths moved.
Tests: full unit + lint suites green (594 passed, 1 pre-existing
unrelated collection error in test_host_vars.py).
Out of scope for this commit (one-off readers where the cache adds
nothing): cli/validate/inventory, cli/fix/vars_main_files,
cli/create/inventory/{host_vars,services_disabler,mirror_overrides},
cli/create/role, cli/meta/callorder, cli/meta/roles/lifecycle_filter.
Those each read their target file at most once per invocation, so
caching would just add an indirection without saving disk I/O.
Many CLI tools walk the same `roles/<role>/<config|vars|meta>/...yml`
files multiple times during a single invocation (dependency
resolution, service resolution, role introspection, ...). Each direct
`yaml.safe_load(path.read_text())` re-parses YAML and re-touches disk;
for `infinito create inventory` over the ~600-role tree this dominates
wall-clock time.
This change introduces `utils/yaml_cache.py`, a small in-process,
per-absolute-path memoising loader, and migrates the three existing
ad-hoc loaders to delegate to it.
`utils/yaml_cache.py` exposes:
- `load_yaml(path, *, default_if_missing=_MISSING)`: cached read,
dict only at the YAML root, raises `FileNotFoundError` by default
for missing files (callers can opt into a default value).
- `dump_yaml(path, data)`: write-through that evicts the cached entry,
so a CLI tool that edits a file mid-run sees the new content on the
next read.
- `invalidate(path)`: explicit per-path drop for the rare cases where
an external process touches the file.
- `_reset_cache_for_tests()` for unit-test isolation.
Cache semantics are documented in the module docstring; the key point
is that the same dict instance is returned on cache hit (no deep
copy), so callers that want to mutate the result MUST copy first.
This trades that constraint for ~5x fewer disk reads during the hot
resolution loops without measurably increasing memory.
Migrated wrappers (each keeps its existing public surface + domain
error type so call sites stay untouched):
- `cli/create/inventory/yaml_io.py` — `load_yaml(path)` /
`dump_yaml(path, data)`. The legacy "missing file -> {}" contract
stays via `default_if_missing={}`. Non-mapping roots still surface
as `SystemExit`.
- `cli/meta/applications/resolution/combined/yaml_utils.py` —
`load_yaml_file(path)` re-raises errors as `CombinedResolutionError`.
- `cli/meta/applications/resolution/services/yaml_utils.py` —
`load_yaml_file(path)` re-raises errors as `ServicesResolutionError`.
Tests: `tests/unit/utils/test_yaml_cache.py` covers cache hit,
path-resolution sharing, missing-file behaviour with and without the
default, non-mapping rejection, write-through eviction, and explicit
invalidation. All existing unit + lint test suites stay green.
Phase 2 (auditing the remaining direct `yaml.safe_load` call sites in
`cli/` and migrating the hot ones) is intentionally out of scope here
to keep the diff focused; this commit is the prerequisite SPOT.
The Playwright E2E gate previously fired through `MODE_CI`, a boolean
that OR-ed three env-var markers (`GITHUB_ACTIONS`, `ACT`,
`INFINITO_MAKE_DEPLOY`) AND-ed with an `INFINITO_SKIP_E2E` opt-out.
That mechanism duplicated information `RUNTIME` already carries:
- `make deploy-*` already bakes `RUNTIME=dev` into the inventory at
init time (`RUNTIME_VARS_JSON='{"RUNTIME":"dev"}'`).
- CI runners (act / GitHub Actions) detect `RUNTIME=act|github` from
their canonical env markers via `cli.meta.runtime.detect_runtime`
during init.
- A bare `ansible-playbook` from a production shell stays at the
default `RUNTIME=host`.
Switching the gate to `RUNTIME in ['dev', 'act', 'github']` therefore
covers every case `MODE_CI` covered, AND folds the gate into the
single existing runtime SoT rather than maintaining a parallel one.
Changes:
- `tasks/stages/02_server.yml` E2E include now keys on `RUNTIME`.
- `group_vars/all/01_modes.yml` drops the `MODE_CI` block; a short
note explains the retirement.
- `scripts/meta/env/ci.sh` deleted (it only existed to export
`INFINITO_MAKE_DEPLOY=1`).
- The 5 `scripts/tests/deploy/local/deploy/*.sh` entry-points stop
sourcing `ci.sh`.
- `cli/deploy/development/deploy.py` no longer forwards the four
retired env markers into the container; the runtime decision is
already serialised into the inventory the deploy stage consumes.
- `scripts/tests/deploy/local/deploy/reuse-kept-app.sh` drops the
same four `-e` flags from its `docker exec`.
- `docs/requirements/006-playwright-service-gated-tests.md` gets a
prepended "MODE_CI retired" update note; the historical sections
remain for context but the dead `ci.sh` link target was reduced to
inline code.
Behaviour: any runtime where E2E was fired before still fires it now,
and the previous opt-outs have a single replacement: pass
`RUNTIME=host` (or `--vars '{"RUNTIME":"host"}'`) to skip Playwright on
a deploy that would otherwise have run it.
Splits all 23 inline SQL bodies in `roles/web-app-odoo/tasks/03_oidc.yml`
into 17 standalone `.sql` files under `roles/web-app-odoo/files/oidc/`,
referenced from the tasks via `lookup('file', 'oidc/<name>.sql')`.
Why:
- Inline `query: >` heredocs hide SQL from grep, syntax highlighters,
and SQL-aware tooling. Extracting them to dedicated files makes the
queries trivially diffable + parseable + reviewable.
- The two tasks that previously interpolated Jinja straight into SQL
(`'{{ ODOO_OIDC_AUTHORIZE_URL }}'` etc. for the Keycloak provider
upsert) move to `named_args` with `%(name)s` placeholders, removing
the SQL-injection surface those interpolations carried.
What also changes (kept tight to stay in scope):
- The five-line DB connection block (db / login_user / login_password /
login_host / login_port) is deduplicated via a `module_defaults` block;
the one task hitting the Keycloak DB instead of Odoo overrides them
inline.
- Seven `INSERT INTO ir_config_parameter ... ON CONFLICT (key) DO UPDATE`
tasks now share `upsert_ir_config_parameter.sql` (key + value via
named_args). The two specialised template-user-id upserts (which need
a subquery for the value) share
`upsert_ir_config_parameter_template_user_id.sql`.
Net effect: tasks file shrinks from 389 to 192 lines, every SQL body
sits in its own annotated `.sql` file with a header comment describing
intent, and the deploy-time behaviour is unchanged (same statements,
same ordering, same idempotency guards, just better-organised).
Introduces per-role `meta/variants.yml` for declaring multi-variant
deploys (e.g. WordPress Single-Site vs. Multisite) and reshapes the
development deploy CLI to materialise one inventory folder per variant
and iterate them with co-located sync+async passes.
Why:
- The previous single-inventory + Pass-1-then-Pass-2 split deployed each
variant twice and ran the async pass against a host left in a
different variant's state, masking real bugs.
- Inventory and runtime concerns were tangled: variant selection lived
as a runtime extra-var and required cross-stage coordination.
How:
- `roles/<role>/meta/variants.yml` (file format docs at
docs/contributing/artefact/files/role/variants.md): YAML list of
deep-merge overrides on top of `config/main.yml`. Empty entry `{}` is
variant 0 (= legacy payload). 6 roles get their dev overrides moved
here from inventories/dev.yml; WordPress declares 2 variants.
- `utils/runtime_data.py` gains `_build_variants` / `get_variants` and
drops the retired `_active_variants` runtime selector. Loader always
returns variant 0 as the default; non-zero variants reach
`get_merged_applications` as inventory-level `applications.<app>`
overrides baked at init time.
- New `cli/deploy/development/inventory.py`: `DevInventorySpec`,
`plan_dev_inventory_matrix` (pure planner shared between init and
deploy), `build_dev_inventory_matrix` (materialises one folder per
round at `<dir>-<R>`, or `<dir>` when single-variant). Single-folder
paths preserve full backward compat.
- `cli/deploy/development/deploy.py`: matrix loop with per-round
cleanup (only apps whose variant changed AND that are about to
re-deploy), `--variant <idx>` (or `VARIANT=` env) to pin a single
round, and `--full-cycle` (or `FULL_CYCLE=true`) which interleaves
sync + async per round so the async re-deploy targets exactly the
host state the matching sync deploy just produced.
- `cli/deploy/development/exec.py` gains repeatable `--env KEY=VAL` so
bash callers can inject per-run context into in-container helpers
without inlining heredocs.
- `scripts/meta/env/inventory.sh` exports `INVENTORY_VARS_FILE`
(Python mirror in `cli.deploy.development.common.DEV_INVENTORY_VARS_FILE`;
test_inventory_vars_file_spot.py locks them against drift).
- `make deploy-fresh-purged-apps` and `scripts/tests/deploy/ci/dedicated.sh`
switch to a single init + single deploy `--full-cycle`. The
`make deploy-reuse-*` bash scripts honour `VARIANT=` by suffixing
`INVENTORY_DIR`/`INVENTORY_FILE`.
- The 5 `scripts/tests/deploy/local/deploy/*.sh` scripts now delegate
their in-container heredocs to dedicated helpers in
`scripts/tests/deploy/local/utils/`, removing the bash-in-bash
quoting hell.
Tests:
- `tests/unit/utils/test_variants.py` (renamed from
`test_application_variants.py`): loader edge cases, variant-zero
default, retired-`_active_variants` guard.
- `tests/unit/cli/deploy/development/test_inventory.py` (new): planner,
variant baking, builder, `filter_plan_to_variant`, `DevInventorySpec`.
- `tests/unit/cli/deploy/development/test_matrix_deploy.py` (new):
per-round deploy filter, cleanup decisions, `--variant` pinning,
`--full-cycle` per-round interleave.
- `tests/unit/cli/deploy/development/test_inventory_vars_file_spot.py`
(new): drift guard for the bash <-> Python literal pair.
- `tests/unit/cli/deploy/development/test_exec.py` extended for `--env`.
Docs:
- `docs/contributing/artefact/files/role/variants.md` (new): file format.
- `docs/contributing/design/matrix-deploy.md` (new): runtime mechanism
(folder layout, --variant, --full-cycle, cleanup logic).
- `docs/contributing/actions/deploy.md` (new): contributor dev-deploy
flow (make targets, --variant pinning).
- `docs/administration/deploy.md` slimmed to admin-only (was mixed
admin+dev), cross-links to the new contributor doc.
- `docs/agents/action/iteration/role.md` keeps only agent-iteration
recipes; mechanism explanation moved out and linked.
- `docs/contributing/documentation.md` adds the "no self-SPOT
declaration" rule; `applications.md` renamed its SoT header for
consistency.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CLI installer already sets the admin password via --admin-password.
Running user:reset-password immediately after install races on Joomla's
per-invocation CLI session warm-up and intermittently fails with
"The user <name> does not exist\!" (observed in CI run 24915917415,
centos). Gate the reset on joomla_installed.rc == 0 so it only runs
on re-deploys, where it still enforces password rotation against an
existing user.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drop the `DEPLOY_CI_EXPORTS := INFINITO_MAKE_DEPLOY=1` declaration
from the Makefile and the per-recipe `$(DEPLOY_CI_EXPORTS)` prefix
from every `deploy-*` target. The marker now lives in a dedicated
on-demand env module `scripts/meta/env/ci.sh` and is sourced by every
deploy entry-point under `scripts/tests/deploy/local/deploy/*.sh`,
which keeps the export aligned with the rest of the project's env
layer (`scripts/meta/env/all.sh` and friends).
`ci.sh` is intentionally NOT composed into `all.sh`: the marker
flips `MODE_CI` to `true` and MUST surface only when a deploy
actually runs, not for unrelated `make test*` / `make build*`
recipes that share the same `BASH_ENV`.
`docs/requirements/006-playwright-service-gated-tests.md` and the
`scripts/meta/env/README.md` document the new SPOT and the
always-on / on-demand distinction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The tool now pins branch.main.pushRemote to origin, so main pushes go
upstream while other branches keep landing on the personal fork. The
layout table and the new Push Routing section reflect this and list
the three git-config keys the tool writes. Also tighten the sandbox
rule to MUST per the docs RFC-2119 style.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
On `delete` events the workflow runs from the default-branch context, so
`github.ref_name` evaluated to "main" and the concurrency group collided
with entry-push-latest.yml's main group — cancelling the merge-commit
CI: Push run on main. `github.event.ref` carries the actually deleted
branch name, so the group matches the intended push-CI group.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drop the project-local copy of scripts/git/sign-push.sh in favour of
the git-maintainer-tools CLIs (git-setup-remotes, git-sign-push),
which now ship as a standalone PyPI/GitHub package so the same helpers
can be reused across every maintainer repo without vendoring shell
scripts.
The package is declared in pyproject.toml's `dev` extras and lands on
$PATH via `make install-python-dev` (which already runs
`pip install -e .[dev]`). Consumers invoke the CLIs directly; there
are no make wrappers because doing so would just duplicate help text
the tool itself already prints.
Fallout:
* scripts/git/sign-push.sh deleted.
* Makefile: sign-push target removed (no replacement target).
* .claude/settings.json: deny rule for `make sign-push*` replaced
with direct `git-setup-remotes*` / `git-sign-push*` denies so the
sandbox blocks the real binaries, not just the make alias.
* CLAUDE.md: operator instruction points at the bare `git-sign-push`
CLI + `make install-python-dev` for setup.
* docs/contributing/artefact/git/remotes.md: new tool-focused page.
Describes the layout and the two CLIs; does not duplicate raw git
commands already covered by git-maintainer-tools.
* docs/contributing/artefact/git/README.md: trimmed per readme_md.md
(docs READMEs must not act as a file index).
* docs/contributing/tools/make.md: Git section now documents the
external CLIs and the install entry point, not a wrapper target.
- Remove every em-dash thought break (4 in 004, 24 in 006, 2 in
the WordPress role README) per the "no em-dashes" rule in
`docs/contributing/documentation.md`.
- Replace "is forbidden" with the RFC 2119 keyword "MUST NOT" in
005 and 006.
- Disambiguate the ambiguous `[rbac.md](...)` links in the role
README, the administration SPOT, and the design SPOT by giving
each link a descriptive label ("Administration RBAC" /
"RBAC design"), per the link-text rules in `documentation.md`.
- Move the emoji decorations on `roles/web-app-wordpress/tasks/
plugins/README.md` headings from before to after the heading
text and turn the U+2192 arrow into prose so the document
matches the heading-emoji rule.
- Update `docs/contributing/design/iam/rbac.md` so the LDAP
layout, Keycloak mapper, and `path_join` paragraphs reflect
the flat-CN + per-app mapper implementation that requirement
005 actually ships, instead of the hierarchical-OU vision the
page was originally drafted against.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
