Add a dedicated lifecycle reference and back it with concrete role
metadata changes:
- `docs/contributing/design/services/lifecycle.md`: enumerate the
values `meta/services.yml.<entity>.lifecycle` MAY take and the
criteria each value commits the role to. Linear axis runs from
`planned` -> `pre-alpha` -> `alpha` -> `beta` -> `rc` -> `stable`
-> `maintenance` -> `deprecated` -> `eol`, with the inner five
forming the tested envelope (matrix-deploy + Playwright gate
applies). `beta` requires single sign-on against
web-app-keycloak / svc-db-openldap whenever the upstream software
ships an OIDC or LDAP adapter; `rc` additionally requires a live
burn-in deploy on infinito.nexus.
- Introduce a parallel `unsupported` tier off the linear axis for
roles the project ships without test or maintenance commitment
(proprietary products, demo prototypes). The role's `README.md`
MUST state the unsupported status explicitly.
- `docs/contributing/design/services/layout.md`: cross-link the new
lifecycle.md as the semantic counterpart to the on-disk shape
documented under `run_after` and `lifecycle`.
- Move web-app-confluence, web-app-jira, web-app-magento from
`alpha`/`planned` to `lifecycle: unsupported`. They are commercial
proprietary products / catalogue demonstrations the project does
not commit to maintain. Their meta entries flip to `unsupported`
to match.
- `tests/lint/repository/test_role_meta_layout.py`: extend
`ALLOWED_LIFECYCLES` to cover the full set introduced in
lifecycle.md (`rc`, `maintenance`, `eol`, `unsupported`) so the
lint guard accepts the documented values without false positives.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both `roles/web-app-mig/vars/main.yml` and
`roles/web-app-sphinx/vars/main.yml` resolved `container_port` via
`lookup('config', application_id, 'services.' ~ entity_name ~ '.port')`,
but neither role's `meta/services.yml` declares a flat `port` key —
they declare `services.<entity>.ports.{inter, local.http}`. Deploy on
arch crashed at compose-template render time with
`The lookup plugin 'config' failed: Key 'port' not found in dict`
(GHA universal-deploy run 25028370090 jobs 73306260785 + 73306260788).
Match the convention used by web-app-listmonk / -gitea / -shopware
and read `services.<entity>.ports.inter` instead.
The existing `tests/integration/lookups/test_config_paths.py`
literal-path scanner only captures the first quoted segment of a
`lookup('config', ...)` call, so `~`-concatenated paths (where a
Jinja variable expands inside the dotted path) were validated only
as wildcard prefixes. That contract is too loose: any role with a
top-level `services` dict satisfied
`'services.' ~ entity_name ~ '.port'`, even when the resolved path
did not exist for the specific role.
Add a `test_wildcard_paths` method that:
- captures the entire path expression in any
`lookup('config', <app>, <expr>)` whose `<expr>` contains `~`,
- parses the expression token by token, treating quoted strings as
literals and any bareword between `~` markers as `*`,
- joins the parts and splits on `.` into a wildcard path
(for example `'credentials.' ~ X ~ '_jwt_secret'` becomes
`credentials.*_jwt_secret`),
- resolves the role from the file's `roles/<role>/...` path when
the lookup's app argument is a Jinja variable,
- validates the wildcard path through the same fallback chain as
the literal-path test (application defaults -> users defaults
-> credentials in defaults / schema -> images presence), with
per-segment fnmatch globbing so partial-segment wildcards like
`*_jwt_secret` resolve correctly.
Verified: reverting the two role fixes makes the new test report
`'services.*.port' has no match in application defaults / schema for
role 'web-app-mig'` (and the same for sphinx) with file/line refs.
Restoring the fixes turns it green again.
Also drop one extra `..` segment from three relative links in
`docs/contributing/design/inventory.md` so the markdown lint check
passes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two design docs under docs/contributing/design/:
- variants.md (renamed from matrix-deploy.md): explains the
folder-per-round wrapper that consumes per-role meta/variants.yml
and iterates variants at deploy time. Headings normalised to
sentence-case; the "legacy unsuffixed inventory path" wording
rewritten to drop the historical-context phrasing forbidden by
documentation.md.
- inventory.md (new): explains how a single round's inventory is
assembled out of meta/services.yml plus the variant payloads,
which inclusion mechanisms exist (operator --apps, run_after walk,
shared-service auto-include via strict `is True`), what lives in
the inventory file vs. host_vars/<host>.yml, and when Jinja
actually gets rendered (only at play runtime, via
utils.cache.applications._render_with_templar). Worked example
documents the WordPress + Discourse wiring as a reference case.
Cross-links updated wherever they pointed at matrix-deploy.md or
its concept name as a file:
- docs/agents/action/iteration/role.md
- docs/contributing/actions/deploy.md
- docs/contributing/artefact/files/plugins/lookup/applications.md
- docs/contributing/artefact/files/role/variants.md
- scripts/tests/deploy/local/deploy/reuse-kept-app.sh (path comment)
Plain-text occurrences of "matrix-deploy" as a concept term inside
Python comments, log messages, and test docstrings stay as-is; only
file-link references are migrated.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Documentation MUST describe the current state of the subject only.
Phrases like "this replaces X", "previously X did Y", "formerly known
as X", "the legacy X", or "after the refactor" rot as soon as the next
refactor lands and confuse readers who arrive in the present. Anchor
historical context in commit messages, pull request descriptions, and
changelogs instead, where it is tied to a point in time.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Brings up the package-cache compose service alongside the existing
registry-cache, gated by the same `cache` profile. Bootstrap helper
(scripts/docker/cache/package.sh) auto-rotates the admin password from
the well-known default, accepts the Sonatype CE EULA (3.71+ blocks
artifact downloads with HTTP 403 until /v1/system/eula is POSTed —
disclaimer must be sent verbatim with its unicode quotes), enables
anonymous read, creates one default blobstore, and registers the MVP
proxy set: apt-debian, apt-ubuntu, pypi-proxy, npm-proxy, helm-bitnami,
raw-githubusercontent. The runner gets pip.conf / .npmrc /
sources.list snippets bind-mounted in (each defaulting to /dev/null
when the cache profile is inactive) so the deploy is byte-identical
with cache off.
Cache plumbing:
- scripts/meta/env/cache/{registry,package}.sh: per-cache env defaults,
hostname-derived admin password seed (uses `hostname` command, not
$HOSTNAME shell var, so the value is identical across interactive
shells, non-interactive subshells, and Python subprocesses).
- scripts/docker/cache/{registry-ca,package}.sh: container-side
bootstrap helpers, both moved under scripts/docker/cache/.
- scripts/system/cache/clean.sh + `make cache-clean`: wipes both
/var/cache/infinito/core/cache/{registry,package} dirs and stops
the cache containers.
- cli/deploy/development/compose.py: Profile gate exports the three
new INFINITO_PACKAGE_CACHE_* client-snippet paths and runs the
bootstrap helper after the package-cache reports healthy.
Compose service uses sonatype/nexus3:3.91.1-ubi pinned, runs as
root (UBI image refuses to chown a fresh bind-mount), and forces
NEXUS_SECURITY_RANDOMPASSWORD=false so the bootstrap can log in with
admin/admin123 before rotating; the random-password mode otherwise
auto-completes onboarding silently and leaves no readable secret on
disk for an automated bootstrap to pick up.
Tests: tests/unit/cli/deploy/development/test_compose.py grows two
new assertions (cache services listed in `up` args when local; build
tests pin CI=true to retain ci-only assertions).
Closes req-012.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds Sonatype Nexus 3 OSS as a second service under the existing
`cache` compose profile, complementing (not replacing) the rpardini
docker registry-cache. Covers apt / pypi / npm / helm / raw via
proxy repos. Mirrors the registry-cache wiring conventions: env
script under scripts/meta/env/, strict ${VAR:?…} contract, Profile
gate, /dev/null bind-mount fallback when the profile is inactive.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The three INFINITO_REGISTRY_CACHE_* variables previously carried their
defaults in compose.yml as `${VAR:-...}`, which baked /tmp paths and
a static 32g cache size into the YAML. Move the defaults into a new
shell module that is sourced via `scripts/meta/env/all.sh` and tighten
compose.yml to require the variables strictly via `${VAR:?...}`.
Why
* Adapt cache size to the host. Hard-coding 32g hurts hosts where
`/tmp` is a small tmpfs (typical Linux: ~half of RAM); the cache
would either be capped above the partition or starve other tmpfs
users. Computing the default as half the free disk space at the
cache path leaves a usable budget for the rest of the system on
small hosts and grows on large ones.
* Single source of values. With defaults in shell, the compose YAML
no longer carries duplicated tunables; `compose.yml` just consumes
what the env layer provides. Same pattern as the existing
INFINITO_CONTAINER strict consumption.
Changes
* New scripts/meta/env/registry_cache.sh exports the three vars.
CACHE_MAX_SIZE = `df --output=avail -B1G "$path" / 2`, walking up to
the closest existing ancestor when the cache path itself does not
exist yet (it is created on first start by `bind:
create_host_path: true`). Minimum 1g, fallback 2g if `df` fails.
* scripts/meta/env/all.sh sources the new module so the Makefile-driven
flow picks it up automatically.
* compose.yml: three `${VAR:-default}` -> `${VAR:?Source scripts/meta/env/registry_cache.sh ...}`
so the absence of the env layer fails fast with a precise message.
* Stale block comment above the registry-cache service (CI-only,
/tmp-defaults, etc.) collapsed to a one-line pointer to the README,
per documentation.md "default to no comments / explain only the
non-obvious why".
* docs/contributing/environment/variables.md and
docs/contributing/artefact/files/compose.yml.md describe the new
defaults source and the half-free-disk computation.
Verified
`bash -c 'source scripts/meta/env/all.sh && docker compose --profile ci --profile cache config'`
resolves the three vars (CACHE_MAX_SIZE: 3g on a host with 7G free
in /tmp). Without sourcing, compose fails with the exact
"Source scripts/meta/env/registry_cache.sh ..." message.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move the registry-cache pull-through proxy out of the always-on `ci`
profile into its own `cache` profile, and select it based on the runtime
environment instead of statically.
Why
The proxy only earns its keep when image pulls amortize across many
runs on the same host. That holds on developer machines (cross-run
image dedup, persistent host disk) but not on GitHub-hosted runners
(fresh disk per job), where the proxy startup overhead is net loss
without the corresponding payoff.
How
* Compose: split `registry-cache` into its own `cache` profile;
`infinito.depends_on.registry-cache` gets `required: false` so Compose
does not auto-start the proxy via the dependency when the profile is
inactive; the systemd drop-in bind-mount source is gated by the new
`INFINITO_REGISTRY_CACHE_PROXY_CONF` env var with `/dev/null` default,
so dockerd loads an empty drop-in (no `HTTP_PROXY`, no `ExecStartPre`)
when the proxy is not running and pulls images directly.
* New `cli/deploy/development/profile.py` `Profile` class is the SPOT
for which compose profiles activate. KISS + SRP: `is_ci()` reads the
three CI signals (`GITHUB_ACTIONS`, `RUNNING_ON_GITHUB`, `CI`),
`registry_cache_active()` is its strict inverse, and `args()` returns
the matching `--profile ...` flag list.
* `Compose` and `down.py` consume the same `Profile` so up/down cannot
drift on which services are part of the stack.
* Documentation under `compose/registry-cache/README.md`,
`docs/contributing/artefact/files/compose.yml.md` and
`docs/contributing/environment/variables.md` describes the new toggle
and the `INFINITO_REGISTRY_CACHE_PROXY_CONF` env var. Pre-existing
documentation.md violations in those files (one SPOT self-label, one
em dash, two path-as-link-text instances) corrected in the same pass
per the fix-nearby rule.
* Tests: 14 new `Profile` unit tests in `test_profile.py` plus two
scenario tests in `test_compose.py` that lock down the
`--profile ci` (CI runner) vs `--profile ci --profile cache` (local)
argument shape.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduces a `registry-cache` service (rpardini/docker-registry-proxy) in
the `ci` profile that MITM-caches every outbound docker registry pull
made by `infinito`'s inner dockerd, drastically cutting bandwidth on
re-deploys, parallel CI shards, and local fresh-purges.
Wiring (all bind-mounted via compose.yml, no Dockerfile or entry.sh
staging, so edits hot-reload on next container start):
- CA bundle at /opt/registry-cache-ca (read-only),
- systemd drop-in at /etc/systemd/system/docker.service.d/registry-cache-proxy.conf,
- registry-cache-ca.sh at /usr/local/bin/registry-cache-ca.sh.
The drop-in registers registry-cache-ca.sh as ExecStartPre. Compose
gates `infinito` on `registry-cache` `service_healthy` so the CA is on
disk before dockerd boots; the install script supports both Debian
(`update-ca-certificates`) and Arch/Fedora (`update-ca-trust extract`)
trust stores, detected by command availability.
Also reorganizes compose/ into per-service subdirectories
(compose/coredns/, compose/registry-cache/) with README.md per directory
following the documentation guidelines, adds scripts/docker/README.md,
and nests the registry-cache host paths under /tmp/infinito/core/
(mirror/ for blobs, ca/ for the MITM CA bundle).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
req-011 — descriptive role-level metadata
* Move logo / homepage / video / display out of
meta/main.yml.galaxy_info into a new project-owned
roles/<role>/meta/info.yml (file-root convention, no `info:` wrapper).
* 78 role meta/main.yml files migrated via ruamel round-trip
(key-order / quoting / inline-comments preserved). Roles with none
of the four fields are left untouched (no empty info.yml created).
* web-app-dashboard/lookup_plugins/docker_cards.py reads logo +
display from meta/info.yml; description / galaxy_tags continue to
come from galaxy_info (Galaxy-spec fields).
* docker_cards.py now routes ALL its file reads through the
process-wide caches: load_yaml_any for vars/main.yml,
meta/main.yml, meta/info.yml; read_text from utils.cache.files for
README.md. The orphan `import yaml` is gone.
New utils/cache/files (renamed from tests/utils/fs)
* The process-wide filesystem cache (read_text, iter_project_files,
iter_project_files_with_content, _all_project_files) is not
test-specific; it now lives under utils/cache/files.py next to the
other process-wide caches. 34 consumer imports rewritten across
tests/ and docs/. The empty tests/utils/ package is removed.
* New unit suite tests/unit/utils/cache/test_files.py — 12 cases
covering read_text, iter_project_files (skip-dirs, extensions,
exclude_tests, exclude_dirs, walk-cache invariants), the combined
iterator, non-UTF-8 silent skip, and _reset.
utils/cache redundancy cleanup
* applications.py + users.py + cli/create/inventory/services_disabler.py
drop their direct yaml.safe_load calls and their private
_load_yaml_mapping/_load_yaml_variant_list copies; everything routes
through utils.cache.yaml.load_yaml / load_yaml_any. The dead
utils.cache.base._load_yaml_mapping is removed.
* users.py:_load_store_users (tokens.yml read) likewise migrated.
* _load_yaml_variant_list moves out of base.py to applications.py
(its only consumer) and is renamed _load_variants_overrides — also
routes through the cached loader now.
* utils.cache.yaml._reset_cache_for_tests is renamed _reset (kept as
alias) for parity with the per-domain reset helpers.
* utils.cache.files grows a _reset() that clears its two LRU caches.
* utils.cache.__init__._reset_cache_for_tests now orchestrates ALL
six caches (applications, users, domains, base, yaml, files) so
tests get clean state across cache layers — needed because
applications/users now share the yaml cache and would otherwise
serve stale parses across reset boundaries.
* utils.cache.files imports PROJECT_ROOT from base instead of
redefining it.
* tests/unit/utils/cache/test_data.py + test_base.py: obsolete
TestLoadYamlMapping / TestLoadYamlVariantList classes deleted
(covered by test_yaml.py + test_variants.py).
* README.md updated for the new module list (files, yaml) and the
six-module orchestrator.
Galaxy schema lint
* tests/lint/repository/test_role_meta_main_galaxy_schema.py: split
galaxy_info field requirements into hard-required {author,
description, license} and soft-recommended {min_ansible_version,
platforms}. Soft-recommended -> emit a GitHub Actions ::warning::
annotation via utils.annotations.message.warning (titled
"meta/main.yml galaxy_info" with file= pointing at the offender),
no test failure. Rationale: min_ansible_version + platforms are
only enforced by Galaxy when *publishing*, which Infinito.Nexus
does not do.
* Galaxy lint goes green (was 226 offenders before req-011 +
license_url/repository/documentation strip; was 62 before this
relaxation; now 0 hard offenders, 62 soft warnings).
* roles/svc-opt-keyboard-color/meta/main.yml: backfill `license`
(only role missing the hard-required field).
New lint
* tests/lint/repository/test_role_meta_info.py — enforces meta/info.yml
file-root convention, the four-field allowlist (logo, homepage,
video, display), and shape (logo.class string, homepage/video
non-empty strings, display bool). Also fails if any
meta/main.yml.galaxy_info still carries one of the four migrated
fields.
Docs
* docs/contributing/design/services/layout.md: new section
"Descriptive Role-Level Metadata — meta/info.yml" with allowed-
fields table, optionality rule, materialised path, and consumer
note. File-Layout table grows the meta/info.yml row;
meta/main.yml row gets the explicit "no logo/homepage/video/display"
note. req-011 added to header and Related Pages.
* utils/cache/README.md: files.py + yaml.py rows added; six-module
reset orchestrator documented.
Validation
* Unit: 1573 passed.
* Lint: 55 passed.
* Integration: 161 passed (1 skipped) via pre-commit hook.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Specifies the move of project-internal descriptive role-level fields
(`logo`, `homepage`, `video`, `display`) out of
`meta/main.yml.galaxy_info` into a new project-owned
`roles/<role>/meta/info.yml` (file-root, no `info:` wrapper).
Same anti-pattern that req-010 fixed for `run_after` / `lifecycle` and
that the strip in 0ab817a83 already handled for
`license_url` / `repository` / `documentation`. After this requirement
lands, the Galaxy-schema lint
(`tests/lint/repository/test_role_meta_main_galaxy_schema.py`) will
go green for every remaining offender.
Includes:
* target layout + file-root convention;
* known consumer (`web-app-dashboard/lookup_plugins/docker_cards.py`);
* acceptance criteria for file migration, schema rules, loader,
consumer rewrites, lint, docs, atomicity;
* validation app set + final iteration loop.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add CI_ENABLE_AUTO_UPDATES (default unset/false). Workflow update
jobs and Dependabot PRs are skipped/auto-closed unless the variable
is set to true, so update noise is opt-in.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Follow-up to d002796a5. The formula `infinito_nexus_${INFINITO_DISTRO}` is
now defined in exactly one place — `scripts/meta/env/defaults.sh:18`,
exported outside the load-once guard so it tracks INFINITO_DISTRO across
matrix iterations (caller re-sources defaults.sh after distro changes).
Every other site reads INFINITO_CONTAINER strictly from env; nothing
re-derives the formula, no lazy/colon-dash fallbacks, no per-call defaults.
Production code aligned:
- cli/deploy/development/common.py: add resolve_container() (strict env
reader, raises SystemExit on unset). make_compose() calls it as an
early-fail guard so a missing defaults.sh source surfaces here instead
of as a confusing compose.yml interpolation error later.
- cli/deploy/development/deploy.py: stop re-deriving from compose.distro;
use resolve_container().
- compose.yml: container_name reads ${INFINITO_CONTAINER:?...} strictly,
so callers that bypass defaults.sh fail fast with a clear hint.
- scripts/meta/resolve/image/local.sh: drop ${INFINITO_DISTRO:-debian}
fallback; require INFINITO_DISTRO via :? error.
- scripts/tests/deploy/ci/dedicated.sh: drop local _container fallback.
- scripts/tests/deploy/local/deploy/fresh-kept-app.sh: drop redundant
${INFINITO_DISTRO:-debian} reassignment after sourcing env/all.sh
(defaults.sh already defaults the value).
- scripts/tests/deploy/local/deploy/fresh-purged-app.sh: drop
target_container fallback; INFINITO_CONTAINER is already in env.
- scripts/tests/deploy/local/exec/container.sh: drop the lazy
${INFINITO_CONTAINER:=infinito_nexus_...} default; defaults.sh source
in env/all.sh already guarantees the value.
- scripts/tests/dns.sh: source defaults.sh so callers only need to set
INFINITO_DISTRO (the implicit dependency was previously a footgun).
- scripts/system/tls/trust/{wsl2,linux}.sh: same — source defaults.sh,
drop the ${INFINITO_CONTAINER:-infinito_nexus_debian} fallback in
linux.sh.
- .github/workflows/test-dns.yml: drop redundant
INFINITO_CONTAINER: infinito_nexus_${{ matrix.distro }} from per-step
env blocks (3 sites). The env now flows from INFINITO_DISTRO via the
defaults.sh sourced inside the scripts.
- tests/unit/cli/deploy/development/test_matrix_deploy.py: replace the
module-level os.environ.setdefault('INFINITO_CONTAINER', 'infinito_nexus_arch')
shim — which itself encoded the formula — with a module-level patch of
resolve_container that returns an opaque test-fixture string. Drop the
now-unused distro="arch" field from the _args() Namespace fixture
(deploy.handler stopped consuming it after the --distro arg removal).
Documentation aligned:
- docs/contributing/environment/variables.md: rewrite the INFINITO_CONTAINER
row to spell out the SPOT contract — set automatically by defaults.sh,
never re-derived elsewhere, callers re-source defaults.sh after
changing INFINITO_DISTRO mid-process.
- docs/contributing/artefact/files/compose.yml.md: replace the
INFINITO_DISTRO row (which described the now-removed
`infinito_nexus_${INFINITO_DISTRO:-debian}` interpolation) with an
INFINITO_CONTAINER row noting the strict ${INFINITO_CONTAINER:?...} read.
Verified:
- Formula derivation: 1 production hit (the SPOT in defaults.sh:18).
- INFINITO_CONTAINER fallback patterns (:-/:=): 0.
- --distro CLI arg refs: 0 (one comment explaining the removal).
- make test green: 1553 unit + 168 integration tests, no skipped suites.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the agreed-upon move of run_after: and lifecycle: from
meta/main.yml.galaxy_info into meta/services.yml.<primary_entity> alongside
the rest of each role's service metadata. Primary entity is derived via
get_entity_name(role_name); multi-entity roles (bluesky/matrix) get a
dedicated metadata-holder entry next to their compose entities. Lifecycle
allow-list anchored on values actually present in roles today (planned,
pre-alpha, alpha, beta) plus stable/deprecated forward-compat. A small
helper (utils/roles/meta_lookup.py) hides the entity derivation from
callers and degrades gracefully on missing meta/services.yml. Depends on
both req-008 and req-009.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the agreed-upon migration of group_vars/all/{08_networks,09_ports}.yml
into per-role meta/server.yml.networks and meta/services.yml.<entity>.ports
with the always-category-keyed shape, the inter/local/public split, and the
top-level port: -> ports.inter rename + int-cast. Defines PORT_BANDS in
group_vars/all/08_networks.yml as the single source for suggester and lint,
specifies cli meta {ports,networks} suggest helpers, and pins the lint rules
including the flat-set host-bound port collision check that excludes inter.
Depends on req-008.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Captures the agreed-upon migration of every role's metadata to single-
responsibility files under meta/: meta/schema.yml (with new optional default:
field), meta/users.yml, meta/server.yml, meta/rbac.yml, meta/services.yml
(file-root keyed by entity_name; long-tail config inlined), and
meta/volumes.yml. Drops the compose. prefix from all consumer paths and
enumerates the lookup plugins to update.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The 934-line `utils/cache/data.py` monolith mixed cross-cutting helpers
(constants, yaml-loaders, deep-merge, signatures, render machinery)
with three independent domain caches (applications, users, domains).
Per the KISS/SPOT/DRY guidelines, split it along domain boundaries so
each file has a single, well-scoped responsibility:
utils/cache/__init__.py — package; owns _reset_cache_for_tests()
orchestrator
utils/cache/base.py — cross-cutting helpers: constants,
yaml-loaders, _deep_merge, cache-key +
content-fingerprint signatures, the
cross-domain re-entry guard, and the
templar-render machinery (the only
ansible-coupled symbol, lazy-imports
its dependency)
utils/cache/applications.py — variants + defaults +
get_merged_applications. Strictly
ansible-free at import time so the
GitHub Actions runner-host CLI path
(cli.deploy.development.init →
plan_dev_inventory_matrix →
get_variants) keeps working without
ansible installed
utils/cache/users.py — user defs, token store hydration,
alias materialization,
get_user_defaults, get_merged_users
utils/cache/domains.py — get_merged_domains, derived from the
merged applications view
utils/cache/data.py — DELETED. Façade is gone; the package
exports nothing the domain modules
don't already
Backward-compat: the facade was deleted (per operator request) and all
36+ consumers — 18 plugins, 2 CLI entry points, 9 test files, 4 doc
files, plus a role-internal lookup_plugin — were rewired to import
directly from the owning domain module. `_reset_cache_for_tests` is
now exposed at `from utils.cache import _reset_cache_for_tests`.
Tests added (per-module, focused):
tests/unit/utils/cache/test_base.py (31 cases)
tests/unit/utils/cache/test_applications.py ( 9 cases)
tests/unit/utils/cache/test_users_module.py (21 cases; suffix
disambiguates from
the existing
tests/unit/plugins/
lookup/test_users.py)
tests/unit/utils/cache/test_domains.py ( 6 cases)
The ansible-block invariants from CI runs 24934007615 and 24935979190
are now pinned via subprocess isolation: each ansible-less import test
spawns a fresh `python3 -c '…'` with a meta_path filter denying every
`ansible*` import and asserts the import + invocation succeed in that
clean process. The earlier in-process sys.modules-manipulation shims
fragmented module identity and broke under Python 3.13 inside the
infinito container; subprocess isolation has neither problem.
Doc-style audit (per docs/contributing/documentation.md):
- Removed em-dash from utils/cache/README.md (split into two
sentences)
- Updated stale `utils.runtime_data` reference in
docs/contributing/design/matrix-deploy.md
- Removed two SPOT self-declarations
(docs/agents/files/group_vars/applications.md and
docs/contributing/artefact/files/plugins/lookup/users.md) per
the documentation rule that SPOT status is conferred by
cross-links, not by the page asserting it about itself
Empirical proof:
- All four domain modules import without ansible (subprocess
isolation tests)
- get_variants(roles_dir=ROLES_DIR) returns 99 apps in the
ansible-blocked subprocess
- 142 cache tests + 1085 wider-suite tests pass; the only
remaining failures (test_host_vars free-function fixture bug,
test_runtime_performance timing flake) are pre-existing and
unrelated to this refactor (verified via `git stash`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both in-process cache layers — the cached YAML loader and the cached
runtime-data getters — now live next to each other under
`utils/cache/`. The package gains a README that documents the
"orthogonal layers, no third cache" rule so future contributors hit
the right primitive on the first look.
Moves:
- `utils/yaml_cache.py` -> `utils/cache/yaml.py`
- `utils/runtime_data.py` -> `utils/cache/data.py`
- `tests/unit/utils/test_yaml_cache.py` -> `tests/unit/utils/cache/test_yaml.py`
- `tests/unit/utils/test_runtime_data.py` -> `tests/unit/utils/cache/test_data.py`
- `tests/unit/utils/test_variants.py` -> `tests/unit/utils/cache/test_variants.py`
New:
- `utils/cache/__init__.py` (empty per project convention)
- `utils/cache/README.md` describing the two layers, when to reach for
which, lifetime, test-only helpers, and design guidelines (RFC 2119
keywords, emoji-after-headline, no self-SPOT declaration, no
slash-in-link-text).
- `tests/unit/utils/cache/__init__.py`
Mechanical updates:
- All imports across cli/, plugins/lookup/, tests/, and roles/ rewritten
from `utils.yaml_cache` -> `utils.cache.yaml` and
`utils.runtime_data` -> `utils.cache.data` (40+ files).
- `REPO_ROOT` parents-index in the moved tests bumped from `parents[3]`
to `parents[4]` to reflect the deeper nesting.
- Six broken markdown links in docs/ updated to the new module paths
(test_markdown_links lint test stays green).
- Two test files that used the `from utils import runtime_data` form
now use `from utils.cache import data as runtime_data` so the local
alias remains.
Tests: 848 passed across unit + lint suites (1 pre-existing collection
error in test_host_vars.py unrelated to this change).
The Playwright E2E gate previously fired through `MODE_CI`, a boolean
that OR-ed three env-var markers (`GITHUB_ACTIONS`, `ACT`,
`INFINITO_MAKE_DEPLOY`) AND-ed with an `INFINITO_SKIP_E2E` opt-out.
That mechanism duplicated information `RUNTIME` already carries:
- `make deploy-*` already bakes `RUNTIME=dev` into the inventory at
init time (`RUNTIME_VARS_JSON='{"RUNTIME":"dev"}'`).
- CI runners (act / GitHub Actions) detect `RUNTIME=act|github` from
their canonical env markers via `cli.meta.runtime.detect_runtime`
during init.
- A bare `ansible-playbook` from a production shell stays at the
default `RUNTIME=host`.
Switching the gate to `RUNTIME in ['dev', 'act', 'github']` therefore
covers every case `MODE_CI` covered, AND folds the gate into the
single existing runtime SoT rather than maintaining a parallel one.
Changes:
- `tasks/stages/02_server.yml` E2E include now keys on `RUNTIME`.
- `group_vars/all/01_modes.yml` drops the `MODE_CI` block; a short
note explains the retirement.
- `scripts/meta/env/ci.sh` deleted (it only existed to export
`INFINITO_MAKE_DEPLOY=1`).
- The 5 `scripts/tests/deploy/local/deploy/*.sh` entry-points stop
sourcing `ci.sh`.
- `cli/deploy/development/deploy.py` no longer forwards the four
retired env markers into the container; the runtime decision is
already serialised into the inventory the deploy stage consumes.
- `scripts/tests/deploy/local/deploy/reuse-kept-app.sh` drops the
same four `-e` flags from its `docker exec`.
- `docs/requirements/006-playwright-service-gated-tests.md` gets a
prepended "MODE_CI retired" update note; the historical sections
remain for context but the dead `ci.sh` link target was reduced to
inline code.
Behaviour: any runtime where E2E was fired before still fires it now,
and the previous opt-outs have a single replacement: pass
`RUNTIME=host` (or `--vars '{"RUNTIME":"host"}'`) to skip Playwright on
a deploy that would otherwise have run it.
Introduces per-role `meta/variants.yml` for declaring multi-variant
deploys (e.g. WordPress Single-Site vs. Multisite) and reshapes the
development deploy CLI to materialise one inventory folder per variant
and iterate them with co-located sync+async passes.
Why:
- The previous single-inventory + Pass-1-then-Pass-2 split deployed each
variant twice and ran the async pass against a host left in a
different variant's state, masking real bugs.
- Inventory and runtime concerns were tangled: variant selection lived
as a runtime extra-var and required cross-stage coordination.
How:
- `roles/<role>/meta/variants.yml` (file format docs at
docs/contributing/artefact/files/role/variants.md): YAML list of
deep-merge overrides on top of `config/main.yml`. Empty entry `{}` is
variant 0 (= legacy payload). 6 roles get their dev overrides moved
here from inventories/dev.yml; WordPress declares 2 variants.
- `utils/runtime_data.py` gains `_build_variants` / `get_variants` and
drops the retired `_active_variants` runtime selector. Loader always
returns variant 0 as the default; non-zero variants reach
`get_merged_applications` as inventory-level `applications.<app>`
overrides baked at init time.
- New `cli/deploy/development/inventory.py`: `DevInventorySpec`,
`plan_dev_inventory_matrix` (pure planner shared between init and
deploy), `build_dev_inventory_matrix` (materialises one folder per
round at `<dir>-<R>`, or `<dir>` when single-variant). Single-folder
paths preserve full backward compat.
- `cli/deploy/development/deploy.py`: matrix loop with per-round
cleanup (only apps whose variant changed AND that are about to
re-deploy), `--variant <idx>` (or `VARIANT=` env) to pin a single
round, and `--full-cycle` (or `FULL_CYCLE=true`) which interleaves
sync + async per round so the async re-deploy targets exactly the
host state the matching sync deploy just produced.
- `cli/deploy/development/exec.py` gains repeatable `--env KEY=VAL` so
bash callers can inject per-run context into in-container helpers
without inlining heredocs.
- `scripts/meta/env/inventory.sh` exports `INVENTORY_VARS_FILE`
(Python mirror in `cli.deploy.development.common.DEV_INVENTORY_VARS_FILE`;
test_inventory_vars_file_spot.py locks them against drift).
- `make deploy-fresh-purged-apps` and `scripts/tests/deploy/ci/dedicated.sh`
switch to a single init + single deploy `--full-cycle`. The
`make deploy-reuse-*` bash scripts honour `VARIANT=` by suffixing
`INVENTORY_DIR`/`INVENTORY_FILE`.
- The 5 `scripts/tests/deploy/local/deploy/*.sh` scripts now delegate
their in-container heredocs to dedicated helpers in
`scripts/tests/deploy/local/utils/`, removing the bash-in-bash
quoting hell.
Tests:
- `tests/unit/utils/test_variants.py` (renamed from
`test_application_variants.py`): loader edge cases, variant-zero
default, retired-`_active_variants` guard.
- `tests/unit/cli/deploy/development/test_inventory.py` (new): planner,
variant baking, builder, `filter_plan_to_variant`, `DevInventorySpec`.
- `tests/unit/cli/deploy/development/test_matrix_deploy.py` (new):
per-round deploy filter, cleanup decisions, `--variant` pinning,
`--full-cycle` per-round interleave.
- `tests/unit/cli/deploy/development/test_inventory_vars_file_spot.py`
(new): drift guard for the bash <-> Python literal pair.
- `tests/unit/cli/deploy/development/test_exec.py` extended for `--env`.
Docs:
- `docs/contributing/artefact/files/role/variants.md` (new): file format.
- `docs/contributing/design/matrix-deploy.md` (new): runtime mechanism
(folder layout, --variant, --full-cycle, cleanup logic).
- `docs/contributing/actions/deploy.md` (new): contributor dev-deploy
flow (make targets, --variant pinning).
- `docs/administration/deploy.md` slimmed to admin-only (was mixed
admin+dev), cross-links to the new contributor doc.
- `docs/agents/action/iteration/role.md` keeps only agent-iteration
recipes; mechanism explanation moved out and linked.
- `docs/contributing/documentation.md` adds the "no self-SPOT
declaration" rule; `applications.md` renamed its SoT header for
consistency.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop the `DEPLOY_CI_EXPORTS := INFINITO_MAKE_DEPLOY=1` declaration
from the Makefile and the per-recipe `$(DEPLOY_CI_EXPORTS)` prefix
from every `deploy-*` target. The marker now lives in a dedicated
on-demand env module `scripts/meta/env/ci.sh` and is sourced by every
deploy entry-point under `scripts/tests/deploy/local/deploy/*.sh`,
which keeps the export aligned with the rest of the project's env
layer (`scripts/meta/env/all.sh` and friends).
`ci.sh` is intentionally NOT composed into `all.sh`: the marker
flips `MODE_CI` to `true` and MUST surface only when a deploy
actually runs, not for unrelated `make test*` / `make build*`
recipes that share the same `BASH_ENV`.
`docs/requirements/006-playwright-service-gated-tests.md` and the
`scripts/meta/env/README.md` document the new SPOT and the
always-on / on-demand distinction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The tool now pins branch.main.pushRemote to origin, so main pushes go
upstream while other branches keep landing on the personal fork. The
layout table and the new Push Routing section reflect this and list
the three git-config keys the tool writes. Also tighten the sandbox
rule to MUST per the docs RFC-2119 style.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drop the project-local copy of scripts/git/sign-push.sh in favour of
the git-maintainer-tools CLIs (git-setup-remotes, git-sign-push),
which now ship as a standalone PyPI/GitHub package so the same helpers
can be reused across every maintainer repo without vendoring shell
scripts.
The package is declared in pyproject.toml's `dev` extras and lands on
$PATH via `make install-python-dev` (which already runs
`pip install -e .[dev]`). Consumers invoke the CLIs directly; there
are no make wrappers because doing so would just duplicate help text
the tool itself already prints.
Fallout:
* scripts/git/sign-push.sh deleted.
* Makefile: sign-push target removed (no replacement target).
* .claude/settings.json: deny rule for `make sign-push*` replaced
with direct `git-setup-remotes*` / `git-sign-push*` denies so the
sandbox blocks the real binaries, not just the make alias.
* CLAUDE.md: operator instruction points at the bare `git-sign-push`
CLI + `make install-python-dev` for setup.
* docs/contributing/artefact/git/remotes.md: new tool-focused page.
Describes the layout and the two CLIs; does not duplicate raw git
commands already covered by git-maintainer-tools.
* docs/contributing/artefact/git/README.md: trimmed per readme_md.md
(docs READMEs must not act as a file index).
* docs/contributing/tools/make.md: Git section now documents the
external CLIs and the install entry point, not a wrapper target.
- Remove every em-dash thought break (4 in 004, 24 in 006, 2 in
the WordPress role README) per the "no em-dashes" rule in
`docs/contributing/documentation.md`.
- Replace "is forbidden" with the RFC 2119 keyword "MUST NOT" in
005 and 006.
- Disambiguate the ambiguous `[rbac.md](...)` links in the role
README, the administration SPOT, and the design SPOT by giving
each link a descriptive label ("Administration RBAC" /
"RBAC design"), per the link-text rules in `documentation.md`.
- Move the emoji decorations on `roles/web-app-wordpress/tasks/
plugins/README.md` headings from before to after the heading
text and turn the U+2192 arrow into prose so the document
matches the heading-emoji rule.
- Update `docs/contributing/design/iam/rbac.md` so the LDAP
layout, Keycloak mapper, and `path_join` paragraphs reflect
the flat-CN + per-app mapper implementation that requirement
005 actually ships, instead of the hierarchical-OU vision the
page was originally drafted against.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Implements requirements 005, 006, and 007.
005: per-application Keycloak `group-ldap-mapper`s anchor each app's
role groups under `/roles/<application_id>/<role_name>` (and, for
tenant-aware apps, `/roles/<application_id>/<tenant>/<role_name>`).
The flat-CN LDAP layout keeps each CN globally unique so Keycloak's
group-tree resolver never collides; per-app mappers use the LDAP
`description` as the visible Keycloak group name to keep the path
free of redundant prefixes. Migration purges legacy pre-005 entries
and forces a Keycloak `logout-all` so stale tokens drop pre-005
group paths.
006: per-service `<SERVICE>_SERVICE_ENABLED` env flags and the
`service-gating.js` helper let role specs `skipUnlessServiceEnabled`
cleanly when a shared service is missing or `SERVICES_DISABLED` is
set; `MODE_CI` markers (`INFINITO_MAKE_DEPLOY`, `GITHUB_ACTIONS`,
`ACT`) propagate through the deploy CLI into the test stage.
007: WordPress->Discourse round-trip Playwright scenario gated on the
`web-app-discourse` deployment.
WordPress Playwright runs 5/5 green on the single-site flow; the
multisite + Discourse scenarios are correctly skipped when their
service prerequisites are absent. The `rbac_group_path` lookup is the
single sanctioned producer for OIDC group paths and is wired into
every consumer (yourls, lam, minio, openwebui, pgadmin, phpldapadmin,
phpmyadmin, prometheus, xwiki).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The tool now pins branch.main.pushRemote to origin, so main pushes go
upstream while other branches keep landing on the personal fork. The
layout table and the new Push Routing section reflect this and list
the three git-config keys the tool writes. Also tighten the sandbox
rule to MUST per the docs RFC-2119 style.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Drop the project-local copy of scripts/git/sign-push.sh in favour of
the git-maintainer-tools CLIs (git-setup-remotes, git-sign-push),
which now ship as a standalone PyPI/GitHub package so the same helpers
can be reused across every maintainer repo without vendoring shell
scripts.
The package is declared in pyproject.toml's `dev` extras and lands on
$PATH via `make install-python-dev` (which already runs
`pip install -e .[dev]`). Consumers invoke the CLIs directly; there
are no make wrappers because doing so would just duplicate help text
the tool itself already prints.
Fallout:
* scripts/git/sign-push.sh deleted.
* Makefile: sign-push target removed (no replacement target).
* .claude/settings.json: deny rule for `make sign-push*` replaced
with direct `git-setup-remotes*` / `git-sign-push*` denies so the
sandbox blocks the real binaries, not just the make alias.
* CLAUDE.md: operator instruction points at the bare `git-sign-push`
CLI + `make install-python-dev` for setup.
* docs/contributing/artefact/git/remotes.md: new tool-focused page.
Describes the layout and the two CLIs; does not duplicate raw git
commands already covered by git-maintainer-tools.
* docs/contributing/artefact/git/README.md: trimmed per readme_md.md
(docs READMEs must not act as a file index).
* docs/contributing/tools/make.md: Git section now documents the
external CLIs and the install entry point, not a wrapper target.
Add three follow-up requirements that extend the work landed in 004:
- 005 WordPress Multisite support + LDAP per-application OU refactor:
introduces opt-in rbac.tenancy schema, migrates every rbac-carrying
role onto the ou=<application_id>,ou=roles,... hierarchy, adds a
central rbac_group_path lookup plugin as the only sanctioned way to
compute OIDC group paths, mandates one-time LDAP rewrite plus a
Keycloak logout-all to flush stale tokens, and specifies the
WordPress mu-plugin contract for per-site roles and grant_super_admin
semantics.
- 006 Service-gated Playwright tests: per-service <NAME>_SERVICE_ENABLED
env flags, shared helper at roles/test-e2e-playwright/files/
service-gating.js with hard-fail on unknown services, test.skip() with
flag-named reason, explicit per-role refactor checkboxes for every
existing spec, and a new MODE_CI variable (GitHub Actions + nektos/act
+ make-driven deploy) replacing the accidental DOCKER_IN_CONTAINER
proxy at the test-e2e-playwright include site.
- 007 WordPress -> Discourse post round-trip: Playwright scenario that
publishes a WP post and asserts the corresponding Discourse topic
exists, gated on DISCOURSE_SERVICE_ENABLED, with teardown that covers
draft and partial states and an API key scoped for topic delete.
Also update the role Playwright spec guide to carry the Option B
service-gating rules (per-service boolean flags, helper-only reads,
hard-fail on unknown services) so contributor and requirement SPOTs
stay in sync.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Defines the end-to-end contract for auto-provisioning LDAP groups for
every role that declares an rbac: block, gated on group_names, plus
the OIDC flow from Keycloak into WordPress and the Playwright
verification that exercises the full pipeline for three exemplary
WordPress roles (subscriber, editor, administrator).
Key decisions captured in the criteria:
- Scope covers every role under roles/ with an rbac.roles block, not
just web-app-* — svc-*, web-svc-* etc. included when they declare
rbac.
- Gating: a role's groups are provisioned only when application_id is
in group_names on the current host.
- Naming is the flat pattern <application_id>-<role_name>; full DN
stays under the existing ou=roles container (no new OU introduced),
which preserves every existing <application_id>-administrator DN
exactly so current bind/search filters keep working.
- Administrator is still implicitly added even when not listed in
rbac.roles, matching today's build_ldap_role_entries behavior.
- Keycloak sync + OIDC groups claim must deliver WordPress role
mapping without any alternative path (no direct LDAP bind from WP);
if the current OIDC plugin config doesn't deliver that, it is fixed
in the same work item.
- Idempotency is required at two levels: re-running the provisioning
is a no-op, and the Playwright test preserves biber's starting
group membership across a full run.
- The "serial" constraint applies only to the test: the three role
sequences (subscriber → editor → administrator) run one after
another so a regression in one mapping does not mask another.
- Iteration follows the Role Loop + Playwright Spec Loop with
SERVICES_DISABLED="matomo,email".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Rename the third flag returned by lookup('service', ...) from 'needed'
to 'required' across code, tests and design docs. Same semantics: True
when some deployed app has the service with both enabled: true AND
shared: true, directly or transitively through its own enabled service
dependencies.
The new name reads as a contract ("this service is required by a real
consumer") rather than a suggestion ("something needs it"), which
matches the stricter-than-enabled semantic the flag already carries.
Touchpoints:
- plugins/lookup/service.py: _is_service_needed → _is_service_required,
any_needed var, dict key in _compute_flags, docstring rationale.
- roles/sys-utils-service-loader/tasks/main.yml: caller switched to
.required.
- roles/web-app-wordpress/config/main.yml + tasks/plugins/wp-discourse.yml:
wp-discourse plugin gate uses lookup('service', 'discourse').required;
compose.services.discourse declared as shared-service dependency so
the service loader auto-loads web-app-discourse when inventory sets
enabled: true. The legacy include_tasks: utils/load_app.yml in
plugins/wp-discourse.yml is removed — the service loader now handles
it.
- tests/unit/plugins/lookup/test_service.py: 25 assertions and 6 test
method names renamed (test_needed_* → test_required_*,
test_transitive_needed_resolved,
test_direct_provider_is_not_needed_without_shared_flag).
- tests/integration/services/test_transitive_dependencies.py: one
assertion renamed.
- docs/contributing/design/services/base.md: Lookup-Plugins section
replaces the 'needed' bullet with an expanded 'required' entry that
documents when consumers should gate on it and why plain enabled is
not enough.
Also moves wordpress-app-specific keys (title, max_upload_size,
plugins) under compose.services.wordpress to match the convention
the other role keys already follow, and updates vars/main.yml lookup
paths accordingly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Second-wave rollout of tests/utils/fs:
- 6 tests switched to the shared content cache via read_text() (replacing
raw open()/Path.read_text()): roles/includes/test_existence,
roles/dependencies/test_unnecessary, vars/test_facts_not_overridden,
services/test_sys_requires_id, lint/ansible/roles/
test_service_core_first_task_run_once, lint/ansible/roles/
test_compose_resource_limits.
- 4 unit-lookup tests moved their fixture root from <repo>/.tmp_test_* to
tempfile.TemporaryDirectory() so transient files land in /tmp (matching
the AGENTS.md rule). Manual recursive-delete teardowns removed —
TemporaryDirectory.cleanup() covers that path safely.
- filters/test_usage.py: algorithmic inversion. Old loop was
O(N_filters × M_files × 4_regex); new scan walks files once, applies two
master-alternation regexes (bare `| name` and `{% filter name %}`) plus
one combined regex over all Python callables, and records hits per filter
via the capture group. The two dropped Jinja patterns were strict
subsets of the bare pattern, so the hit set is identical — only the
complexity changes (3.6s → 0.2s, ~18x).
- docs/agents/action/testing.md: new "Filesystem Scans — Use the Shared
Cache" section codifies the rule so future agent-written tests route
through tests.utils.fs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Spell out the verbatim env-var form (`SERVICES_DISABLED="matomo,email"`)
and call out that the value is a comma-separated provider-key list — not
a glob or a `web-app-*.compose.services.*` path. Tighten the side-effect
note so the auto-removed provider roles are explicitly excluded from
`APPS`, not just from "the deploy".
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Restructure the matomo/email provider disable rule into a labeled
key/value block (WHEN/ACTION/DEFAULT/ON YES/ON NO/SIDE EFFECT/PERSIST)
so agents can parse it deterministically. Switch the disable mechanism
from per-app `compose.services.*` flags to appending `matomo,email` to
`SERVICES_DISABLED` on the deploy command.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- playwright.spec.js: add "biber logs into nextcloud via OIDC and logs
out" scenario covering the non-administrator persona required by the
MUSTs; parameterize loginToStandaloneNextcloud on username/password,
add a logoutStandaloneNextcloud helper, and wrap biber's first login
in a retry so the LDAP first-login-provisioning delay does not flake.
- playwright.env.j2: expose BIBER_USERNAME and BIBER_PASSWORD so the
spec can read biber credentials from the rendered env.
- docs/contributing/.../playwright.specs.js.md: codify the MUST that
existing scenarios may not be deleted while optimizing a spec.
- roles/web-app-nextcloud/docs/LDAP.md: document the biber first-login
caveat against fresh Nextcloud + LDAP deployments.
Validated via scripts/tests/e2e/rerun-spec.sh web-app-nextcloud:
3 passed (talk admin settings, dashboard to nextcloud login, biber).
Shorten the module name now that it backs more than just lookup plugins
(validator CLI, tests, dashboard role). Updates all 26 import sites and
adds a unit test covering the pure helpers.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces 15 per-shape Bash entries that existed as sandbox auto-allow
workarounds with a single Bash(*) wildcard. Gating responsibility moves
entirely to ask (mutating gh/docker/git verbs) and deny (structurally
destructive shapes). Documentation updated across settings.md,
security.md, and sandbox.md to reflect the new model; em-dash clause
separators removed throughout per documentation.md style rules.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Moves the SYSTEM_EMAIL_* resolution logic out of group_vars/all/03_email.yml
into a dedicated Ansible lookup plugin, so every role template reads outgoing
mail settings via lookup('email') (global) or lookup('email', application_id)
(per-app). Per-app compose.services.email is merged on top of the resolved
defaults; SYSTEM_EMAIL_* variables remain usable as an inventory escape hatch.
Also splits the email docs: administrator-facing configuration stays under
docs/administration/configuration/, plugin contract and extension rules move
to docs/contributing/design/services/email.md. services.md is relocated to
services/base.md so the services/ subtree can host per-service design pages.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move installation.md and rename administration.md to deploy.md under
docs/administration/, delete docs/guides/, and add a scope-only
README.md per the docs-README rules. Update README.md, SUPPORT.md,
and scripts/tests/environment/README.md cross-links.
Refresh roles/web-app-decidim/README.md to match the current OIDC
patch set (decidim-core omniauth.rb + omniauth_helper.rb only, no
secrets.yml on Rails 7.2, ENV-prefixed credential names).
Replaces narrow `Bash(PYTHONPATH=* python *)` with `Bash(*=* python* *)`
so test invocations carrying multiple env-var assignments
(e.g. `PYTHON=python3 PYTHONPATH=. python3 -m unittest ...`) and
`python3` binaries no longer trip the auto-allow heuristic. Sandbox
remains the primary containment layer, so widening the allow shape
adds no new blast radius.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move flat tests/integration/test_*.py files into thematic subfolders
(action_plugins, application_id, categories, compose, domains, filters,
handlers, jinja, linting, oauth2_oidc, passwords, ports, roles, run_once,
services, workflows), strip redundant prefixes from filenames, and
adjust path-depth references by +1 level. Add __init__.py and a scope-
focused README.md per cluster so the directory purpose is discoverable
without duplicating the integration.md SPOT. Update design/services.md
to point at the new test locations.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Resolve merge conflict in web-app-peertube/vars/main.yml; migrate
remaining `applications |` and `users.*` references to
`lookup('applications')` / `lookup('users', ...)` in web-app-odoo and
web-app-decidim (compose, env, playwright, setup tasks).
- web-app-peertube: split OIDC tasks into oidc/{enable,disable}.yml,
add install_mem_limit filter + PEERTUBE_MEM_LIMIT /
PEERTUBE_OIDC_PLUGIN_INSTALL_MEM_LIMIT vars to raise the cgroup ceiling
during the one-off `npm run plugin:install` exec (prevents OOM-SIGKILL
that was surfacing as "Container is restarting"); add
upsert-oidc-plugin.sql.j2 and diagnostics.sh.
- CI: test-deploy-{local,server,universal,workstation}.yml tweaks plus
scripts/github/{dump_runner_state,enlarge_swap}.sh for constrained
runners.
- cli/meta/applications/ressources: new CLI entry with unit tests.
- Misc config/docs updates across dashboard, mailu, mattermost, asset,
cdn, file roles and contributing docs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Plugin install via `container exec` shares the cgroup with the long-running
peertube node process. Combined RSS was tripping the 8g mem_limit and
SIGKILLing peertube, which surfaced as "Container is restarting" on
subsequent exec attempts.
- Temp raise mem_limit + unlimited swap for the install window, restore
the compose-defined value on the happy path only
- Derive the bumped ceiling from PEERTUBE_MEM_LIMIT + install heap cap
via a new role-local `install_mem_limit` filter (overhead=4 covers
native RSS and spawned npm/node-gyp workers)
- Extract the SQL upsert into templates/upsert-oidc-plugin.sql.j2
- Move enable/disable OIDC tasks under tasks/oidc/ and rename the
diagnostics script
- Document the extract-code convention in docs/contributing/tools/ansible.md
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Swap size was free(/)-25, which took ~99 GB on the 145 GB root fs.
Remaining ~25 GB then ran out mid-deploy during Playwright image
pulls (ENOSPC). Replace with a fixed 16 GB cap and use mountpoint
to detect a real /mnt partition instead of [ -d /mnt ], which
matched the empty placeholder dir on single-partition runners.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Introduce scripts/github/enlarge_swap.sh that consumes all remaining
space on / minus a 25 GB buffer (Docker layers, checkout, runtime
state), falling back to /mnt when / is too small. Replaces the
default ~4 GB runner swap so transient memory spikes (e.g. peertube
plugin install, issue #162) don't trip the host OOM-killer on the
16 GB GitHub-hosted runner.
Wire the script into all four deploy test workflows (local, universal,
server, workstation) and flip free-disk-space to the aggressive config
(tool-cache, large-packages, docker-images, swap-storage all true):
the deploy actually runs inside the infinito container, so host-side
toolchains, language runtimes, and the default Docker layer cache are
dead weight. Update the workflow SPOT doc accordingly.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
