`get_user_defaults()` builds every user's email/lastname/description with
a hardcoded `primary_domain="{{ DOMAIN_PRIMARY }}"` Jinja placeholder and
relies on `_render_with_templar` to resolve it later. In some Ansible
templating contexts (notably swarm-mode tasks delegated to STACK_HOST)
the templar can't fully resolve `{{ DOMAIN_PRIMARY }}` against the
inventory expression `{{ lookup('env', 'INFINITO_DOMAIN') }}`, so the
raw placeholder gets cached on the very first miss and reused across
every subsequent `lookup('users', ...)` call in that play. The leak
surfaced downstream as
`MATOMO_ADMIN_EMAIL=administrator@{{ DOMAIN_PRIMARY }}` in the rendered
matomo bootstrap.env, which `docker stack deploy` then refused with
`template: expansion:1: function "DOMAIN_PRIMARY" not defined`.
Two robustness fixes in `get_merged_users` so the cache never holds a
half-resolved placeholder:
1) Pull `DOMAIN_PRIMARY` / `SYSTEM_EMAIL_DOMAIN` from
`templar.available_variables` when missing from the explicit
`variables=` kwarg, so the cache key + downstream resolution see the
same domain regardless of who invokes the lookup.
2) Walk the materialized users dict and replace literal
`{{ DOMAIN_PRIMARY }}` substrings with the resolved primary-domain
value (computed via the existing `_normalize_domain_candidate`-style
logic) *before* handing the dict to `_render_with_templar`. Deterministic
in pure Python, no dependency on templar succeeding for that one
placeholder.
Validated in act-swarm-zombie t11: the exact `MATOMO_ADMIN_EMAIL=...
{{ DOMAIN_PRIMARY }}` error that broke t10 at line 13172 is gone, and
the run progressed past matomo-bootstrap into the post-deploy phase.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Refactors four roles that previously used `# nocheck: compose-chdir-in-task` /
`compose-verb-in-task` / `container-exec-resolver` markers to follow the
canonical swarm-aware pattern:
- web-app-bigbluebutton: adds BBB_POSTGRES_CONTAINER_ADDRESS /
BBB_GREENLIGHT_CONTAINER_ADDRESS vars; 02a_postgres_password_sync.yml +
03_administrator.yml use `container exec -i {{ ..._CONTAINER_ADDRESS | quote }}`;
02_compose.yml's compose-build-phase tasks (generate-compose +
prebuild etherpad) move into the new
02b_bootstrap_compose_project.yml (>3 entries cannot live in a block)
under a single `when: DEPLOYMENT_MODE \!= "swarm"` include guard.
- web-app-baserow: BASEROW_APP_CONTAINER_ADDRESS var + 02_manager_ops.yml
converted to container exec + container restart canonical pattern.
- web-app-matomo: MATOMO_APP_CONTAINER_ADDRESS var; 07_patch_database.yml
swaps MATOMO_LOCAL_CID for the proper resolver-backed address.
- web-app-matrix: 01_docker.yml's two consecutive `compose pull` /
`compose up` tasks extract into 03_compose_pull_and_up.yml (with the new
`# include-gated: when: DEPLOYMENT_MODE \!= "swarm"` header marker) and
the previous 03_webserver.yml renames to 04_webserver.yml to keep the
numeric file sequence contiguous.
Supporting lint infrastructure changes so file-level include-gated
sub-task lists survive the swarm-compat lints:
- utils/annotations/task_gate.py: `is_task_compose_only_gated` now walks
up nested parent blocks (a single `when: DEPLOYMENT_MODE \!= 'swarm'`
on `- block:` exempts every child task underneath) and a new
`is_file_compose_only_by_header` recognises files declaring
`# include-gated: when: DEPLOYMENT_MODE \!= "swarm"` in their first
five lines as compose-only-by-include.
- tests/lint/ansible/shell/test_no_compose_chdir_in_shell_tasks.py and
test_no_compose_verb_in_shell_tasks.py call the header helper before
the per-line scan, so include-only sub-files are exempt.
- tests/unit/utils/annotations/test_task_gate.py covers both new
behaviours (recursive parent walk plus file header) end-to-end.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Migrate every role's meta/volumes.yml from ad-hoc dict-of-named-volumes
to a canonical dict-of-dicts schema where the YAML key is the semantic
short name and the entry body carries type / name / source / mounts /
nfs / mode / read_only. The optional 'name:' field is the docker volume
name (defaults to the key); the previous 'docker_name:' field is gone.
What's in this commit (210 paths):
Infrastructure (23)
- New filter plugins/filter/{compose,container}_volumes.py — render the
top-level volumes:/configs:/secrets: section and per-service mount
blocks from the canonical meta. container_volumes uses the SEMANTIC
name as per-service mount source so it resolves against compose_volumes'
top-level dict key (docker_name as source produced 'undefined volume'
on swarm).
- New lookup plugins/lookup/{compose,container,volume}_volumes.py.
volume(application_id, name) returns a normalised single entry;
volume(application_id) returns the whole canonical dict.
- utils/cache/applications.py keeps a sibling registry
_CANONICAL_VOLUMES_BY_ROLE outside the merged applications payload so
raw Jinja source: strings never enter the templar walk (root-cause of
the 200x cache-miss + recursive get_merged_applications storm).
- utils/roles/applications/mounts.py — schema validator + normaliser;
list-shape is rejected.
- plugins/lookup/nginx.py: _volume_path_by_name reads canonical directly.
- scripts/tests/deploy/swarm/_context.sh: NFS_VOLUMES extraction handles
both legacy dict and canonical dict shapes.
- roles/svc-prx-openresty/handlers/restart_and_validate.yml: gate
compose-only validate+reload behind a single when-block; swarm uses
service-update instead (bind-mounts of a FILE pin source inode at
task-create time, so atomic write-tmp+rename leaves the running task
on a deleted inode).
- roles/sys-svc-compose/tasks/utils/nfs_prep.yml: switch to
lookup('volume', application_id) | dict2items so NFS subdir
pre-creation works under the new cache split.
Tests (12)
- New strict lints flagging inline mount sections in compose templates,
single-file bind mounts, and legacy lookup('config', ..., 'volumes.X.Y')
call sites — all volume-mount surfaces MUST flow through container_volumes
/ compose_volumes / volume lookups.
- Integration: test_volumes_compose_chain + test_volume_constants_are_lookups
accept both the legacy and the new lookup('volume', app, 'X').name forms
during the migration window.
- Unit tests for the new lookup, filter, mounts helper, plus updated
nginx lookup test.
- Prometheus role-wiring test reads canonical meta for the
alert_rules.yml bind mount.
Role migrations (175)
- 62 meta/volumes.yml converted to dict-of-dicts canonical shape
- 63 templates/compose*.yml.j2 replace inline volumes:/configs:/secrets:
with lookup('container_volumes', application_id, '<service>') |
indent(N) per service + a single trailing lookup('compose_volumes',
application_id) for the top-level block
- 50 vars/main.yml drop dead volume constants and refactor 23 remaining
legacy lookup('config', ..., 'volumes.X.Y') to lookup('volume',
application_id, 'X').Y across 9 roles
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds `act-swarm-zombie app=<id>` which runs the swarm matrix-app test and leaves
the DinD cluster + lab network alive afterwards (INFINITO_KEEP_SWARM_NODES=true
forwarded via the new ACT_ENV passthrough in workflow.sh) so failures can be
post-mortem inspected with `make act-swarm-exec node=… cmd=…`,
`make act-swarm-shell node=…`, and finally released with `make act-swarm-down`.
Consolidates the swarm-test topology (manager / worker / nfs-server names, lab
network, NFS export base) into scripts/tests/deploy/swarm/topology.sh as the
single source of truth: 13_cleanup.sh and shell_node.sh source it directly,
the GitHub Actions workflow ingests it into $GITHUB_ENV via a setup step,
replacing the duplicate env: block entries.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The swarm-compat lints (`compose-chdir-in-task`, `compose-verb-in-task`,
`docker-compose-raw-call`) currently flag every offending line and only
accept per-line / file-head `# nocheck:` markers. That is noisy for
two patterns that are objectively safe:
1. Tasks gated by `when: DEPLOYMENT_MODE \!= 'swarm'` (or
`== 'compose'`) never reach a swarm host - the compose verbs inside
are legitimate.
2. YAML comment lines, handler-trigger list items, and human-readable
metadata fields (`name:`, `msg:`, `title:`, `label:`, `description:`)
contain the word `compose` as documentation, not as a shell call.
The lints now skip both classes automatically.
- utils/annotations/task_gate.py: new helper
`is_task_compose_only_gated(lines, idx)`. Walks the surrounding
task block via a line-based heuristic and matches `when:` against
the compose-only DEPLOYMENT_MODE expressions. Documented as
deliberately shallow (no YAML parse).
- tests/lint/ansible/shell/test_no_compose_chdir_in_shell_tasks.py and
tests/lint/ansible/shell/test_no_compose_verb_in_shell_tasks.py:
add a file-level `is_suppressed_in_head` skip, the
`is_task_compose_only_gated` skip, and -- for the verb lint --
three extra regex exclusions for notify-list items, YAML metadata
fields, and quoted list items.
- tests/lint/ansible/shell/test_no_docker_compose_raw_call.py: skip
YAML comment lines (bluesky and jenkins each had a `# … docker
compose build …` doc-comment that was being flagged as a real call).
- tests/unit/utils/annotations/test_task_gate.py: 21 cases covering
block-bounds + gate detection, including a `TestKnownLimitations`
class that pins the heuristic's blind spots (folded-scalar `when:`,
list-form `when:`, `in`/`not in`, unquoted literal, outer-`when:`
inheritance into `block:` children) so a future YAML-aware refactor
knows exactly which assertions to flip.
Tier-1 changes verified live in act-12 (mariadb stack converged, postgres
stack started, swarm-readiness gate fired correctly for docker/mariadb/
postgres/openresty).
* `plugins/filter/compose_volumes.py` + `plugins/lookup/compose_volumes.py`:
thread `dir_var_lib` through filter (required keyword, no default)
and lookup (reads from `vars_["DIR_VAR_LIB"]`, raises if missing). Replace
native NFS driver with a bind-mount from the host's already-NFS-mounted
`DIR_VAR_LIB` because native `type: nfs` re-handshakes portmap, which is
unreachable in nested DinD. Drop the FilterModule wrapper so the deprecated
pipe form keeps failing fast per the lookup-only SPOT contract.
* `tests/unit/plugins/{filter,lookup}/test_compose_volumes.py`:
thread `DIR_VAR_LIB` through every fixture (no default-fallbacks).
* `roles/sys-svc-compose/tasks/utils/post_deploy.yml`:
compose-mode db-health probe gated to `DEPLOYMENT_MODE \!= 'swarm'`; new
swarm-mode convergence gate runs the extracted `swarm_stack_ready.sh`
via `ansible.builtin.script` with `STACK` env, addressed by absolute
`playbook_dir`-rooted path so the include resolves from caller roles.
* `roles/sys-svc-compose/files/swarm_stack_ready.sh` (new): probe
`docker stack services` for a single `STACK` and exit non-zero until
every service's running/desired replica count matches.
* `roles/svc-db-mariadb/vars/main.yml`: new `MARIADB_CONTAINER_ADDRESS`
constant wraps the `container_address` lookup so callers reference one
SPOT instead of duplicating the lookup.
* `roles/svc-db-mariadb/tasks/01_core.yml` + `02_init.yml`:
switch root-password probes and bootstrap SQL calls to
`container exec -i {{ MARIADB_CONTAINER_ADDRESS }} sh -c 'mariadb -p"$MARIADB_ROOT_PASSWORD"'`.
Using the container's own `MARIADB_ROOT_PASSWORD` env var mirrors the
healthcheck and matches whatever quoting Docker swarm preserves from
the env_file, sidestepping the compose-vs-swarm env_file quote-strip
divergence.
* `roles/svc-db-postgres/files/set_postgres_superuser_password.sh`:
`eval "container=$container_raw"` so the swarm-mode lookup payload
`"$(/usr/bin/resolve-container-id <stack> <svc>)"` is re-evaluated by
bash and resolves to the live task id at exec time.
* `roles/svc-db-postgres/tasks/01_core.yml`: rename arg to use the new
swarm-aware `POSTGRES_CONTAINER_ADDRESS`.
* `utils/tests/swarm/extend_inventory.py`: when the seed app's primary
service declares `default_placement: manager`, do not add the app to
the worker group, so the swarm test inventory matches the
single-replica placement-pinned semantics.
OpenTalk's images (web-frontend, controller, recorder) live on
registry.opencode.de, which was absent from both _MIRRORABLE_REGISTRIES and
_ALL_REGISTRY_PREFIXES in utils/docker/image/discovery.py. So is_mirrorable_image
returned False, iter_role_images skipped them, the GHCR mirror never copied
them, and the deploy pulled direct from registry.opencode.de -> TLS handshake
timeout (deploy failed).
Add the registry to both sets so the single iter_role_images gate enrolls the
3 images: the mirror-sync copies them to GHCR and the resolver rewrites the
deploy to pull from the GHCR mirror. The prefix entry strips the host so the
mirror destination is well-formed (no doubled registry). Add a regression test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a database-style object-store abstraction: the objstore lookup resolves the active engine (seaweedfs|minio) per consumer and yields a uniform S3 payload, objstore_secret_key is auto-generated like database_password, sys-svc-objstore delegates per-consumer provisioning to the resolved provider's objstore_init entrypoint (lazy-loading the provider when it was not preloaded), and compose/network/volume plumbing supports both a shared central instance and an embedded storage-only instance per consumer.
Add the web-app-seaweedfs provider role: Filer and Master UIs behind the admin-only oauth2-proxy via a role-local nginx and proxy.conf.j2 (S3 API domain stays ungated, admin UIs deny with 403 when SSO is disabled), per-consumer S3 identities from s3.json, idempotent bucket provisioning via weed shell, a /24 on the central-provider subnet range, Playwright coverage for admin/biber/guest plus consumer bucket existence, and unit tests for the objstore lookup and the per-engine compose volumes.
Mark web-app-minio lifecycle: unsupported, move its mc client image into a dedicated services.client entry, align it with the objstore credential contract, and make its consumer provisioning idempotent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds a `swarm:` subcategory under `svc:` in roles/categories.yml so the
shared-cluster role is properly invokable, and renames the existing
`svc-swarm` role + inventory group to `svc-swarm-node` (with
`get_entity_name`-aligned primary key `node:` in meta/services.yml).
`svc-swarm-manager` stays as the manager-marker role, becoming invokable
via the same new category and re-keyed to `manager:`.
Consumers of the cluster-orchestration service (svc-registry-cache,
svc-registry-docker, svc-swarm-manager) now reference the service-key
`node:` (was `swarm:`), matching the renamed primary entry. The planner
empirically resolves `service_registry['node']` -> `svc-swarm-node` and
emits it into --include for stack-host deploys, so the CI failure on
`universal-svc-registry-docker` (`svc-swarm: not present in inventory`)
no longer reaches `ValidDeployId`.
Breaking change for operators: inventory groups named `svc-swarm:` must
be renamed to `svc-swarm-node:`. `DEPLOYMENT_MODE` reads
`groups['svc-swarm-node']`, so a stale `svc-swarm:` group silently falls
back to compose mode.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The swarm test rig provisions its inventory via derive_includes.py
walking each APP_ID's transitive meta-deps. svc-registry-docker is
not a meta-dep of any web app (intentionally — it would otherwise
spin up an unused registry container in compose-mode), but the
sys-svc-compose handler "swarm: retag built images for local
registry and push" (handlers/main.yml:116) requires the registry
to be reachable at swarm-mgr-01:5000 whenever an app builds its
own image (web-svc-simpleicons, etc). Without the include the
handler crashed with "dial tcp 192.168.244.3:5000: connection
refused".
Add svc-registry-docker to _EXPLICIT_INCLUDES alongside
svc-swarm-manager and svc-storage-nfs-client. The role already
carries default_placement: manager in meta/services.yml, so the
constructor auto-places swarm-mgr-01 in the svc-registry-docker
group and tasks/groups/svc-registry-roles.yml deploys it ahead of
the first app stack. Verified with:
APP_ID=web-svc-simpleicons python -m utils.tests.swarm.derive_includes
which now emits svc-registry-docker in the include list.
The previous commit's default.yml load turned MODE_DEBUG (and therefore
MODE_ASSERT) on for the swarm CI, which activates the DNS check in
tasks/stages/01_constructor.yml:12 that resolves
test.{{ DOMAIN_PRIMARY }}. The pipeline failed on all four hosts
(mgr-01, wrk-01, wrk-02, nfs-server) because:
1. write_hosts_entries did not emit test.<domain> at all.
2. 02_start_swarm_nodes wrote /etc/hosts only on the manager.
Add test.<domain> to the emitted set and write the resulting block to
mgr + both workers + nfs-server so the assertion succeeds everywhere
the play targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The inventory-vars-file SPOT drift guard
(tests/unit/cli/administration/deploy/development/test_inventory_vars_file_spot.py)
flagged the literal "inventories/development/default.yml" in the
docstring as a missed SPOT migration. Refer to "the development
inventory" in prose instead — the env var name (INFINITO_DOMAIN) is the
contract; the consuming file's path is not.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The previous DOMAIN_PRIMARY fallback was reading a non-existent env var
(DOMAIN_PRIMARY is an Ansible variable, not an env var) and defaulted to
infinito.localhost. In swarm-CI that produced /etc/hosts entries for the
wrong domain, so dashboard's "Wait for dashboard branding assets" task
got NXDOMAIN for file.infinito.example and exhausted its 60 retries.
INFINITO_DOMAIN is the SPOT (default.env:107, exported via
scripts/meta/env/load.sh which 02_start_swarm_nodes.sh sources). Read it
directly and let KeyError surface a misconfigured caller — matches
scripts/tests/dns.sh's "${INFINITO_DOMAIN:?...}" convention.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
svc-cache-registry -> svc-registry-cache
svc-docker-registry -> svc-registry-docker
svc-docker-swarm -> svc-swarm
svc-docker-swarm-manager -> svc-swarm-manager
categories.yml: drop svc.docker, add svc.registry. Deliberately no svc.swarm
category so get_entity_name('svc-swarm') stays 'swarm' instead of collapsing
to '' (which would break discover_role_services).
Entity keys updated to match the new svc.registry prefix:
services.cache-registry.* -> services.cache.*
services.registry.* -> services.docker.*
01_constructor: replace svc-docker bootstrap-loop entry with explicit
include_role of svc-swarm + bootstrap loop over [svc-registry, svc-storage].
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
inventories/development/swarm.yml carries only swarm-mode-specific
overrides and omits cross-cutting cred seeds (OIDC.CLIENT.SECRET,
biber, administrator password). Until now the swarm CI provision step
did not pass --vars-file, so default.yml was never merged into the
generated host_vars; the first deploy that preloaded web-app-mailu as
a shared email service then hit object of type 'dict' has no attribute
'SECRET' when sys-svc-compose rendered roles/web-app-mailu/templates/
env.j2. Affected jobs in CI run 27006276488: web-app-chess,
web-app-espocrm, web-app-funkwhale, web-app-prometheus.
Match the server-CI pattern (scripts/tests/deploy/local/reset/
inventory.sh and apps/initialize/all.sh both pass --vars-file
${INFINITO_INVENTORY_VARS_FILE}). 05_provision_inventory.sh now does
the same: the env loader sourced via _context.sh already exports
INFINITO_INVENTORY_VARS_FILE in the script shell, so the path
resolves through the SPOT.
write_extras.py re-reads default.yml users dict and patches the
runtime SSH key onto admin authorized_keys before emitting the
extras file. Without this, the deploy-time -e @swarm-nfs-extras.yml
would REPLACE the host_vars users entry (ansible later-wins on
top-level dicts) and wipe biber + the admin password the
--vars-file just baked in.
Two distinct breakages that let the workflow fail before ever reaching
the ansible deploy:
* `python3 -m utils.tests.swarm.*` (and friends) ran with the
actions-setup interpreter that has no project deps, so the imports
of `plugins.filter.canonical_domains_map` blew up on `from ansible.errors`.
The swarm preamble scripts now source `scripts/meta/env/load.sh` so
PATH picks up the venv `make install` populated, and the workflow itself
uses `install-force` (under `act --bind` the build/install.stamp is the
host's and would short-circuit the in-container install) plus pushes
`\${VENV}/bin` onto GITHUB_PATH for downstream steps.
* `ansible-playbook -e @file` replaces top-level dicts rather than
deep-merging them. The runtime-only extras file was dropping
`storage.nfs.{export_base,version}`, `storage.backend`, and
`swarm.{registry.port,network}` that the static
`inventories/development/swarm.yml` ships. Re-state those keys
in write_extras.py so the merge ends up with everything the roles need.
* tests/unit/cli/administration/deploy/swarm/test_closure.py — drop
the `parents[6]` + `nocheck: project-root-import` combo. The
file-path loader still bypasses the dedicated CLI's heavy package
__init__, but the path now comes from the canonical
`from utils import PROJECT_ROOT` so the project-root-import lint
passes without a per-line opt-out.
* utils/tests/swarm/write_extras.py — restructure the
`Path.read_text()` call onto a single line so the
`# nocheck: cache-read` annotation lands on the call site itself.
Autoformat had previously line-broken the chained `.strip()`
expression and pushed the marker onto a different line, making
the suppression invisible to the lint scanner.
* tests/integration/infrastructure/compose/test_compose_volumes_call_required.py —
the `test_no_template_uses_deprecated_pipe_form` walker used a raw
`roles_dir.rglob("templates/**/*.j2")`, which trips
`test_no_raw_project_walk` in the lint suite. Switch to the cached
`iter_project_files()` helper with a roles-prefix + `/templates/`
filter; behaviour is unchanged but the walk goes through the
shared SPOT.
* roles/svc-cache-registry/meta/services.yml — annotate the role
with `# nocheck: required-by-coverage`. svc-cache-registry is
intentionally opt-in via `default_placement: manager` plus the
swarm CLI closure, NOT via a downstream `required_by` chain, so
the lint's non-invokable-without-required_by check would otherwise
reject it.
`make autoformat` passes. `make test` is now green across lint /
test-external / test-integration / test-lint / test-unit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
CI failures from run 26987751848:
* `cache-read` (3 hits): annotate the `Path.read_text()` calls in
utils/tests/swarm/{write_extras,extend_inventory}.py — every one
reads a file the same script just wrote, so the cache helper would
serve stale content. Per `# nocheck: cache-read` carve-out.
* `project-root-import`: annotate the `parents[6]` in the swarm
closure test loader. The file-path loader bypasses the package
__init__ to avoid pulling in the heavy dedicated CLI imports.
* No-unittest-tests: tests in test_closure.py inherited only via
`_BaseClosureCase` (which itself extends `unittest.TestCase`), but
the AST lint walks bases literally and does not follow indirect
inheritance. Switch each `Test*` class to multi-inherit
`(_BaseClosureCase, unittest.TestCase)` so the discovery check
recognises them as runnable.
* `hardcoded-dns-resolver`: the per-block head suppression in
inventories/development/swarm.yml was insufficient; the lint scans
per-line and needs the marker on the actual `dns: "1.1.1.1"` line.
Verified locally: 16 of the previously-failing tests now pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New ``cli.administration.deploy.swarm`` package wraps the strict
``cli.administration.deploy.dedicated`` runner with a layered closure
so swarm deploys never require operators (or test scripts) to
enumerate every infra role:
effective --id = seed (operator --id, else inventory_role_groups)
∪ dep-walk(seed via applications_if_group_and_all_deps)
∪ default-placement safety net
The dep-walker is the same SPOT
(``utils.roles.applications.in_group_deps.applications_if_group_and_all_deps``)
the runtime constructor consumes via ``lookup('applications_current_play')``,
so the CLI-side closure and the in-play role set stay in sync.
The dedicated CLI keeps strict explicit-only semantics: ``main`` was
split into ``parse_args`` + ``run_from_args`` so the swarm wrapper
can share parser, mode wiring, validator and runner verbatim and
only mutate ``args.id``.
Meta-side: ``svc-db-postgres`` gains ``default_placement: manager``
(semantic symmetry with mariadb). ``default_placement`` is now a
PLACEMENT hint only ("if deployed, run on the manager"); deploy
decisions come from the dep-walk. Apps that declare ``mariadb:``
pull svc-db-mariadb; apps that declare ``postgres:`` pull
svc-db-postgres; apps that declare neither get no central DB.
Shared utils extracted so the four consumers stay aligned:
* ``utils.inventory.groups.inventory_has_group``: YAML + INI static
inventory group probe (extracted from ValidDeployId).
* ``utils.roles.meta_lookup.{get_role_default_placement,
iter_roles_with_default_placement}``: single source of truth for
``default_placement`` discovery (lookup plugin is now a thin wrapper).
Test pipeline (``scripts/tests/deploy/swarm/*``):
* Heredoc-Python shells converted to proper modules under
``utils.tests.swarm.*`` and invoked via ``python -m``:
- extend_inventory.py (was 08_extend_inventory.sh)
- write_extras.py (was 09_write_extras.sh dynamic half)
- derive_includes.py (replaces 06's hardcoded includes array)
- write_hosts_entries.py (replaces 02's hardcoded /etc/hosts heredoc;
derives every canonical + www-redirect domain from APP_ID via
canonical_domains_map + generate_all_domains, so the test
works for any app in the matrix without per-app maintenance)
* 01_create_network.sh deleted (single docker command inlined in workflow).
* 11_deploy.sh deleted (swarm CLI now called directly in workflow).
* 09_write_extras.sh: static half moved to inventories/development/swarm.yml.
* Remaining shell scripts renumbered 01-13 without gaps.
* 08_wait_converge.sh: case block on DB_DEP replaced with single
conditional (container name == DB_DEP).
* 06_provision_inventory.sh: hardcoded includes array replaced with
`mapfile -t includes < <(python -m utils.tests.swarm.derive_includes)`.
* Workflow YAML step references updated to new script names + module
invocations.
New tests (39 covering the closure surface):
* tests/unit/cli/administration/deploy/swarm/test_closure.py (14)
* tests/unit/utils/inventory/test_groups.py (11)
* tests/unit/utils/roles/test_meta_lookup.py (+3 for default_placement)
* tests/unit/utils/roles/validation/test_deploy_id.py (unchanged
8 tests still pass after refactor onto the shared util).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the four-container Jitsi Meet stack (web, prosody, jicofo, jvb) with a
3-variant matrix: V1 oauth2-proxy gate + Keycloak LDAP federation, V2
no-auth anonymous, V3 direct prosody mod_auth_ldap2 against svc-db-openldap.
Auth model is oauth2-proxy at the reverse proxy because Jitsi has no
native OIDC adapter — JWT-via-Keycloak would require a custom token
server outside this repo. The role injects a fixed-position
`[data-injected-logout]` button via templates/javascript.js.j2 so the
no-built-in-logout React SPA satisfies the persona contract's
no-direct-URL logout rule. V2/V3 declare PERSONA_BIBER_BLOCKED /
PERSONA_ADMINISTRATOR_BLOCKED because the shared persona helpers only
model the OIDC redirect chain.
Final FULL-matrix `mode=reinstall full_cycle=true` run green across all
three variants (V1: 6/6 pass, V2/V3: 2 pass + 4 blocked skips).
Also lands utils/parse_trace.py — a small zipfile helper used during
the iteration to extract URL traces from playwright .trace bundles.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds INFINITO_VENV_DIR=/opt/venvs/infinito to default.env (registered
in passthrough's STATIC_KEYS so make dotenv propagates it to .env) and
threads it through the build chain: compose.yml forwards it as a build
arg; scripts/image/push.sh asserts and --build-args it; the Dockerfile
declares the matching ARG/ENV and derives PYTHON/PIP/PATH from the
variable. scripts/meta/env/python.sh now prefers the value when set
before its fallback discovery so the SPOT is effective in env
resolution.
Fixes the dev image build regression where 'infinito --version' in
scripts/docker/entry.sh exited 127. Root cause: the Dockerfile RUN
SHELL uses bash -lc, which sources /etc/profile and resets PATH to
the Debian default, dropping ${INFINITO_VENV_DIR}/bin from it. The
fix invokes the script via its absolute path "${INFINITO_VENV_DIR}/bin/infinito"
so the post-install version check no longer depends on a PATH that
the login-shell init has just clobbered.
scripts/lint/wrapper.sh's existing PATH-reset note now references the
variable instead of the literal path. test-install-make.yml and
test-install-pkgmgr.yml read INFINITO_VENV_DIR from default.env up
front, pass it into their pkgmgr containers, and source the venv
activate script via the variable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Root cause of step-08 deploy failures across all 4 test-development
distros in run 26572559126: `make compose-deploy mode=reinstall
apps=web-app-dashboard INFINITO_SERVICES_DISABLED=matomo` (step 07)
made make export the cmdline var into the recipe's shell env, which
reached `runner.py` → `make setup` → `make dotenv`. The dotenv
generator's `setdefault` then baked the runtime-only value into the
on-disk `.env`. Step 08 sourced that .env, filtered `web-app-matomo`
out of its primary apps, and exited with "All primary apps disabled".
Fix:
- Drop the 7 routing knobs (apps, mode, purge, bundles, disable,
full_cycle, variant) from `passthrough.STATIC_KEYS` and from
`default.env`. They are per-invocation; .env is the persistent SPOT
and must not capture them.
- Switch every consumer from `os.environ["KEY"]` strict access to
`os.environ.get("KEY", "")` so the missing-key path stops at "no
filter" instead of a KeyError.
- Rename the env-var surface to match the make-alias surface 1:1
(lowercase): INFINITO_APPS → apps, INFINITO_SERVICES_DISABLED →
disable, INFINITO_BUNDLES → bundles, INFINITO_FULL_CYCLE →
full_cycle, INFINITO_VARIANT → variant, INFINITO_DEPLOY_MODE → mode,
INFINITO_PURGE_ENTITIES → purge. The make alias was the user-facing
name in every doc; routing-knob env vars now use the same name so
the cmdline form, the recipe env-prefix, and the consumer reads
all line up.
- Collapse the Makefile `compose-deploy` recipe to just the two
knobs whose alias differs from the env name (`type` →
INFINITO_DEPLOY_TYPE, `debug` → INFINITO_DEBUG). GNU Make
auto-exports cmdline variables into the recipe's shell env, so the
identical-name `$(if $(apps),apps="$(apps)")` lines were pure
noise.
- Drop `disable: ${disable}` from compose.yml. The dev CLI builds
`extra_env` per `docker exec` call and forwards `disable` via
`-e`, so the container env never needs the value frozen at
compose-up time.
- Update the 3 actual leak call sites
(07/08/09_deploy_*.sh) to the make-alias form
(`disable=matomo`, `full_cycle=true`, `variant=1`).
- Add `SIM112` to ruff's ignore set: the lowercase env names are
the deliberate make/env parity, not a typo.
Repro before fix: `INFINITO_SERVICES_DISABLED=matomo make dotenv`
wrote `INFINITO_SERVICES_DISABLED=matomo` to `.env`. After fix:
`disable=matomo make dotenv` leaves `.env` untouched.
Tests: `make test` and `make autoformat` green (2711 tests, 8 lint
sub-targets).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
`web-opt-rdr-www` generates a redirect vhost
`<NGINX_DIR>/conf.d/servers/{http,https}/www.<domain>.conf` for every
deployed app's canonical and alias domains. Those redirect vhosts are
not part of any app's `server.domains.{canonical,aliases}` list, so the
entity-keyed purge primitive in `utils/cleanup/nginx_vhosts.py` did
not yield them and `iter_vhost_files_for_entity` left them on disk.
Between matrix-deploy rounds this leaked the redirect vhosts of the
previous round's broader app set into the next round's narrower one.
On run 26451828677 job 77879571696 (web-opt-rdr-www on debian) the
round 1 -> round 2 transition
round 1/2 inv=github-0 apps=[15 apps incl. web-svc-cdn,
web-svc-simpleicons, web-svc-css, ...]
round 2/2 inv=github-1 apps=[web-opt-rdr-www]
dropped the bare `cdn.infinito.example.conf` etc. via the existing
purge but kept the matching `www.cdn.infinito.example.conf`,
`www.icon.infinito.example.conf`, ... behind. Round 2 only redeploys
web-opt-rdr-www, whose `www_redirect_domains` lookup now only emits
its own `www.w3redirect.<primary>` redirect, so nothing reissues the
stale www-prefixed vhosts.
`sys-ctl-hlth-csp` discovers domains by scanning .conf filenames in
the nginx servers dir, so it still probed the stale `www.<x>.conf`
hosts. nginx served them with a cert whose SAN no longer covered the
hostname (the per-domain cert path baked into the conf had been
removed alongside the bare vhost), the TLS handshake returned
`net::ERR_SSL_UNRECOGNIZED_NAME_ALERT`, the CSP probe exited 11, and
`sys-service-terminator : Apply state` failed after 10 retries.
Extend `iter_vhost_files_for_entity` to also yield the
`www.<domain>.conf` variant for every domain it already yields,
skipping domains that already begin with `www.` so a role that
declares its canonical with a leading `www.` (web-opt-rdr-www's own
`www.w3redirect.<primary>`) does not produce `www.www.<x>.conf`. The
existing two- and three-step purge primitives (db / compose / dir /
nginx) and the orchestrator in `scripts/container/purge/apps.sh` keep
their entity-keyed contract; only the per-entity yield set widens.
Coverage:
- test_yields_www_redirect_variant: per-app www-prefixed file is
yielded alongside the bare vhost.
- test_www_prefixed_domain_not_double_prefixed: a www-prefixed
canonical does not synthesize `www.www.<x>` even when such a file
exists on disk (defense in depth against a future regression that
prepends unconditionally).
- test_removes_www_redirect_variant: end-to-end purge removes both
bare and www variant, leaves an unrelated www-prefixed file alone.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Pruned per project convention "no unnecessary code comments":
- module docstring: 23 lines → 1 line (the file name + symbol already
state intent; full incident write-up belongs in the commit log)
- wipe_database_entries docstring: 11 lines → 2 lines
- six inline rationale comments in tests/__main__ removed (test
names already encode the intent)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Matrix-deploy variant transitions wipe a consumer's compose stack and
its named DB volume (`compose down -v` in
scripts/container/purge/entity/compose.sh) but `databases.csv` at
$FILE_DATABASE_SECRETS survives the purge. When the next round drops
the consumer from its inventory, the stale baudolo-seed row remains
behind. The subsequent backup run reads that row, attempts
`mariadb-dump -u<entity> -p<old_pw>` against the freshly initialised
RDBMS user that no longer exists, and the systemd unit fails with
"Access denied for user 'matomo'@'127.0.0.1' (using password: YES)"
(job 77797356000 on run 26428080957: matomo dump in
svc-bkp-container-2-local variant 1, where round 1 seeded matomo but
round 2 only deploys postgres + mariadb + svc-bkp itself).
Add `utils.cleanup.databases_csv` mirroring the existing
`utils.cleanup.tokens` pattern: parse the baudolo-seed CSV format
(';'-separated, four columns instance/database/username/password, with
header — verified against baudolo.seed.__main__:check_and_add_entry)
and strip every row whose `database` or `username` column matches an
entity derived from the supplied app ids via
utils.roles.entity_name.get_entity_name. Header and blank lines are
preserved verbatim; the file is rewritten only when at least one row
was removed.
Wire the helper into scripts/container/purge/apps.sh right after the
token-store wipe so every matrix-deploy inter-round purge invokes it
alongside the other cleanups. Operator-triggered purges
(`INFINITO_APPS=<app> bash scripts/container/purge/apps.sh`) also get
the cleanup — that is a behaviour change relative to before, where
the row would survive an operator-triggered entity teardown.
Coverage: 12 unit tests in tests/unit/utils/cleanup/test_databases_csv.py
covering the happy path, the multi-app case, the
`database='*'`/username-match case, the unknown-app no-op, header and
blank-line preservation, the all-rows-removed case, malformed short
rows, and the legacy-header-less file. `make test` green across all 5
targets.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Since 576fad84f (2026-05-10) moved role images from
`defaults/main.yml` into `meta/services.yml`, matrix-deploy's variant
baker has been writing the FULL effective config (services.yml +
variants.yml deep-merged via `get_variants`) into host_vars through
the `--vars` JSON consumed by `apply_vars_overrides`. That populates
`applications.{role}.services.{svc}.image` (and `.version`, .cpus,
etc.) with the upstream value, which then silently disables
`apply_mirror_overrides` for every role with an explicit image
declaration: the default `if_missing` policy short-circuits when the
field is already non-blank, so the rendered compose / Dockerfile FROM
keeps pulling from the upstream registry rather than the GHCR mirror.
The regression went unnoticed because Docker Hub usually serves the
pull, but it eventually surfaced as a BuildKit 429 on
`docker.io/library/php:8.2-apache` during the suitecrm Dockerfile
build (job 77602110531 on run 26362676803). Nextcloud, moodle,
mariadb consumers, postgres consumers, simpleicons, phpmyadmin,
phpldapadmin, jenkins, baserow, fider, ... all share the same latent
defect — only suitecrm happened to hit the rate limit first.
Fix the root cause instead of papering over per-role with
`mirror_policy: force`: introduce `get_variant_overrides_only` next
to `get_variants` and switch payload.py to it. The new function
returns the raw `meta/variants.yml` entries per role — no merge with
`meta/services.yml`. matrix-deploy now bakes only the variant-
specific overrides into host_vars; the role's services.yml fields
flow through the normal Ansible lookup chain at deploy time, leaving
`image`/`version` blank in host_vars so `apply_mirror_overrides`
populates them from `mirrors.yml` under its default `if_missing`
policy.
Behavior preserved for the other `get_variants` consumers:
* utils/manager/inventory.py:104 (schema-driven credentials) keeps
the full merged variant payload.
* planner.py:58 (variant-count only) is unaffected.
* legacy_resolver.py:42 (CombinedResolver runtime services_overrides)
keeps the full merged view.
Coverage:
* 3 new tests in test_applications.py for the new helper (single
empty variant when no meta/variants.yml, raw override without base
merge, cache invariance).
* test_payload.py + test_builder.py mocks repointed from `get_variants`
to `get_variant_overrides_only`.
* `make test` green across all 5 targets (lint, unit, integration,
external, test-lint) — 188 tests total.
Side-effects assessed:
* Repo-wide grep confirms no template / task accesses
`applications.X.services.Y.image` directly via hostvars — every
consumer goes through `lookup('config', …)` which still deep-merges.
* Roles previously pulling upstream now resolve via the GHCR mirror
whenever the image is already synced (the `images-mirror-missing.yml`
workflow keeps the destination populated).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`_apply_one_role_special_rules()` in utils/manager/inventory.py used
`bool(resolve_database_service_key(...))` to decide whether to
auto-generate a `database_password` credential for a role. The
resolver raises when multiple RDBMS keys are enabled — which is the
correct contract for compose_volumes / container_depends_on /
plugins.lookup.database (they need an unambiguous single key) but the
wrong contract for inventory's credential-generation pass: variant 1
of svc-bkp-container-2-local intentionally enables both mariadb and
postgres to exercise the engine-side dump path against both engines
at once, and the deploy aborted with `ValueError: multiple direct
database services are enabled: mariadb, postgres` during inventory
generation (job 77602095377 on run 26362676803).
Add a non-raising sibling helper `has_single_database_service()` that
returns True iff exactly one RDBMS service is enabled, and switch
inventory's call site to it. Multi-DB roles now skip the auto-
generated `database_password` (which was a phantom credential for
svc-bkp anyway — no schema.yml, no cross-role lookup; the role's own
`04_seed-database-to-backup.yml` receives `database_password` as a
caller-passed task var, not as a self-credential lookup).
`resolve_database_service_key` keeps its strict raise semantics for
the compose_volumes / container_depends_on / database-lookup callers,
which still need a single unambiguous key.
Add unit coverage for both functions (3 + 4 tests) in
tests/unit/utils/roles/applications/services/test_database.py.
Static check only (per operator request).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`generate_all_domains(include_www=True)` synthesizes `www.X` for every
canonical domain X and pushes them into CURRENT_PLAY_DOMAINS_ALL, but
they are never registered in the global canonical_domains_map or in
applications[*].server.domains. The strict `tls`/`cert` lookups
therefore raised `domain 'www.X' not found (domains/applications)`
when web-opt-rdr-www iterated REDIRECT_WWW_DOMAINS and called
`lookup('tls', 'www.X', 'enabled')` in the redirect-domain.yml TLS
gate — failing job 77602110568 on run 26362676803 (and cascading
through sys-front-tls/vars/main.yml, sys-front-tls/tasks/02_http.yml
and sys-front-tls-selfsigned/tasks/01_core.yml, all of which call
`lookup('tls', application_id, …)` with the same leaked www.X).
Add a single fallback branch to `resolve_term`: if the strict lookup
fails AND the term starts with `www.`, retry both resolver paths
(legacy reverse + applications) against the bare domain. The fix is
strictly additive — every previously successful term and every non-www
miss preserves its current behavior.
Coverage:
* test_www_prefix_falls_back_to_bare_domain
* test_www_prefix_with_unknown_bare_still_raises
* test_non_www_unregistered_still_raises (regression guard)
Verified statically via the in-container `resolve_term`:
www.infinito.example -> (svc-prx-openresty, infinito.example)
www.rdr-www.infinito.example -> (web-opt-rdr-www, rdr-www.infinito.example)
The global self-signed cert SAN list (cert.py:193) is built from
CURRENT_PLAY_DOMAINS_ALL, so it already covers the www.X variants —
nginx serves the redirect vhost with that cert and the TLS handshake
on the hlth-webserver check matches.
Static check only (per operator request).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Flatten records from all summaries and sort by ISO-Z timestamp so the
step-summary table is chronological across apps/variants/passes instead
of grouped per file.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`_build_application_defaults` returned variant 0 of every role with a
meta/variants.yml, which then deep-merged with the variant-N override
that build_dev_inventory bakes into the inventory. Any service flag
that variant N did not explicitly disable carried through from
variant 0 into the runtime applications map. The most visible casualty
was svc-bkp-container-2-local variant 1 (postgres + mariadb only),
where gitea + keycloak leaked in via variant 0, sys-service-loader
pulled in svc-db-openldap via keycloak.services.ldap, and the deploy
crashed in sys-svc-compose/04_files.yml on openldap's env.j2 with
"credentials is defined in schema at roles/svc-db-openldap/meta/schema.yml
but not set in application 'svc-db-openldap'".
Defaults now come from each role's variant-free base config
(meta/<topic>.yml plus schema-defaults, no variants overlay). Variants
stay accessible via get_variants for any caller that explicitly needs
a specific variant. Inventory-level applications.<app> overrides win
exactly as before.
Supporting changes that keep the variant matrix consistent under the
new defaults model:
- New lint guard tests/integration/roles/meta/variants/
test_non_baseline_explicit_disables.py: for every dynamic
service-key declared in a role's meta/services.yml that variant 0
pins to literal true, every non-baseline variant MUST explicitly
pin the same key to either literal true or to literal
enabled: false + shared: false. Statically-enabled keys are exempt
(same rule as test_auth_isolation.py).
- roles/web-app-{fusiondirectory,lam,nextcloud,phpldapadmin}/meta/
variants.yml: explicit `false` pins for dashboard / email / logout /
matomo / prometheus / talk in restrictive variants so deep-merge no
longer hands them the deleted variant-0 inheritance.
- roles/web-app-moodle/templates/playwright.env.j2: gate the LAM
block on `lookup('config', application_id, 'services.lam.enabled',
false)`. LAM is a transitive helper, not a moodle-owned service, so
the default-bearing lookup tolerates the missing path when the
variant does not deploy LAM.
- tests/unit/utils/cache/test_variants.py: rename the legacy
"always variant zero" coverage to TestMergedApplicationsUsesBaseConfig
and add test_get_application_defaults_omits_variant_zero_only_keys
as the regression guard for the leak this commit fixes.
- docs/contributing/artefact/files/role/variants.md +
docs/contributing/design/variants.md: variant 0 now manually
enumerates every dynamic services.yml key with
enabled: true + shared: true; `{}` is only permitted when the role
has no dynamic-enabled service-key.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
InventoryManager.resolve_schema_includes_recursive previously read the
root role's base meta/services.yml when discovering shared providers,
regardless of which matrix variant the planner had pinned for the
round. For a role whose variants.yml enables additional services per
variant (e.g. svc-bkp-container-2-local variant 2 enabling
services.ldap), the base config had no `enabled+shared` flag for those
services, so the recursive resolver never visited the variant's
shared providers — their credentials were never generated, and the
deploy crashed in the provider's env.j2 with "credentials is defined
in schema at roles/<provider>/meta/schema.yml but not set in
application '<provider>'".
Wires the active variant through the existing inventory pipeline:
- utils/manager/inventory.py: InventoryManager(variant=None|int). When
set, load_role_config_by_path overlays variants.yml[N] onto the root
role's meta config (only for the root — transitive providers keep
their own base lookup). variant=None preserves the pre-refactor base
behavior for every standalone caller.
- cli/administration/inventory/credentials: new --variant N flag,
forwarded to InventoryManager.
- cli/administration/inventory/provision: new --app-variants <JSON>
flag ({app_id: variant_index}), forwarded per-app to the credentials
subprocess.
- cli/administration/deploy/development/inventory/builder.py: emits
--app-variants from spec.variant_selectors() when the round pins any
variant. Apps without variants.yml collapse to a single empty
overlay in get_variants so this is a no-op for them.
10 unit tests cover: variant overlay for root role, variant=None
fallback to base, out-of-range fallback, no overlay on non-root
paths, --variant CLI forwarding (set + default), --app-variants
generator forwarding (set + omit), builder --app-variants emission
(set + omit).
Caveat: bkp-c2l CI variant 1 still loads svc-db-openldap transitively
at runtime (outside the planner's variant closure) — a separate
sys-service-loader / role-meta bug, not addressed here.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
default.env decomposed into INFINITO_PARENT_IMAGE_{OWNER,TAG,VARIANT}
atoms. New utils/env/handlers/infinito_parent_image.py owns the URL
composition `ghcr.io/{owner}/pkgmgr-{distro}:{tag}` as the single
definition site; scripts/image/{push,build}.sh consume the handler-
emitted INFINITO_PARENT_IMAGE from .env instead of inlining their own
format string.
Side effect: `compose up --build` now picks the parent image that
matches INFINITO_DISTRO. Previously default.env's hardcoded
pkgmgr-arch:stable won regardless, so debian-named containers were
silently built FROM the arch base. scripts/image/build.sh already
overrode per-distro, so its behavior is unchanged.
Other cleanups landed in the same pass:
- Slim variant removed end-to-end (Dockerfile stage, --variant CLI
flag, slim suffix logic). Zero in-repo callers used it.
- compose.yml build-args INFINITO_PARENT_IMAGE / INFINITO_SRC_DIR now
carry the same `:?Run 'make dotenv'…` guard as the runtime entries,
so a missing .env fails fast with a clear message instead of with
buildkit's `base name should not be blank`.
Note: fork-CI (pull_request_target trusted-helpers path) will stay red
until main's scripts/image/push.sh and scripts/meta/wait/images.sh
adopt the INFINITO_* names too — that is the deliberate cost of
keeping a single SPOT instead of carrying a bridge alias.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cluster 2b — two distinct root causes for "required role(s) did not
execute" in universal-pass:
* svc-bkp-remote-2-local: block-level `when` re-evaluated between
tasks; `flag.yml` set `run_once_*` before the dependency loop, so
the loop skipped — including sys-ctl-cln-bkps. Split into
main.yml/01_core.yml mirroring svc-bkp-container-2-local.
* web-opt-rdr-domains: misclassified as universal because the server
rule only matched `web-app-`/`web-svc-`. Universal constructor does
not load web-opt-roles.yml, so the role never executed. Extend the
server include regex to also match `web-opt-`.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Cluster 2a (hlth-msmtp failed=65 in universal single-app deploys):
the health-check script sent "To: administrator" but the administrator
Unix user is only provisioned in server-pass, so postfix rejected the
recipient. Replace the inline Jinja conditional with a role-local
lookup plugin that picks administrator.email (external), "administrator"
(local + user exists) or "root" (local + missing).
Side change: SIM105 cleanup in utils/install/lint/__main__.py.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
build/ is bind-mounted; under docker userns-remap the container root
maps to host uid 65534 (nobody), so files written from inside the
container are nobody-owned on the host. Whichever side created build/
first locked the other out: a stale nobody-owned build/ blocked the
host touch of build/install.stamp, and a host-owned 0755 build/
blocked the container from writing build/install-lint-<hash>.stamp.
Stamp writers now chmod the build dir to 0777 after creating it. The
first creator wins (only owner can chmod); both writers then place
their own stamps without colliding on directory ownership.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements docs/requirements/021-sso-flavor-migration.md atomically:
* Schema: every role's meta/services.yml replaces the parallel
`oauth2:` / `oidc:` top-level blocks with a single `sso:` block
carrying `enabled`, `shared`, and a `flavor` discriminator
(`oidc` | `oauth2` | `saml`; default `oidc`). Flavor-specific
sub-keys live under `sso.<flavor>.*` (`sso.oauth2.{origin,acl,
allowed_groups}`, `sso.oidc.plugin`).
* Provider role: roles/web-app-oauth2-proxy/ is deleted in its
entirety; its 5 sidecar templates move under
roles/web-app-keycloak/templates/sso_proxy/, and the per-consumer
rendering task lives at roles/web-app-keycloak/tasks/sso_proxy.yml.
web-app-keycloak now declares `provides: sso`.
* Runtime: new resolver utils/roles/applications/services/sso.py
centralises the SSO predicates (`is_enabled`, `is_proxy_gated`,
`is_oidc_native`) and the oauth2 sub-values
(`oauth2_origin_{host,port}`, `oauth2_acl`, `oauth2_allowed_groups`).
Three lookup plugins consume the resolver: `sso` (single-app
predicate access), `sso_proxy_consumers` (enumerates oauth2-flavored
consumers), `sso_oidc_plugin` (renamed from `oidc_flavor`, Nextcloud
plugin selector). sys-stk-backend, sys-svc-compose, sys-svc-proxy
and plugins/filter/compose_volumes.py all gate on
`lookup('sso', ..., 'is_proxy_gated')` instead of manually combining
`enabled` and `flavor`.
* Identifier renames repo-wide: `services.<entity>.ports.local.oauth2`
→ `ports.local.sso`; `PORT_BANDS.local.oauth2` → `PORT_BANDS.local.sso`;
`OAUTH2_PROXY_*` env/Jinja vars → `SSO_PROXY_*`;
`oauth2_proxy_cookie_secret` → `sso_proxy_cookie_secret`;
`OAUTH2_SERVICE_ENABLED` + `OIDC_SERVICE_ENABLED` collapse into
`SSO_SERVICE_ENABLED`; Playwright gating helpers
(isServiceEnabled/skipUnlessServiceEnabled/requireService/safeIsEnabled/
safeSkipUnlessEnabled) move from `oidc`/`oauth2` to `sso`.
* Dual-block resolution: per req's Background-Table, web-app-bookwyrm
is `flavor: oauth2`; web-app-gitea and web-app-friendica keep
`flavor: oauth2` with their existing `oauth2.acl.blacklist`
(/user/login) and `oauth2.acl.whitelist` (federation + discovery
endpoints) — preserving the operative oauth2-proxy gating.
* Tests: the two mutual-exclusion guards
(tests/integration/iam/oauth2_oidc/test_mutual_exclusive.py and
test_acl_mutual_exclusive.py) are deleted per Decision #10;
test_oauth2_contract.py is renamed and rewritten to
test_sso_contract.py, asserting that `flavor: oauth2` roles carry
`oauth2.origin.{host,port}` plus `ports.local.sso`; test_sso.py
(web-app integration) now requires the unified `sso:` block with an
explicit flavor or a `# nocheck: sso` opt-out; new
tests/lint/repository/no_legacy_sso_paths/ guards the source tree
against re-introducing the legacy strings.
* Docs: req 21 itself has every AC checked off, and reqs 006, 008,
009, 013, 015, 016, 019, 020 plus 10+ role READMEs and
docs/contributing/actions/testing/suppression.md are updated to the
unified SSO naming.
Empirically validated end-to-end against the validation app set
(web-app-keycloak, web-app-nextcloud, web-app-bookwyrm, web-app-gitea,
web-app-friendica, web-app-dashboard) in matrix variants 0 and 1; v2
also passes the 4 target roles' Playwright after a documented
pre-existing KC_HOSTNAME-drift workaround (separate concern, not
caused by this refactor).
BREAKING CHANGE: every consumer of `services.oauth2.*`,
`services.oidc.*`, `OAUTH2_PROXY_*`, `OAUTH2_SERVICE_ENABLED`,
`OIDC_SERVICE_ENABLED`, `oauth2_proxy_cookie_secret`,
`oauth2_proxy_application_id`, `ports.local.oauth2`, the
`web-app-oauth2-proxy` role, or the legacy gating helper service names
(`oidc`/`oauth2`) must migrate to the unified `sso` shape documented
in docs/requirements/021-sso-flavor-migration.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Closes the gap left by test_image_versions, which silently skips any
non-semver pin (latest, stable, branch tags). The new test_image_reachable
walks every entry from iter_role_images and probes the Docker Registry v2
manifest endpoint via HTTPS with anonymous bearer-token negotiation
(docker.io, ghcr.io, quay.io, mcr.microsoft.com). HTTP 404 hard-fails as
a broken pin; auth/rate-limit/5xx/network errors warn. Bounded by per-
request, per-probe, and global deadlines and parallelized via
INFINITO_WORKER_FETCH.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
INFINITO_CMD is a per-invocation argument, not a config value — drop it
from default.env, the passthrough STATIC_KEYS, and the dotenv-generator
test, and rename to lowercase `cmd` in Makefile targets, exec scripts,
internal callers, and user-facing docs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous regex anchored the `name` capture at line start and used
lazy `.+?`, which in production captured the full
`<ts> p=<pid> u=<user> n=ansible INFO| <role>` prefix as the role name
instead of just `<role>`. The CI step summary then rendered every row
with the timestamped log header and the `total` filter silently failed
(because `name` was no longer equal to `"total"`).
Add an optional non-capturing `(?:.*\bINFO\|\s+)?` group so the
ANSIBLE_LOG_PATH prefix is consumed when present and `name` captures
just the role token (or `role : task`). Test sample didn't carry the
log prefix, so the bug slipped through — add a prefixed-format test.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Move role_is_invokable / role_has_required_by / role_has_nocheck out
of plugins/filter (where they were flagged as test-only filters) into
utils/roles/required_by_coverage.py. All three use load_yaml_any,
read_text, and ROLE_FILE_META_SERVICES from the canonical SPOTs.
- Refactor the verifier (cli.meta.roles.services.called) the same way:
iterate roles/*/ directly, look up meta/services.yml via the constant,
parse with load_yaml_any, and tolerate a missing roles_dir (unit tests
point at a fake /repo).
- Make the mailu-dependency integration test yaml-aware for meta/
services.yml: walk the parsed tree and skip values inside any
`required_by` subtree. `required_by` is inverse coverage metadata for
the deploy verifier, not a "consumer of mailu" reference, so previous
regex-on-text matching produced false positives for cert-deploy.
- sys-ctl-mtn-cert-renew is gated by `lookup('tls', application_id,
'mode') == 'letsencrypt'`, which the verifier can't express via
role-id/category match — mark nocheck with the reason inline.
- Add README.md for cli/meta/roles/services (CLI-category convention).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Each role's Playwright spec is staged the same way `test-e2e-playwright` does at deploy time (central config + personas tree + service-gating) and `npx playwright test --list` runs against it. `--list` skips the browser launch and any test body, so this catches refactor regressions like missing helpers, broken exports, or syntax errors in seconds instead of waiting for a full deploy. Stub env values keyed off the variable-name suffix satisfy the few specs that build URLs or call `.replace()` at module-load time.
Parallelism caps at `INFINITO_WORKER_CPU` (same env knob as `scripts/make/parallel.sh`); output is replayed in `SPEC_FILES` order after all workers drain so stdout stays deterministic. `@playwright/test` ships as a devDep via the existing eslint `npm ci`. Workflow wires into `ci-orchestrator.yml` behind the `lint-gate`.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`role_summary.py` no longer truncates to a configurable top-N: drops
`DEFAULT_TOP_N`, `TOP_N_ENV_KEY`, and `_read_top_n`; `_format_table`
renders all rows from `parse_role_times`. Renames the Markdown
heading and the corresponding GitHub Actions step in the four
`test-deploy-*` workflows from `Top role runtimes` to
`Role runtimes` for accuracy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace the seven separate deploy-* make targets with a single `make
deploy` recipe that maps short Make variables to the canonical
INFINITO_* environment variables and dispatches via the new
scripts/tests/deploy/local/deploy/main.sh router. main.sh and all
sub-scripts continue to read the long-form names, so Ansible passthrough
and container forwarding stay unchanged.
Short Make CLI now exposes apps, mode, purge, type, bundles, disabled,
and full_cycle as aliases. The router picks the verb folder
(initialize, reinstall, update) and scope (all, selection) based on the
chosen mode plus the presence of INFINITO_APPS or INFINITO_BUNDLES.
Also folds in the global INFINITO_TEST_DEPLOY_TYPE -> INFINITO_DEPLOY_TYPE
rename across default.env, passthrough handler, dotenv test, CI deploy
workflows, discovery scripts, and the server stage task.
Active contributor and agent-iteration docs, role-level Administration
and README files, the bundles README, and the deploy reset helper are
all retargeted to the new short CLI. docs/requirements/ entries stay
untouched as historical audit logs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
