Add plugins/lookup/container_image.py: returns the full `image: "<ref>"`
line so templates drop the redundant image: prefix and surrounding
quotes. Supports `custom=True` to auto-construct `<entity_name>_custom`
for locally-built images, plus explicit `image=`/`version=` overrides
for exotic cases (taiga back-auth/front-oidc variants, decidim infinito
suffix, etc.).
* 25 unit tests for the lookup cover compose vs swarm prefix, custom
flag, overrides, tag_only=True, missing version warnings.
* New strict lint (tests/lint/.../test_no_raw_image_in_compose_templates)
fails on any raw `image:` line in roles/*/templates/*.j2 unless
nocheck-suppressed. Legacy `image: "{{ lookup(...) }}"` form no
longer accepted.
* Migrate 159 image refs across ~80 roles to the standalone
`{{ lookup('container_image', application_id, '<service_key>') }}`
form. 26 roles use `custom=True` (mariadb, postgres, openldap,
jenkins, jira, gitlab, nextcloud, mediawiki, shopware, wordpress,
odoo, joomla, openproject, bookwyrm, espocrm, hugo, littlejs,
fediwall, roulette-wheel, openresty, confluence and more).
* Fix svc-db-redis application_id leak: service.yml.j2 is included
into consuming roles where application_id evaluates to the outer
role, so the image lookup needs the hardcoded 'svc-db-redis'.
* Integration tests (test_templates_require_image,
test_build_requires_image) recognise the standalone lookup as an
implicit `image:` at the same indent.
* Mediawiki swarm tasks: add explicit `:{{ MEDIAWIKI_VERSION }}` to
the registry-prefixed image string because the custom var is now
bare.
Only 3 legitimate exotic nocheck-suppressed image lines remain:
matrix-mdad runner (nested services.matrix.runner.image path),
matrix mautrix bridges (per-iteration dynamic) and svc-registry-docker
(chicken-and-egg self-reference).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a mode-aware `container_address` lookup plugin and a
`resolve-container-id` helper script that resolve swarm task IDs at
exec time (compose mode returns bare names unchanged). Migrates ~25
roles + ~28 task files from `command:` to `shell:` so the lookup's
`$(...)` subshell evaluates correctly, and extends `database` lookup
with an `address` key for cross-role DB exec.
Three new lint tests enforce the contract: every container exec
target goes through the resolver, lives behind an IS_STACK_HOST gate
(walks the include graph one hop deep), and uses `shell:` rather than
`command:` when the target is a Jinja reference. Existing nocheck
suppressions on bare-name DB exec sites (nextcloud redis, listmonk,
moodle OIDC, mailu) are replaced with the lookup-based form.
Fixes the systemd Environment= injection in sys-ctl-cln-disc-space
(it never expanded $(...)) by passing STACK / SERVICE_KEY parts and
resolving inside the script. Extracts shopware admin and mediawiki
extension install bodies into dedicated .sh files invoked via env.
Rewrites password-bearing shell tasks to either `| quote` directly or
the `-e VAR=... | quote` + `sh -c '... ${VAR} ...'` env-bridge pattern.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
DEPLOYMENT_MODE is now derived per role from group membership:
swarm iff len(groups[application_id]) > 1, else compose. The cluster
infrastructure flag SWARM_CLUSTER_AVAILABLE is true whenever any host
sits in svc-docker-swarm, so DB and proxy services render attachable
overlay networks even when their own group lists only one host.
All service_is_stateful flags are removed across svc-db-mariadb,
svc-db-postgres, svc-db-openldap, svc-prx-openresty, svc-docker-registry
and svc-db-mariadb's compose template; the inventory shape now expresses
the same intent. Cluster-wide handlers (swarm build/push/validate/deploy,
external network creation, DB healthcheck) gain run_once + delegate_to
the manager so they execute exactly once regardless of how many hosts
the role's play targets. CA-injection and compose validate are gated to
compose-mode only; their compose-config strictness breaks when the
deploy block coexists with container_name under swarm.
Dead per-host swarm-resolve infrastructure is removed: swarm_resolve_
container.sh, resolve_container.yml, resolve_self.yml, the MariaDB-
specific swarm_mariadb_cid.sh, the cross-host template replication
step, and the _swarm_retries / _swarm_delay defaults. With run_once +
delegate_to: manager replacing the old "render every template on every
swarm peer" pattern, every consumer becomes a noop on workers and
collapses to the manager's one-shot path.
The web-app-nextcloud memcache override mapping flips from the legacy
"single" sentinel to "compose" (semantically equivalent under the new
model). Requirement 023 is updated to reflect the new model and drops
the now-irrelevant per-role opt-out / placement-constraint clauses.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds the svc-docker-swarm, svc-docker-swarm-manager, svc-storage-nfs-server
and svc-storage-nfs-client roles, the compose_volumes filter NFS support,
the test-deploy-swarm-nfs CI workflow with its swarm-nfs-pilot script set,
and the deploy_mode-aware compose/swarm wiring across mediawiki, mariadb
and sys-svc-* helpers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh:
- rename run_sort_claude_settings worker to run_claude (matches the
renamed scripts/lint/claude.sh).
- drop the EXIT trap that wiped status_dir; cleanup happens explicitly
after `wait` returns, so a parallel worker that interrupts the
parent never races a vanished status dir.
- each worker now captures its tool's exit code and writes one of
OK / SKIP / FAIL. Tolerated `expected violations remain`-style exits
(markdownlint-cli2 rc≤1, ansible-lint rc≤2) still count as OK; only
real failures count as FAIL.
- all workers run to completion regardless of individual tool failures
(so the user always sees the full picture); after the summary, the
script exits 1 if any worker reported FAIL.
* scripts/lint/sort_claude_settings.sh -> scripts/lint/claude.sh
(history preserved via git mv); update test_claude_settings_sorted.py
fix-hint accordingly.
Plus working-tree drift across roles/, group_vars/, inventories/,
requirements/, and .pre-commit-config.yaml accumulated from the
make-test setup phase regenerating generated artefacts during recent
test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* add `image` and `version` to compose.services.openresty in config/main.yml
* replace hardcoded OPENRESTY_IMAGE and OPENRESTY_VERSION in vars/main.yml
* resolve image and version dynamically via `lookup('config', ...)`
* align role with centralized config-driven image management pattern
https://chatgpt.com/share/698c9d3f-5ea0-800f-9fef-1a5b7e4c8b60
Replace usages of the get_app_conf filter with the config lookup plugin
across roles and templates. Adjust quoting where required to keep YAML
and Jinja syntax valid. No functional changes intended.
https://chatgpt.com/share/69860678-cef8-800f-8402-7fe6231fdc2d
Add end-to-end support for reserved usernames and tighten CAPTCHA / Keycloak logic.
Changes:
- Makefile: rename EXTRA_USERS → RESERVED_USERNAMES and pass it as --reserved-usernames to the users defaults generator.
- cli/build/defaults/users.py: propagate flag into generated users, add --reserved-usernames CLI option and mark listed accounts as reserved.
- Add reserved_users filter plugin with and helpers for Ansible templates and tasks.
- Add unit tests for reserved_users filters and the new reserved-usernames behaviour in the users defaults generator.
- group_vars/all/00_general.yml: harden RECAPTCHA_ENABLED / HCAPTCHA_ENABLED checks with default('') and explicit > 0 length checks.
- svc-db-openldap: introduce OPENLDAP_PROVISION_* flags, add OPENLDAP_PROVISION_RESERVED and OPERNLDAP_USERS to optionally exclude reserved users from provisioning.
- svc-db-openldap templates/tasks: switch role/group LDIF and user import loops to use OPERNLDAP_USERS instead of the full users dict.
- networks: assign dedicated subnet for web-app-roulette-wheel.
- web-app-keycloak vars: compute KEYCLOAK_RESERVED_USERNAMES_LIST and KEYCLOAK_RESERVED_USERNAMES_REGEX from users | reserved_usernames.
- web-app-keycloak user profile template: inject reserved-username regex into username validation pattern and improve error message, fix SSH public key attribute usage and add component name field.
- web-app-keycloak update/_update.yml: strip subComponents from component payloads before update and disable async/poll for easier debugging.
- web-app-keycloak tasks/main.yml: guard cleanup include with MODE_CLEANUP and keep reCAPTCHA update behind KEYCLOAK_RECAPTCHA_ENABLED.
- user/users defaults: mark system/service accounts (root, daemon, mail, admin, webmaster, etc.) as reserved so they cannot be chosen as login names.
- svc-prx-openresty vars: simplify OPENRESTY_CONTAINER lookup by dropping unused default parameter.
- sys-ctl-rpr-btrfs-balancer: simplify main.yml by removing the extra block wrapper.
- sys-daemon handlers: quote handler name for consistency.
Context: change set discussed and refined in ChatGPT on 2025-11-29 (Infinito.Nexus reserved usernames & Keycloak user profile flow). See conversation: https://chatgpt.com/share/692b21f5-5d98-800f-8e15-1ded49deddc9
