Adds a lint-packages target/workflow that parses the generated packaging metadata with each ecosystem's native parser (dpkg-parsechangelog, rpmspec, bash -n for the PKGBUILD), failing on any parser stderr so warnings cannot slip through. Wired into the make lint fan-out and ci-orchestrator's lint gate; an install group provisions the parsers (best-effort, cross-distro).
Hardens package_mirror so the generated changelogs parse without warnings: every RPM body line gets a '- ' continuation prefix (a column-0 '*' was read as a new %changelog entry -> 'bad date'), and the archived-releases notice is folded into the oldest kept entry instead of trailing after the final entry (free text there made dpkg-parsechangelog warn). Regenerates packaging/debian/changelog and the Fedora spec accordingly.
Adds a test-lint check (test_package_changelog_completeness) asserting both package changelogs list every CHANGELOG.md + docs/changelog release and that their inline window matches CHANGELOG.md, plus unit regressions for the RPM continuation prefix and the Debian footer fold.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
These commits were made with --no-verify, which skipped make test, leaving three latent lint failures. utils/paths.py: use the cached utils.cache.files.read_text (verified no import cycle) instead of a plain read. .ansible-lint-ignore: add the pixelfed resolve_host_cid load-failure entry (templated playbook_dir include, false positive). library/database_query.py: ruff line-wrap. make test is now fully green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The compose-deploy inventory validator is spawned as a bare subprocess without the env build, so os.environ["INFINITO_DIR_VAR_LIB"] raised KeyError at import and aborted all compose deploys before Ansible ran. Fall back to the default.env SPOT (no hardcoded literal); read it plainly to avoid a utils.paths -> utils.cache import cycle.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Set DEFAULT_MAX_STORAGE to 350GB (was 400GB) and make it the single source of truth: the INFINITO_VARIANT_BUNDLE_MAX_STORAGE env handler's setdefault and the resolve_max_storage error message now reference DEFAULT_MAX_STORAGE instead of repeating the literal. Update the default-resolution unit test accordingly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dev/act ganesha server is NFSv4-only, but the swarm-test runtime extras forced storage.nfs.version: 3, so every mount failed 'Protocol not supported'. Set version 4 (write_extras + dev swarm inventory). The client probe mounted the Root_Squash export_base and a root probe-write failed; point it at the No_Root_Squash state path. Both client mounts hardcoded 'hard', so a torn-down test server wedged nodes into D-state; gate them to soft in dev/act (reusing the controller mount's gate). Hoist the shared mount-opts string into the STORAGE_NFS_MOUNT_OPTS SPOT (group_vars/all/15_storage.yml), used by all three mounts.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SPOT: declare the var-lib base once in default.env as INFINITO_DIR_VAR_LIB; the env build derives INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR from it; reference it everywhere (utils/paths.py, scripts via ${...:?}, tests, the dev inventory's DIR_VAR_LIB injection) instead of hardcoding the literal. group_vars keeps the literal default; the env-lookup lives in the dev inventory per the no-ansible-env-lookups lint rule. pull_specific_host.py --folder is now required and the role passes {{ DIR_BACKUPS }}.
Swarm e2e: collect_playwright_reports.sh docker-cp's the reports off the manager plus a workflow upload step, so swarm Playwright traces reach CI artifacts (was compose-only).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace 'lookup(applications) | node_max_old_space_size(app, service)' with the 'lookup(node_max_old_space_size, app, service)' lookup form (it resolves the app config itself), dropping the boilerplate at all 4 call sites (nextcloud whiteboard, peertube x2, suitecrm). Extract the shared decimal to_bytes size parser into utils/sizes.py, used by the lookup and the peertube install_mem_limit filter (removing the cross-plugin import).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the single utils/github/profile/role_summary.py script with a
focused CLI package split per responsibility (model, logparse, csvio,
render, github, sources). Output format is selectable via --format
(table default, plus markdown, csv, json); source is auto-detected as an
Ansible log, a role-runtimes CSV, or a GitHub Actions run/job URL (CSV
artifacts downloaded via gh).
The compose workflow now writes the CSV as a downloadable artifact and
appends the per-variant markdown to the step summary, and hard-fails
when the log is missing or carries no profile_roles data so a broken
metrics capture can no longer pass silently.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "Role runtimes" step read /tmp/ansible-<app>.log produced by a
docker cp out of the infinito container. That cp silently failed
(2>/dev/null || true) because the ansible log was not at the expected
in-container path, so role_summary reported "log not found".
The profile_roles table is already on the deploy step's stdout, so tee
it into the file role_summary reads and drop the broken cp (it also
raced the tee on the same path).
role_summary now segments the log by the matrix-deploy round/pass
markers and emits one per-role table per variant round; falls back to a
single combined table when no markers are present.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The local Path(__file__).resolve().parents[2] root computation tripped the project-root-import lint; use the canonical from utils import PROJECT_ROOT instead.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Open a new CI runner bundle as soon as it would exceed INFINITO_VARIANT_BUNDLE_SIZE variants OR INFINITO_VARIANT_BUNDLE_MAX_STORAGE (default 400GB) cumulative min_storage, so storage-heavy variants are not stacked onto one runner that then runs too long. Every job now carries its explicit variant CSV in the name; empty only for roles without variants.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Collect and aggregate each service's min_storage (summed like mem) and surface it as a column in the detail, per-variant and per-role views, the JSON output and the filter/order fields.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The required_by coverage verifier treated every flat 'required_by: {categories/roles}' as required in BOTH deploy modes. But sys-ctl-rpr-container-hard/soft repair compose compositions (DIR_COMPOSITIONS) and gate their body to compose; declaring required_by:[web] flat made the verifier expect them on swarm too, where they correctly skip, so web-opt-rdr-domains swarm CI failed 'required role(s) did not execute'. Let required_by nest under 'compose:'/'swarm:'; the verifier infers the deploy mode from the role set, mirroring DEPLOYMENT_MODE's own groups['svc-swarm-node'] definition, and enforces only the matching mode.
Migrate all 21 existing flat declarations to mirror under both modes (behavior unchanged) and mark sys-ctl-rpr-container-* compose-only. role_has_required_by reads both forms. New lint test_required_by_mode_keys requires the explicit compose:/swarm: form so the flat-vs-gate contradiction can't recur.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
container_ports reads application_id from the play vars, where sys-stk-full leaves it as the include var '{{ sys_stk_full_application_id }}'. _as_str() stripped its TrustedAsTemplate tag, so templar.template() silently refused it on ansible-core >=2.19, leaving the raw string and failing every shared-service compose/swarm render ('Application ID {{ ... }} not found'). Re-trust via trust_as_template, mirroring plugins/action/stack_host_template.py, and import it directly (drop the dead <2.19 ImportError fallback).
Also fix a stale import in utils/roles/applications/config.py: AnsibleUndefined moved from ansible.utils.unsafe_proxy to ansible.template in ansible 2.20, so the old ImportError fallback always used a no-op stub and isinstance(obj, AnsibleUndefined) silently never matched. Import it directly from ansible.template.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Formatter-only changes from make autoformat: wrap a long Jinja
expression in the dashboard manager-ops task and reformat the swarm
extend_inventory test util. No behavior change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
extend_inventory._host_topology placed the app-under-test on the workers whenever the role had no default_placement: manager. When APP_ID is svc-swarm-manager itself, that leaked the workers into the svc-swarm-manager group, which is the marker group IS_STACK_HOST keys on, so IS_STACK_HOST went truthy on the workers and sys-ca-selfsigned skipped its CA fetch (when: not IS_STACK_HOST). root-ca.crt never landed and 'CA | enforce permissions' failed on both workers. Guard the worker extension so this marker group stays manager-only.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
scripts/meta/resolve/apps.sh reads INFINITO_DEPLOY_MODE (added with the
per-mode skip feature) but the key had no registry entry, failing
tests.lint.filesystem.env.test_used_keys_declared. Add an empty default
to default.env and the key to the static-passthrough STATIC_KEYS so the
CI workflows' compose/swarm value passes through; empty disables the
per-mode skip filtering locally.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a `skip: [compose, swarm]` field on a role's meta/services.yml primary entity
so an app can opt out of the test-deploy matrix per deploy mode, replacing the
hardcoded exclude regex in apps.sh (the "TODO: Remove exclude condition"). The
filter threads through both discovery paths: the CI workflows pass
INFINITO_DEPLOY_MODE -> apps.sh -> `cli meta roles applications type
--skip-mode`, and roundtrip.sh honours the same get_role_skip SPOT per app/mode.
Apply skip: [compose, swarm] to svc-storage-nfs-server and svc-storage-nfs-client
(NFS storage is swarm-only and set up via the cluster inventory, not the matrix)
and web-opt-rdr-www (the previously regex-excluded app).
Covered by get_role_skip and skip_mode-filter unit tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Delete the per-deploy-type workflows (test-deploy-local/universal/
workstation) and the redundant act-all/act-app runner scripts (replaced
by make act-workflow). Only compose and swarm deploy tests remain.
Both now discover the full application matrix (every deployment type,
lifecycle-filtered to alpha/beta/rc/stable) instead of just the server/
web-* scope, via apps.sh resolving applications.type without --type.
Retire INFINITO_DEPLOY_TYPE entirely: discovery is now unconditional and
the deploy step never read it. Removed from apps.sh, ci/all.sh, the local
deploy scripts, the two workflows, default.env, the env passthrough list,
the dotenv generator test, the Makefile compose-deploy target, and docs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hoist the swarm-test node, network and NFS-export-base names out of the shell
harness (00_topology.sh) into default.env (INFINITO_SWARM_*), so the shell
harness and the Python harness (extend_inventory, write_extras) read one source
instead of each hardcoding the suffixes. 00_topology.sh, extend_inventory.py,
write_extras.py and 01/09/11 now read the SPOT; passthrough.py exports the keys
into the generated .env. 00_topology.sh reads default.env directly (not .env)
because the workflow sources it before make dotenv runs.
Rename topology.sh -> 00_topology.sh (sorts ahead of 01_/02_, the order it is
sourced in) and update all 10 references.
Add scripts/tests/deploy/swarm/clean_all.sh + make act-swarm-clean to reclaim
wedged act-swarm state between runs, label-scoped via INFINITO_SWARM_TEST_LABEL
so it never touches unrelated containers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
get_canonical_volumes returned {} when _CANONICAL_VOLUMES_BY_ROLE was empty even
though the defaults cache was warm (get_application_defaults no-ops on a warm
cache, so it never refilled the registry). That left nfs_prep's volume loop empty,
so the NFS subdir for shared/late-loaded services (svc-ai-ollama's ollama_models)
was never created and the swarm bind mount failed ENOENT. Force a real
_build_application_defaults on an empty registry. Adds two regression tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every swarm-test cluster is named so parallel clusters get distinct container and network names. topology.sh and extend_inventory require SWARM_NAME; act-swarm-zombie defaults it to the app id (name= overrides), the CI workflow derives it from the matrix app, and act-swarm-down/shell/playwright require name=. The bare swarm-mgr-01 names are gone (node refs now carry the cluster prefix).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Foundation for the central svc-db-*/svc-dns-* engine roles: the engine lookup and engines descriptor, per-consumer engine credentials in the inventory manager, shared-net overlay rendering, container_depends_on, the sys-svc-engine provisioning dispatcher, and the compose_volumes NFS-placement split. base.yml.j2 gates the embedded redis sidecar on lookup('engine','redis',application_id,'local').
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
system_pkg.py's apt-get update/install failed immediately ('Could not get lock /var/lib/dpkg/lock-frontend') when a concurrent deploy held the dpkg lock, so make autoformat/test could not run while a deploy was in progress. Pass -o DPkg::Lock::Timeout=600 to both apt-get invocations so they wait for the lock instead of failing, and update the unit test to assert the lock-wait. Scope is the dev/lint package installer only; the Ansible deploys use ansible.builtin.apt and are unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Drop the RuntimeBudget guard and its INFINITO_MAX_RUNTIME /
INFINITO_VARIANT_TIME_OVERFLOW env handlers: the deploy round loop now runs
every variant unconditionally instead of skipping (cut) or aborting (fail)
when the projected runtime would exceed the budget. Removes runtime_budget.py,
both env handlers + their registration, the budget calls in deploy.py, the
obsolete unit tests + the TestHandlerBudget class, and the docs reference.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The handler COMMENT rendered a 188-char line in .env; the env-file-schema
lint caps non-nocheck comment lines at 140. Trim it. Also collapses two
ruff-format lines in the variant-split unit tests. Follow-up to the
runner-split variant feature; no behaviour change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Thread a single `variant` selector (CSV) end-to-end so CI can split a
role's matrix deploy across runners.
- variant_select.py (SPOT, shared by init + deploy): one --variant taking
a CSV (2 pins one round, 0,1,2 is a runner bundle, empty = full matrix);
replaces the old separate --variant/--variants.
- planner.filter_plan_to_variants filters the plan to a round subset.
- CI discovery (utils/github/variant_bundles.py) splits roles with more
than INFINITO_VARIANT_BUNDLE_SIZE (default 3) variants into bundles of
consecutive indices, one matrix runner each (5 -> 0,1,2 + 3,4), threaded
via the variant env. test-deploy-{local,universal,workstation,server}.yml
switch to matrix.include with a comma-free variant_slug for artifact/job
names (GitHub expressions have no replace()).
- runtime_budget honours INFINITO_VARIANT_TIME_OVERFLOW (cut|fail) when a
runner cannot finish its slice within INFINITO_MAX_RUNTIME.
Unit tests + docs (variants.md, deploy.md) included.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add INFINITO_MAX_RUNTIME (env handler: 6h on real GitHub Actions, 48h otherwise)
and a RuntimeBudget guard in the dev-deploy variant loop. Before each next
variant round the guard projects elapsed + longest round so far + 30m buffer;
if it would exceed the budget it stops and emits a CI warning annotation
(never fails — the rounds that ran stay deployed).
- utils/env/handlers/infinito_max_runtime.py: new handler (canonical SPOT for
the var; uses ctx.on_gha, materialised into .env by the generator).
- cli/administration/deploy/development/runtime_budget.py: RuntimeBudget +
duration parser (6h/48h/90m/3600/3600s), extracted to keep deploy.py lean.
- deploy.py: loop uses budget.exhausted()/start_round()/end_round().
- tests for both the handler and RuntimeBudget.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the parse/collect/aggregate logic out of the ressources CLI into
utils/roles/applications/services/resources.py so the CLI and the upcoming
variant resource-budget guard share one implementation instead of duplicating
it. The CLI's query/__main__ and the unit tests now import from utils; the
per-module CLI files (parse.py, collect.py, aggregate.py) are removed.
No behavior change: the 43 ressources unit tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The swarm e2e now iterates one provision+deploy+e2e round per app variant,
mirroring the compose matrix deploy, so the keycloak TOTP env-gate is no
longer needed: the variant's services.keycloak.realm.totp_enabled=false is
baked into host_vars per round and resolves at deploy time in swarm too.
New cli/administration/deploy/swarm/matrix.py drives the rounds by reusing
the compose planner (plan_dev_inventory_matrix, _resolve_variant_payloads,
_bake_overrides): per round it provisions with the baked --vars overlay plus
--app-variants, extends the swarm topology, deploys (e2e in-deploy), waits
for convergence, and purges prior-round stacks between rounds via the new
purge_stacks.sh. The workflow collapses its per-step provision/deploy/verify
into one orchestrator step; chaos (seed/drain/assert) still runs once.
- cli/administration/deploy/swarm/matrix.py: NEW round orchestrator
- scripts/tests/deploy/swarm/purge_stacks.sh: NEW per-round swarm purge
- scripts/tests/deploy/swarm/05_provision_inventory.sh: per-round inv dir plus
conditional --vars / --app-variants (standalone path unchanged)
- .github/workflows/test-deploy-swarm.yml: single orchestrator step, timeout 360
- default.env, utils/env/handlers/passthrough.py: register INFINITO_VARS_PAYLOAD
plus INFINITO_APP_VARIANTS passthrough keys
- roles/web-app-keycloak/vars/main.yml: drop the ENVIRONMENT env-gate
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The net refactor (366dc7b70/cb27ff375) migrated every shared-net provider (postgres/mariadb/openldap/ollama/openresty/keycloak) to the overlay registry but missed seaweedfs and minio, so an object-store consumer like baserow never joined the provider's docker network and its S3 client hit ECONNREFUSED on seaweedfs:8333 (the intended direct data-plane path).
Add networks.overlay to both engines so consumers attach, and complete the multi-service shared-net-provider contract: (1) render.py emits a bare auto-IPAM project default for an own shared_net provider (no name, no subnet) so the entity-named external net created by network/create.yml is not re-requested -- this caused 'invalid pool request: Pool overlaps' on the first attempt. (2) a provider_self_alias flag lets the provider's sidecars (proxy/sso-proxy) JOIN the shared net WITHOUT publishing the <entity> alias, so the name resolves only to the main container and baserow's :8333 never round-robins onto a port-less sidecar. (3) seaweedfs/minio 01_core pre-create the external <entity> net via network/create.yml (mirroring web-app-prometheus) so compose adopts it as the project default instead of building <entity>_default.
Verified: baserow compose e2e test-seaweedfs.js passes (upload stored in the bucket, 3.9s); new TestOwnSharedNetProvider locks the contract; full make test green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reconcile main's object-store feature and pre-swarm deploy tasks with the branch's compose/networks/volumes SPOT and swarm lints.
objstore (seaweedfs/minio): standalone compose + embedded service templates migrated to the container_image/container_volumes/container_networks/compose_volumes/compose_networks/depends_on lookups; meta/volumes.yml on the canonical dict schema with s3.json and nginx.conf as type:config; embedded services made swarm-compatible via compose_only/compose_restart.
deploy tasks (erpnext, opentalk, seaweedfs, shopware, snipe-it, wordpress, yourls): container exec routed through the container_address resolver and gated on IS_STACK_HOST, shell module plus stack_host_template, no chdir or compose-verb forms.
Also removed orphaned vars, restored MAILU_INSTANCE_DIR, freed the seaweedfs subnet (192.168.206.0/24), stripped literal quotes from the nextcloud/bookwyrm env templates, and routed main's objstore compose_volumes tests through dir_var_lib.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Centralise the nginx proxy_pass directive in one source of truth (utils/networks/proxy.py + plugins/lookup/proxy_pass.py). Compose keeps the literal 127.0.0.1:<local> upstream; swarm resolves the service over the shared overlay at request time (variable upstream + http resolver) so a rescheduling backend never blocks openresty startup. Converts the html/media/upload/ws location partials, the keycloak oauth2 and login-iframe endpoints, and the logout endpoint to the lookup; adds the swarm resolver and the front-proxy service_key plumbing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend the registry-driven overlay renderer so the central openresty proxy
reaches every web-facing backend over a shared swarm overlay:
- Add consumer kind `web_facing` (matches web-app-*/web-svc-*) so every
web-facing app attaches its primary service to the openresty overlay in
swarm mode; openresty resolves them by swarm service DNS.
- openresty's overlay declares `collect_proxy_resolvable`, harvesting the
keycloak beacon's `auth.<domain>` alias onto its own attachment so each
per-app sso-proxy resolves the auth host to openresty.
- Fix the default-net name collision: a shared_net provider rendering its own
compose must not also emit `name:` on its default overlay (the external net
already carries that name), which made `docker stack deploy` fail with
"network <x> already exists".
Runtime-confirmed: openresty reached keycloak over the overlay (302, no more
connection-refused) once consumers attached.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
OpenTalk's images (web-frontend, controller, recorder) live on
registry.opencode.de, which was absent from both _MIRRORABLE_REGISTRIES and
_ALL_REGISTRY_PREFIXES in utils/docker/image/discovery.py. So is_mirrorable_image
returned False, iter_role_images skipped them, the GHCR mirror never copied
them, and the deploy pulled direct from registry.opencode.de -> TLS handshake
timeout (deploy failed).
Add the registry to both sets so the single iter_role_images gate enrolls the
3 images: the mirror-sync copies them to GHCR and the resolver rewrites the
deploy to pull from the GHCR mirror. The prefix entry strips the host so the
mirror destination is well-formed (no doubled registry). Add a regression test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a database-style object-store abstraction: the objstore lookup resolves the active engine (seaweedfs|minio) per consumer and yields a uniform S3 payload, objstore_secret_key is auto-generated like database_password, sys-svc-objstore delegates per-consumer provisioning to the resolved provider's objstore_init entrypoint (lazy-loading the provider when it was not preloaded), and compose/network/volume plumbing supports both a shared central instance and an embedded storage-only instance per consumer.
Add the web-app-seaweedfs provider role: Filer and Master UIs behind the admin-only oauth2-proxy via a role-local nginx and proxy.conf.j2 (S3 API domain stays ungated, admin UIs deny with 403 when SSO is disabled), per-consumer S3 identities from s3.json, idempotent bucket provisioning via weed shell, a /24 on the central-provider subnet range, Playwright coverage for admin/biber/guest plus consumer bucket existence, and unit tests for the objstore lookup and the per-engine compose volumes.
Mark web-app-minio lifecycle: unsupported, move its mc client image into a dedicated services.client entry, align it with the objstore credential contract, and make its consumer provisioning idempotent.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Convert the two registry-driven networks templates into pure-Python
rendering functions plus thin Ansible lookup plugins:
roles/sys-svc-compose/templates/networks.yml.j2 -> {{ lookup('compose_networks') }}
roles/sys-svc-container/templates/networks.yml.j2 -> {{ lookup('container_networks') | indent(4) }}
The rendering logic lives in utils/networks/render.py as pure-Python
functions that take a registry + closure-style config/database lookups,
making the logic directly unit-testable without a Jinja stub harness.
The lookup plugins are thin wrappers that resolve application_id,
DEPLOYMENT_MODE, and swarm.network.encryption from the templar, build
the service_registry, and forward to the render functions.
Migrate 233 include lines across 85 production templates. Convert
web-app-matrix's hand-coded MDAD networks block (default + conditional
postgres + conditional openldap) to the generic lookup; this adds an
openresty attachment when services.sso.enabled (harmless extra
interface, pinned in test_matrix_mdad_consumer_attaches_postgres_ldap_sso).
Mark local-DB / redis sidecars (sys-svc-rdbms/templates/services/,
svc-db-redis/templates/service.yml.j2) with `# nocheck: networks-literal`
because they intentionally attach to default-only.
Extend the lint to scan ALL roles/*/templates/**/*.yml.j2 (not only
compose.yml.j2) and to forbid the legacy include form; the marker
above is the documented escape hatch.
Coverage:
- tests/unit/utils/networks/test_render.py (30 cases incl. encrypted=false literal pin)
- tests/unit/plugins/lookup/test_compose_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/unit/plugins/lookup/test_container_networks.py (5 cases incl. lookup_loader arg-schema)
- tests/integration/infrastructure/compose/test_networks_render.py (10 snapshot scenarios)
- tests/integration/infrastructure/compose/test_network_includes.py (3 lint rules)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the hand-coded if-chains for db/ldap/ollama/proxy shared overlays
with a single registry-driven loop in sys-svc-{compose,container}/networks.yml.j2.
Each provider declares its overlay metadata under meta/server.yml.networks.overlay;
discover_role_services passes the block through verbatim into the service_registry.
Schema (modes/topology/consumer/aliases/proxy_resolvable) with implicit defaults:
- consumer.kind defaults to services_flags
- consumer.key defaults to <provides or entity_name>
- consumer.flags defaults to [enabled, shared]
- aliases defaults to [entity_name] (shared_net) or [] (default_net)
Openresty becomes a generic alias-collecting proxy: web-app-keycloak declares
a proxy_resolvable beacon with its public auth.<domain> alias; openresty sweeps
the registry at render time and harvests all proxy_resolvable aliases onto its
default-net overlay. Openresty no longer hardcodes keycloak's domain.
Lock the new behaviour with 9 render-snapshot tests covering provider self-render,
consumer attachment, beacon sweep, default-derivation, and compose-mode fallback.
Replace the Ansible-only `_entry.get('canonical') is not defined` filter with
`'canonical' not in _entry` so the templates also render correctly under vanilla
Jinja2 (test harness).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two scalar group-vars are referenced verbatim in
`roles/user-*/meta/users.yml` (`lastname: "{{ ORGANIZATION }}"`,
`description: "owned by {{ SOFTWARE_NAME }}"`). Ansible 2.19's
TrustedAsTemplate gate silently leaves those untagged Jinja
literals unrendered, so downstream consumers received the raw
`{{ ORGANIZATION }}` string. Keycloak rejected the resulting user
updates with `error-person-name-invalid-character` because `{` /
`}` are not valid in a person name.
Add `substitute_scalar_placeholders` next to the existing
`substitute_primary_domain_placeholder`. Both helpers run from
`get_merged_users` immediately before `_render_with_templar`, so the
cache returns fully-resolved scalars regardless of the templar's
trust gate.
Also restructure: move the cache into a `utils.cache.users` package
so the placeholder helpers no longer need a redundant `users_`
prefix:
utils/cache/
users/
__init__.py was utils/cache/users.py (public API
`get_user_defaults` / `get_merged_users` is
unchanged; consumers using
`from utils.cache.users import ...` keep
working via package __init__).
placeholders.py the substitution helpers, dropping the
redundant `users_` prefix.
tests/unit/utils/cache/
users/
__init__.py
test_module.py was test_users_module.py
test_placeholders.py new: 15 unit tests covering both
helpers (nested-structure walk, URL /
port handling for DOMAIN_PRIMARY,
Jinja-in-raw-value resolution, empty /
missing skip paths, and a guardrail
pinning the current
`_SCALAR_USER_PLACEHOLDERS` tuple).
Update README links under `utils/cache/` and
`docs/contributing/artefact/files/plugins/lookup/users.md` to point
at the new package path.
Placeholders covered today: ORGANIZATION, SOFTWARE_NAME. Extend the
tuple in `utils/cache/users/placeholders.py` when a new scalar
group-var appears in user metas.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
