Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.
Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Resolve every 3+-file same-prefix group flagged by the new prefix lint
by moving the files into a subfolder named after the shared prefix and
dropping the prefix from the file names, updating all references:
- utils/env/handlers/infinito_*.py -> infinito/ (dir/, package_cache/
nested a second level); registry imports and the four handler-dir
globs switched to iter_project_files
- utils/install/lint/ansible_*.py -> ansible/ (aliased imports)
- roles/sys-svc-compose/files/swarm_*.{py,sh} -> files/swarm/
- web-app-erpnext/web-app-zammad files/scripts/apply_*.{py,rb} -> apply/
- web-app-odoo files/sql/{select,update}-* and sql/oidc/{grant,insert,
select,update}-* -> matching subfolders
- web-app-keycloak tasks/update/05_ldap*.yml -> 05_ldap/ (internal
includes made role_path-absolute)
- plugins/filter/get_*.py -> plugins/filter/get/ (filter names are the
filters() dict keys, so Jinja usage is unaffected; ansible discovers
the one-level-deep subdir)
Ansible lookup dirs, addon dirs and .github/workflows keep their flat
layout (the filename is the public id / GitHub only discovers flat).
Unit tests mirror the new source layout.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The export subdir was created run_once on the NFS server only; a worker
that had already readdir'd the export kept a negative dentry and swarm
rejected every replicas>1 task with 'readdirent .../_data: no such file
or directory' for the whole converge window. Materialize the bind-device
dir through each node's own mount, guarded by a per-node mountpoint
assert so an unmounted node fails the deploy loudly instead of receiving
a local mkdir that splits the brain and trips the nfs-client
has-local-data assert on the next full deploy.
compose_volumes popped the volume 'nfs' metadata before nfs_prep could
read it, so declared uid/gid/mode never took effect and every subdir
fell back to root 0755; keep the mapping under the compose-spec
extension key x-infinito-nfs (ignored by docker, verified against stack
config and compose config). KIX declares uid/gid 110010100 mode 0770
through it, fixing the backend_api_token EACCES crash on its shared
volume.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Flatten a single list term in the service lookup so query(service, [ids]) does one applications load instead of one per id. Precompute the required service ids in sys-service-loader and gate the preload by set membership, collapsing about 30 applications loads into one.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
svc-runner's CLI test runs a full nested reinstall deploy and needs
75+ minutes, so the uniform 3600s timeout killed it deterministically
in both modes. A new role_tests lookup reads roles/<id>/meta/tests.yml
(cli.timeout, scaled through the timeout lookup) with the existing
default as fallback, keeping the short hang-leash for every other CLI
test. svc-runner gets 4h and pins the nested deploy to variant 0; the
second matrix round only redeployed cdn+dashboard and added no
assertion value.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The custom flag moves from 33 hardcoded template kwargs into the merged
service config: 'custom: true' names the shared <entity>_custom build,
a string names a second distinct '<string>_custom' image inside one app
(nextcloud whiteboard). The image lookup reads the declaration as a
fallback, so config consumers (e.g. backup image matching) resolve the
exact deployed reference from one declaration point. nextcloud cron now
runs the nextcloud_custom build like the app service instead of the
upstream image.
A new services lint couples the declaration to reality in both
directions: custom requires a role Dockerfile, and a role Dockerfile
requires a custom declaration. The eight legacy roles deploying local
builds under upstream tags or ad-hoc image= overrides are ratcheted via
'# nocheck: dockerfile-custom'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New 'image' lookup is the SPOT for resolving <registry>/<image>:<version>
(swarm registry prefix in swarm mode, bare in compose; custom= accepts a
string base for a second distinct custom image inside one app).
container_image becomes a thin wrapper that forwards terms+kwargs and adds
the compose image: "<ref>" line, so all existing template call sites keep
their output byte-identical. The lookup-usage integration test now counts
lookup_loader.get() delegation as production usage.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The OpenTalk controller v0.34.1 verifies its bucket via S3 ListBuckets
membership, but SeaweedFS hides every bucket from an unconfigured
identity (confirmed identical on 4.38 and 4.39), and the embedded sidecar
started with no s3.config, so the controller's signed request got an
empty bucket list and crash-looped on 'bucket missing' although the
bucket existed. The sidecar now runs s3.configure to register the
consumer's access key with Read/Write/List/Tagging on its bucket, the
same grant the central provider gets from its mounted s3.json.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two wrapper behaviours move onto single sources of truth:
Exec timeout: SYS_TIMEOUT_CONTAINER_EXEC (07_schedule.yml, default 7d so a
hard kill never beats a production migration; the dev/CI inventory
tightens it to 60min) reaches every task host via the play environment as
CONTAINER_EXEC_TIMEOUT. The wrapper prefixes only the exec subcommand with
coreutils timeout --kill-after=15. Ansible task timeouts are SIGALRM-based
and slept through the last wedged docker socket for 4.5 hours; the OS
level kill still fires there and returns 124, which until/failed_when and
the rescue diagnostics react to. Unset or 0 keeps todays behaviour.
CA container paths: the bind-mount targets for the injected root CA and
wrapper script lived as string literals in container.py, compose_ca.py
and the filter plugin. They now live once in the CA_TRUST group_vars SPOT
(inject_cert_container, inject_wrapper_container) and reach container.py
as env vars (play environment plus CA_TRUST_SYSTEMD_ENV) and compose_ca.py
as CLI arguments built by the compose_ca_inject_cmd lookup. A caller that
provides the host CA env but not the container paths is half-migrated and
fails loudly instead of silently dropping CA injection; a host without any
CA env keeps the soft fallback. All tests read the paths from the SPOT,
and a regression test asserts the runtime scripts carry no literal copy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
storage.nfs shrinks to the deployment-specific server address: export_base moves into svc-storage-nfs-server's services.yml (the server dictates where exports root) and the mount protocol version into svc-storage-nfs-client's services.yml (the client owns how it mounts). Both are read through utils.storage.nfs loaders by the nfs lookups, the env handler and 00_topology, eliminating the unrendered-variables class for these values; the group_vars assert shrinks accordingly and the swarm test extras stop restating role defaults.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lookup plugins receive ansible variables unrendered, so routing the state subdir through a group_vars jinja value handed state_path the literal '{{ lookup(...) }}' string and the controller mount asked the server for /srv/nfs/{{ lookup('nfs_state_subdir') }} (Connection refused, nextcloud swarm roundtrip). Both nfs lookups now import STATE_SUBDIR directly; the intermediate lookup, the group_vars indirection and the unit-test fixture var that masked the gap are gone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A wedged DinD dockerd hung a single diagnostics docker exec until the 360-minute job timeout cancelled the job together with its artifact uploads (job 86013153099 sat 5h in service logs). Route every diagnostics exec through a timeout wrapper, tail-limit the log dumps, bound showmount, cap the collect step at 30 minutes, and dump the pending services' log tails once on the final converge failure. The NFS state path moves into a single source of truth: STATE_SUBDIR in utils/storage/nfs.py feeds the nfs_state_subdir lookup (deploy side) and the derived INFINITO_SWARM_NFS_STATE_PATH env var (harness side); the diagnostics script requires both path vars instead of defaulting, and the swarm inventory drops its redundant export_base restatement.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The controller crash-looped on a missing S3 bucket and the recorder on an unprovisioned OIDC client while the swarm converge gate blocked before any post-flush task could create either. The embedded seaweedfs sidecar now bootstraps its consumer bucket itself at startup (new seaweedfs_sidecar_script filter: entrypoint-wrapped server, bounded readiness poll, idempotent weed shell bucket create, TERM/INT forwarded to the server), and the Keycloak clients are provisioned before the handler flush. Post-flush bucket tasks stay as an idempotent backstop; the obsolete restart tasks are dropped. Live-validated against chrislusf/seaweedfs:4.38: bucket created, stop behavior identical to the stock entrypoint.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rename the meta/services.yml key across all roles, the roles_with_default_placement lookup (now roles_with_placement), the get_role_default_placement / iter_roles_with_default_placement helpers, and every consumer (swarm closure, compose_volumes filter, nfs_prep, container deploy template, constructor stage, extend_inventory) plus their tests and docs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
compose_networks/container_networks read DEPLOYMENT_MODE directly, so a node-local compose-in-swarm role (compose_mode_force=compose, e.g. the matrix ansible flavor) rendered a swarm overlay a standalone compose container cannot attach to ("network matrix not found"). Honor compose_mode_force like the other mode-aware lookups.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A curl against an accepted-but-silent peer hangs forever: task retries
never fire because the first attempt never returns, and the CI job dies
at the 6h runner cut (proven by the mailu admin-API hang that took the
suitecrm/peertube swarm legs down). The new `curl` filter renders the
hardened base command ("{{ 30 | curl }}" -> curl -s --connect-timeout 5
--max-time 30); scaling composes with lookup('timeout').
test_curl_via_filter enforces it: bare curl invocations in role tasks,
templates and shipped scripts fail the lint with a pointer to the
filter. Plain .sh files, which cannot render Jinja, pass by pinning
--max-time inline; Docker healthchecks are exempt (their timeout field
caps them); `# nocheck: curl-timeout` opts out (catalogued in
suppression.md).
Convert all 34 existing invocations: mailu (replacing the short-lived
MAILU_CURL var), elasticsearch, checkmk, yourls (300s for the
schema-creating installer POST), the telegram alert script, and the
plain scripts of jellyfin, svc-runner, apt_repo, moodle (600s tarball)
and the mattermost provisioner. Mailu's user/token tasks additionally
carry task-level timeouts so a wedged container exec cannot stall the
play.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The freeswitch build fetched libopusenc from launchpad.net, whose
librarian backend rate-limits CI (429s across 11 attempts); three more
build-time fetches carried the same risk. The maintainer publishes an
alangecker/* image for every buildable service, so compose_mods drops
build: wherever the service carries an image and compose pulls instead;
a guard task fails loudly if a future upstream service still builds.
Pin ref to the v3.0.23 release branch (the packaging repo has no tags)
and clone it without submodules via the new compose_repository_recursive
toggle: submodules only feed build contexts and DEV_MODE mounts, and
generate-compose falls back to the committed repos/tags file.
Declare the pulled images in meta/services.yml (updater-managed SPOT,
mirror pre-pull) and the named data volumes in meta/volumes.yml; the
override renders both through the standard container_image /
container_volumes / compose_volumes lookups with the service_name
pattern. Compose merges volumes by target, so the override's named
volumes verifiably replace the clone-local bind mounts (docker compose
config proven). This also heals a split-brain mount: apps-akka,
periodic and prometheus-exporter previously kept bind mounts while
their peers used the named volumes.
Teach container_image and compose_volumes to honour compose_mode_force
(the SPOT the other mode-aware lookups already read): bbb deploys as a
node-local compose stack in swarm, so it must get hub image refs and
plain local volumes there, not registry-prefixed refs and NFS
driver_opts. The unpinned-engine-volume lint learns the same two safe
conditions. Read the embedded Collabora port from web-svc-collabora's
services SPOT instead of hardcoding 9980.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deploy GitLab from the official CNG CE images as six services: webservice (puma) and sidekiq scale to the topology default, workhorse is the sole HTTP entry, gitaly and a one-shot migrations job are pinned singletons, shell runs gitlab-sshd. The Omnibus container, its Dockerfile and the GITLAB_OMNIBUS_* config groups are removed. All rails/workhorse/shell/gitaly configs and shared secrets are pre-rendered by Ansible and mounted read-only; eleven credentials land in meta/schema.yml including an rsa_pem_2048 OIDC signing key (new value_generator algorithm). Postgres, redis and S3 object storage ride the existing lookups; OIDC, SMTP, the public SSH port and the root password are preserved. compose_volumes gains a per-volume nfs:false opt-out so the gitaly repositories volume stays node-local in swarm (upstream forbids git repositories on NFS); nfs_prep skips volumes without driver_opts accordingly. The Nextcloud gitlab integration execs the CNG rails runner. ruby_single_line dies with its only consumer; ruby_dq keeps test coverage. Ports (8181/8080/2222/8075/3808/9122) and service names resolve from single SPOT declarations in meta/services.yml via vars constants; the version pin sits on the webservice image pair so the updater can bump it. Live-validated in compose up to all six services healthy and migrations completed; fresh-install-only semantics for existing Omnibus data are documented.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Workers failed 'docker build: no Dockerfile': the build context, compose file, .env and runner binary were staged on the stack host only while every node deploys its own node-local runner stack. Build and push the image once on the manager (RUNNER_IMAGE_REF via the swarm registry, 04_image.yml) and let workers pull it; render compose.yml (template with lstrip_blocks, matching the replaced stack_host_template action), .env and the extracted run.sh on every node. Add a node_local kwarg to the compose_networks/container_networks/compose_restart/compose_only lookups so this node-local compose stack renders a bridge default network (workers cannot create overlays), keeps restart: unless-stopped (ephemeral runners died after one job) and keeps container_name under swarm; default is unchanged for every other role. Treat build/install.stamp as stale when the venv is missing and exclude ./build from the nested-src copy, closing the nested-e2e ruamel failure class. Regression tests for the node_local render paths; the network include tests now accept lookup kwargs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The swarm branch passed values through raw, so a value with interior
whitespace (MAIL_FROM_NAME='Infinito.Nexus by ...') landed unquoted in
the rendered .env and broke the env parser: the snipe-it service
crashlooped at 0/3 until the converge wait failed in every variant.
Quote (and escape backslash/double-quote) only when the value contains
space/tab/newline; everything else stays raw, so the swarm invariants
hold: no compose $$-doubling, raw $ and double quotes preserved,
and the playwright --env-file path (docker CLI keeps quotes literal)
is unaffected because no playwright env value can contain whitespace.
Compose quoting is untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The sys-service-loader iterates the role-wide static service order, so it
asks the service lookup about keys (e.g. the css alias of cdn) that the
play-local registry built from applications does not contain. The lookup
raised and killed the bootstrap play on every distro.
Fall back to the registry read from roles_dir, the same source the
loader iterates, so every loader-visible key resolves with its provider
role and play-computed flags (false when no consumer is deployed).
Canonical aliases aggregate their primary key's consumer flags. A term
unknown to both registries still raises.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Container-name vars derived their value inconsistently (hard-coded literals, entity_name composites) instead of the role-owned SPOT. Add a lint requiring every *_CONTAINER var to resolve through lookup('config', app, 'services.<x>.name'), the bare entity_name form, or the engine lookup; give each multi-container worker a name: in meta/services.yml (with realistic resource limits) and switch its var to the lookup. Extend the engine lookup with a shared-aware 'container' field (central provider container when shared, embedded <entity>-<engine> otherwise) and gate zammad's embedded memcached sidecar on not-shared so its container_name never collides with the central memcached.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-host rendered applications flip dynamic shared flags with
group_names, so hosts computed divergent ordered lists and Ansible's
result-arrival include registration could deploy consumers before
their engines. The ordered mode now reads the static roles dir; the
per-host when gate in load_service.yml keeps deciding what loads.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ansible flavor deploys matrix as node-local compose even inside a
swarm cluster, but compose_only stripped container_name and friends
because DEPLOYMENT_MODE said swarm, so follow-up exec tasks could not
find the containers. compose_mode_force pins the rendering to compose;
the lookup honors it ahead of the cluster mode.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
resource_filter now falls back service_name -> entity_name -> hard
default, so entity-level limits keep applying to per-service compose
entries. Compose service names drop the entity prefix (kix, jitsi,
zammad, bluesky); the docker container_name keeps it via compose_only.
A new lint forbids '<entity>-'-prefixed service names.
Service names and internal ports are declared once: vars/main.yml
holds <ROLE>_<SERVICE>_SERVICE constants (container names derived via
entity_name), meta/services.yml declares ports.internal consumed
through config lookups. sso-proxy upstreams stack-qualify the origin
host in swarm because short aliases are ambiguous on shared overlays.
Stale proxy.service_key entries fixed for erpnext/shopware/nextcloud.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every compose.yml.j2 now declares services as '{{ service_name }}:' with
its '{% set %}' knobs on two-space lines above the key, includes at four
spaces and if/for tags aligned to the content they wrap. The layout
renders correctly because stack_host_template enables lstrip_blocks for
compose.yml.j2; indent(4, True) in the container base and '{%+' guards
in whitespace-sensitive scalars keep the shared fragments identical
under both render modes. A new lint test enforces the skeleton.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
One entrypoint wrapper imports the project CA into the JVM truststore for penpot and jenkins: env-driven (CA_TRUST_CERT from sys-svc-compose-ca, set in both modes), fail-open without a cert (production), fail-loud with one; writable cacerts are patched in place, read-only layouts fall back to /tmp + JAVA_TOOL_OPTIONS. Replaces penpot's inline compose command and jenkins' role-local entrypoint, which silently skipped the import when cacerts were not writable. ca_cert_container becomes a plain constant since its only Jinja consumer moved into the wrapper.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ca_cert_host derives the host cert path from SOFTWARE_DOMAIN; ca_cert_container carries the fixed in-container bind-mount target. CA_ROOT.cert_host and the test fixtures consume the filter; the standalone runtime scripts (compose_ca.py, container.py) cannot import it and are pinned to the SPOT by the unit test instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
'login source already exists' was matched inline in both the LDAP and the OIDC setup; the filter is the single marker source and normalizes case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The two handler tasks carried the identical four-marker stderr chain inline and matched case-sensitively; the filter is the single marker source and normalizes case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The five docker network connect/disconnect tolerances were inline stderr substring chains, inconsistent between sites (the handler variants matched case-sensitively without a default). The docker_network_noop filter is the single marker source per operation; disconnect additionally tolerates absent networks/containers, which is the proven swarm failure ('No such container: postgres' when the central DB is a swarm service). Drop the now-orphaned DISCOURSE_ERROR_NOT_CONNECTED.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
compute_external_network_roles() + compose_external_networks lookup expose which foreign provider networks a compose project references as external, so the swarm deploy path can pre-create them. network/create.yml probes for an existing network instead of unconditional state:present (the module's delete+recreate on attached networks crashed redeploys) and hard-fails on driver drift instead of silently skipping it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
email.py's environment folds TLS_ENABLED into its "external" base, so a non-external relay (no mailu deployed) under TLS resolved host to the mailu domain while auth/tls/from stayed in localhost mode: an unreachable, inconsistent msmtprc that aborted the sys-ctl-hlth-msmtp unit and the deploy (chess/bridgy-fed/hugo/legal). Gate host on external directly, like tls/auth/from, so a mailu-less deploy targets the loopback postfix that sys-svc-mail-smtp deploys under the same email.external gate. Adds a guarding unit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a 'timeout' lookup plugin that returns round(base * TIMEOUT_FACTOR)
(group_vars default 1, so value-identical to the old literals). One knob
in group_vars stretches every wait in the stack for a slow uplink:
TIMEOUT_FACTOR: 2 doubles them all.
- plugins/lookup/timeout.py + unit tests (identity, factor scaling,
string coercion, error paths).
- group_vars/all/17_resource.yml: TIMEOUT_FACTOR: 1.
- Convert all 34 literal 'timeout:' task keywords / module args across 25
task+handler files to "{{ lookup('timeout', <base>) | int }}".
- tests/lint/repository/yaml/test_timeout_uses_lookup.py enforces the
lookup in every task/handler timeout (opt-out: # nocheck: timeout-lookup).
- swarm.yml pre-deploy build handler: add the missing timeout (same
network-build hang vector as compose.yml's Build compose).
- Drop 13 pre-existing narration comments in the touched files (comment
lint enforces whole-file on any edited file).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In the swarm-DinD test the ansible controller (ansible_connection: docker) cannot read the STACK_HOST-rendered xwiki.properties, so compose_volumes' content hash falls back to the source path and the config name is constant. xwiki deploys twice (bootstrap with the OIDC block off, final with it on after the extension installs), so the config content changes but the name does not, and the final docker stack deploy fails to update the immutable config with 'only updates to Labels are allowed'.
Honor an explicit name: on config/secret entries in compose_volumes (volumes already did), and give xwiki's properties config a name that rotates with the OIDC phase, so the final deploy creates a new config instead of updating in place. Backward-compatible: no other role sets name: on a config. make test green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
compose_volumes emitted a top-level configs:/secrets: entry for every declared
config/secret, but container_volumes already gates each per-service reference on
its mount's when:. So a config whose only mount was gated off (e.g. LDAP
disabled) became an orphan: a top-level object no service references, whose file:
source points at a file the role never renders (same flag). Under docker stack
deploy that source is read at deploy time and the deploy fails; under compose it
was a harmless unused entry.
Gate the top-level emission on the same mount_when_passes container_volumes uses,
so a top-level config/secret exists iff a service actually references it. To share
the predicate, promote the private _mount_when_passes from container_volumes into
utils/roles/applications/mounts and import it in both filters; the bare except
moves with it (inline noqa), so container_volumes no longer needs its BLE001
per-file-ignore.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In swarm the shared openresty runs as an overlay container, so 127.0.0.1 is its own loopback, not the node's. bbb deploys node-local (host network) and its nginx binds the bbb-net gateway 10.7.7.1, a host IP the overlay reaches via inter-bridge routing. resolve_upstream now honors an opt-in services.<entity>.upstream_host in the compose branch (default 127.0.0.1) and bbb sets 10.7.7.1; proxy_pass collapses compose_mode_force apps to the static compose form. Compose keeps working because host-network openresty reaches 10.7.7.1 too. Validated live: curl bbb.talk/b/ returned 200.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Model change: per-user 'reserved: true|false' becomes 'accounts', a list of
host/mailbox/identity account types. Default when absent derives from roles
(bot/administrator -> [mailbox, identity], else [identity]) both for role-file
users and — via _apply_account_defaults after the inventory merge — for
inventory-only users, so users like biber keep their LDAP/Keycloak accounts.
Auto-generated role-suffix reservations get accounts: [].
New 'forward' field: mailu creates <username>@domain -> forward user's email
via the new 04b_manage_alias.yml (create-only; no_log on the loop so user
dicts never hit the log). Mailbox creation and password rotation now gate on
'mailbox' in accounts instead of role names; the mailu rbac role is renamed
mail-bot -> bot (token gate included).
svc-db-openldap drops the provision.reserved flag; OPENLDAP_USERS is always
the identity subset. reserved_usernames/non_reserved_users now classify by
'identity' in accounts, keeping the Keycloak deny-list semantics. root/backup
pin accounts: [host], administrator [host, mailbox, identity], helpdesk
[mailbox].
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
BBB runs the external bigbluebutton/docker project (host networking,
node-local stateful stack) and cannot be a docker stack deploy service.
Add a per-role compose_mode override so it deploys via node-local
compose up on STACK_HOST even inside a swarm cluster:
- sys-svc-compose: compose_mode_force (per-role override) + computed
compose_mode (SPOT) in defaults; up.yml + handlers route on
compose_mode instead of DEPLOYMENT_MODE.
- container_address: honour compose_mode_force so a node-local role's
exec targets its compose container (bare name), not a swarm service.
- web-app-bigbluebutton: compose_mode_force=compose, ungate compose
generation for swarm, flush handlers in-scope, and add
services.bigbluebutton.ports.internal.http for the swarm proxy upstream.
- web-app-matrix: add services.element.ports.internal.http (swarm proxy).
- test_variant_bundles: pin compose mode in the main test so it is not
polluted by another test leaking INFINITO_DEPLOY_MODE=swarm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
database_query is a module: it opens query_file on the host it runs on. In swarm the task is delegated to the stack host, which has no repo checkout at role_path, so query_file is not found. A new action plugin (same name auto-wraps the module) reads query_file on the controller and passes the SQL as 'query', so every role's query_file: works in both compose and swarm with no call-site changes. Reverts the per-call-site fider workaround in favour of this root fix, which also covers mattermost, odoo, openproject and pixelfed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Floor-to-1 now applies only to the topology default (no override term). An explicit service_replicas=0 emits 'replicas: 0' (service created but unscheduled, to be scaled up by a later task) instead of being bumped to 1. Empty/missing inventory group on the default path still floors to 1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
In swarm an app task can be scheduled on a worker, but the role exec'd
`container exec $(resolve-container-id ...)` on the manager -> "No such
container". Resolve the hosting node + node-local container id and delegate
each exec to that node.
- database.py: add a `service_name` want; resolve_host_cid/wait_converged
gain `*_service_name` for cross-stack services (mailu's shared mariadb)
- mailu wait-for-mariadb (DB probe + admin), mastodon db:migrate,
gitea migrate/admin-create, friendica manager-ops/patch/addons/bootstrap
(~13 exec sites) switch to resolve_host_cid + delegate_to
Compose-safe: resolve_host_cid degrades to HOST_NODE=inventory_hostname
(delegate is a no-op) + LOCAL_CID=bare compose container name, matching the
8 roles already using this pattern.
Note: friendica also needs service_replicas=1 (next bundle) so its resolved
container id does not churn across replicas.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve the 80 open CodeQL/code-scanning alerts on this branch.
Security-sensitive fixes:
- cli/administration/deploy/development: break the unsafe import cycle by
extracting the env/arg helpers into a leaf module `env.py`, giving the
load-time DAG compose -> common -> deps -> env.
- web-app-bluesky login-broker server.js: prototype-pollution guard on
cookie parsing, CR/LF + control-char log sanitisation, HTML-escaped
exception text, and a ReDoS-bounded handle regex.
- web-app-baserow SSO: harden the open-redirect `next` handling.
- network diagnose probes: require TLS >= 1.2.
- actionlint installer: validate tar members against path traversal.
- decidim/apt-purge lint regexes: rewrite to avoid catastrophic backtracking.
- 6x Playwright URL regexes anchored; crs-k/stale-branches pinned to a commit SHA.
Mechanical cleanups: unused imports/globals/locals, empty-except comments,
import-and-import-from, implicit-string-concat, multiple-definition,
unreachable/useless-assignment.
False positives are intentionally left in place (Django settings patches,
Odoo __manifest__, side-effect imports, cross-module symbols).
Recurrence guards:
- import-linter contract (pyproject + tests/lint) locks the dev-deploy
package acyclic; a new cycle now fails CI instead of only CodeQL.
- new lint test requires every bandit (S###) `noqa` to carry a justification.
- BLE001 (blind-except) added to ruff as a ratchet (tests/roles exempt,
current production offenders grandfathered).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Conflicts resolved:
- .claude/.gitignore: union of ignore patterns
- plugins/lookup/rbac_group_path.py: drop both unused conflicting imports
(body fetches apps via lookup_loader)
- roles/sys-svc-compose/handlers/main.yml: keep HEAD's refactor (compose-up
handlers live in compose.yml, included per DEPLOYMENT_MODE)
- utils/cache/base.py: keep HEAD's merge_with_defaults/FILE_TOKENS imports
Post-merge fixes for main's newly-enabled lints:
- pyproject.toml: grandfather 5 files under the BLE001 ratchet
- justify 6 pre-existing security # noqa markers (S108/S311/S701)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce plugins/lookup/path_absolute.py as the single source for the
repo-root anchor: lookup('path_absolute', 'roles/...') returns an absolute
path under playbook_dir, joining every term. Replace all hand-built
playbook_dir path constructions across .yml/.yaml and .j2 templates with it:
[playbook_dir, '<rel>'] | path_join -> lookup('path_absolute', '<rel>')
playbook_dir ~ '/<rel>' -> lookup('path_absolute', '<rel>')
{{ playbook_dir }}/<rel> -> lookup('path_absolute', '<rel>')
Unrelated repo_path symbols in cli/meta/roles/applications/resolution and the
single bare discover_playwright_roles(playbook_dir) call are left untouched.
Add tests/lint/ansible/test_playbook_dir_via_lookup.py: fails the build if any
of the three construction forms reappears in .yml/.yaml/.j2, pointing callers
at the lookup. Unit test covers the plugin.
Touching these files pulled their pre-existing narration comments into
test_comments_valid scope; clean them (delete restatement/banners, keep real
trip-wires as Workaround/Upstream/Invariant markers). Drop caution, important
and pitfall from the comment-lint marker vocabulary and remove the comments
that used them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The matomo bootstrap re-injects early-rendered web-svc consumers (such as
web-svc-logout) after obtaining the admin token, but in swarm this never ran:
roles_with_service filters by the manager's group_names, which omits
worker-placed web-svc services, so the re-injection loop was empty and the
logout page shipped without the matomo tracker.
- roles_with_service: add a scope kwarg; scope='all' bypasses the per-host
group_names filter (the merged applications config is already
deployment-scoped, so iterating all of it equals the deployment's consumers).
- 05_bootstrap: enumerate consumers with scope='all', stat each compose project
dir, and re-inject only deployed ones. Avoids a fatal chdir into an undeployed
consumer's missing /opt/compose/<entity>/ that scope='all' would otherwise hit.
- sys-front-inj-matomo: resolve matomo_effective_token (override else
lookup('users')); the disable-gate and the SitesManager getSitesId/addSite
calls use it, so the re-injection passes the fresh bootstrap token and bypasses
the play-start-stale controller-side users cache that left matomo disabled.
scope defaults to 'host' and the override defaults to empty, so every existing
caller and the per-consumer injection path are unchanged. Validated green in
both compose and swarm round 1.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract the csp, domains, and networks blocks out of roles/*/meta/server.yml
into dedicated meta files, each mapping to its own top-level applications key
via the recursive meta-walk:
csp: -> meta/csp.yml -> applications.<app>.csp
domains: -> meta/domains.yml -> applications.<app>.domains
networks: -> meta/networks.yml -> applications.<app>.networks
Residual nginx/proxy keys (status_codes, client_max_body_size, locations,
ip6_enabled, config_upstream_url) stay in server.yml; an emptied server.yml is
deleted. 98 server.yml -> 249 new files, 74 deleted.
- Consumers repointed applications.<app>.server.{csp,domains,networks} ->
applications.<app>.{csp,domains,networks} across templates, vars, tasks,
lookups and filters.
- Infra readers migrated: canonical_domains_map, application_domain_index,
build_ldap_role_entries, service-registry overlay discovery,
roles_with_service, and the file-based readers utils/domains/list.py +
utils/meta/scan.py (read the now-deleted server.yml directly).
- Detection + mapping: _META_TOPICS gains csp/domains/networks;
ROLE_FILE_META_{CSP,DOMAINS,NETWORKS} constants + ROLE_FILES entries.
- New self-expiring lint guard (test_server_meta_no_split_fields) fails if a
server.yml still carries a split field; auto-fails from 2026-12-28 as a
delete reminder.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Consumer lookup plugins now reach the merged config through the Ansible
lookup loader instead of importing get_merged_applications /
get_merged_domains from utils.cache. The merge stays one cached
implementation surfaced via the applications/config/domains/users lookups;
consumers call lookup_loader.get(...).run() (the compose_networks idiom).
- get_merged_applications walks the whole meta/ tree recursively
(meta/**/*.yml to applications.<app>.<path>, addons and any depth) and
folds the raw meta/volumes.yml subtree onto applications.<app>.volumes
after the templar render so the Jinja source: strings stay unrendered.
- 31 consumer lookups rerouted; providers (applications/config/
applications_current_play/users/domains) and the raw-volume accessors
(volume/nginx) keep the cache import behind # nocheck: lookup-cache-import.
- New lint test_no_cache_imports_in_lookups enforces the contract.
- nfs_flavor is now the flavor SPOT (svc-storage-nfs-server + nfs_client_src
resolve via lookup('nfs_flavor')); removed the unused STORAGE_NFS_FLAVOR
group_var. Extracted extract_default_credentials to
utils/cache/schema_credentials.py to keep applications.py under the cap.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
self._templar.template('{{ var }}') no-ops on untrusted strings in ansible 2.19+, so the nfs_* lookups emitted literal Jinja into /etc/exports and exportfs failed (the fsid=0 export never came up). The base lookups now read resolved inputs from 'variables'. nfs_flavor/nfs_client_src read the flavor from the applications variable (the variant-baked config provisioning bakes under applications:), not a re-merge of the role files via get_merged_applications. Adds a real-Templar regression test that fails on the old code.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
