00_core.yml included 01_setup_oidc.yml only when services.sso.enabled
was true, but that file carried the whole tenant bootstrap: wait for
HTTP, count tenants, POST /_api/tenants, create the admin user and
activate the tenant. Only its last task configured the Keycloak
provider. With SSO off no tenant was ever created.
Fider's RequireTenant middleware then answers every tenant route,
/api/v1/tags included, with 307 to /signup, and /signup is open to any
visitor, who can claim the fresh installation and become its
administrator. The MCP read fider_list_tags got that 307 in the swarm
SSO-off matrix row.
The tenant bootstrap moves unchanged into tasks/01_tenant.yml, which
00_core.yml includes on the stack host regardless of SSO. The OIDC file
becomes tasks/02_setup_oidc.yml and keeps only the provider upsert,
still gated on services.sso.enabled. The README states which bootstrap
steps run without SSO.
A unit test evaluates the include and task guards for every SSO,
stack-host and deploy-mode state: a fresh stack host posts exactly one
tenant with and without SSO, an existing tenant is never posted again,
the Keycloak provider is written only with SSO, and a non-stack host
runs none of it. The compose matrix row with SSO off stayed green
because nothing there read a tenant route.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review follow-up on the in-container probes of gitea, shopware and fider.
- fider: the tenant POST over /dev/tcp waited on `head` without a bound,
so a peer that accepts and stays silent blocked the task for good. It
now runs under `timeout` (TIMEOUT_FACTOR-scaled 30 s, the old uri
socket timeout) and reads the status line with the `read` builtin, so
the bound kills the whole probe. Only outcomes Fider never committed
are retried: a refused connect and a 5xx (Fider rolls the request
transaction back on a handler error). A timeout or a closed socket
after the request was sent fails the task instead of resending the
POST; the next run's tenant count decides again.
- fider: both bootstrap tasks get `no_log` behind MASK_CREDENTIALS_IN_LOGS,
as the body carries the administrator's name and email.
- shopware: each wget probe runs under `timeout`, as busybox wget waits
900 s per read by default.
- shopware, gitea: the first readiness wait derives its retries from
lookup('timeout', 300) again, the budget of the wait_for it replaced,
so TIMEOUT_FACTOR scales it.
Refs #1089
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
In swarm, `docker stack deploy` drops the host IP of a short port mapping,
so a `127.0.0.1:<ports.local>` publish lands on the ingress routing mesh
on every node. The provisioning of these three roles only worked because
the mesh answered on the host:
- web-app-gitea waited for `DOCKER_REACH_HOST:<ports.local.http>` with
wait_for and polled `/api/v1/version` there with uri_retry.
- web-app-shopware waited for `DOCKER_REACH_HOST:<ports.local.http>`.
- web-app-fider bootstrapped its first tenant in swarm by POSTing to
`127.0.0.1:<ports.local.http>` on the hosting node.
As long as these calls depend on the mesh, container_ports cannot stop
publishing loopback ports in swarm. The probes now run inside the
service's own container through the resolved `<PREFIX>_LOCAL_CID` on
`<PREFIX>_HOST_NODE`, against the internal port:
- Gitea uses the curl it ships. The first wait accepts any HTTP answer,
as wait_for did. The post-migration wait still requires 200.
- Shopware runs the wget probe its compose healthcheck already uses.
- Fider ships no HTTP client, so bash writes the tenant POST to
/dev/tcp. The body comes in on stdin and the status line is checked
for 200/201.
Fider's compose path is unchanged. The compose render of all three roles
is unchanged. The host-side ingress guard from svc-swarm-node stays in
place until the remaining host-side consumers are moved.
Refs #1089
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The contract check ran as an Ansible task in the middle of every provider deploy, and the two client checks ran the same way. A test that is part of the deploy cannot be run without deploying, cannot be re-run on its own, and reports its verdict as a deploy failure. Twenty-seven task files lose 491 lines to this.
It is now what it always was. Each provider ships files/test/test.sh, templates/test.env.j2 and meta/tests.yml, and test-e2e-cli stages one shared harness beside it: run.sh drives contract.sh in a throwaway container on the provider's own network, then isolation.sh for adapter-fronted providers. The read probe each provider offers moved from the calling task into meta/mcp.yml next to the allowlist it must satisfy, and the schema now rejects a read probe naming a tool outside that allowlist.
The bearer the test presents is the one a client resolves through mcp.credential, via the new mcp_client_bearer lookup over resolve_credential, which moved to utils.roles.applications.mcp so it is the same function mcp_servers renders every client connection with. The probe presented whatever the calling role passed, which for the fifteen adapter-fronted providers is a different secret than any client holds.
Sidecar isolation is read from the engine rather than measured inside the container, so it no longer needs a delegation to the node holding the task: attachment is compared by network name, which drops the gwbridge special case, and a swarm worker says it cannot measure instead of skipping silently.
Retrying is bounded by wall clock against MCP_DEADLINE_SECONDS, which scales through the same lookup('timeout') as the harness budget it has to stay under. A flag that is neither true nor false aborts rather than reading as off.
mcp-provider-probe becomes mcp-provider-cli-test: an enabled provider must ship the test rather than include the task. mcp-probe-bearer becomes mcp-adapter-bearer, tying ADAPTER_BEARER to the declared credential key now that no probe include carries it. A new integration test renders the shared env template for every provider through Ansible's own templar, so a later change to it breaks visibly instead of poisoning twenty deploys quietly.
Hermes and OpenClaw keep their client-side proof as CLI tests of their own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ten provider roles served an MCP surface that other roles consume, but
were never declared as shared services, so the registry did not know
them. Declaring them pulls in the recursion contract every registry
service owes: seven roles move their entry tasks into tasks/00_core.yml
behind the run_once flag, so the service loader cannot include them a
second time.
pretix already had a run_once block, but set the flag last; the flag now
comes first, before the body it protects.
Ten applications with no MCP server of their own get one without a line of code
inside them: qdrant, checkmk, fider, jellyfin, listmonk, pretix, prometheus,
snipe-it, zammad and libretranslate.
The sidecar is confined rather than trusted. It runs read-only, drops every
capability, forbids privilege escalation, publishes no port and joins only its
own provider's network, carrying neither that provider's alias nor the consumer
networks the provider itself joined. Its tool surface is a JSON contract, not
code: which tools exist, which upstream path each maps to and what shape the
arguments take. Adding a tool is a line of JSON, not a new image.
The image is built locally from the audited adapter source with the base image
as a build argument, so all ten share one reviewed build instead of ten upstream
dependencies. Each role provisions its own read-only identity in the language
its application speaks, and every adapter include is guarded by the role's MCP
flag, so switching it off leaves no container behind.
Add a 'timeout' lookup plugin that returns round(base * TIMEOUT_FACTOR)
(group_vars default 1, so value-identical to the old literals). One knob
in group_vars stretches every wait in the stack for a slow uplink:
TIMEOUT_FACTOR: 2 doubles them all.
- plugins/lookup/timeout.py + unit tests (identity, factor scaling,
string coercion, error paths).
- group_vars/all/17_resource.yml: TIMEOUT_FACTOR: 1.
- Convert all 34 literal 'timeout:' task keywords / module args across 25
task+handler files to "{{ lookup('timeout', <base>) | int }}".
- tests/lint/repository/yaml/test_timeout_uses_lookup.py enforces the
lookup in every task/handler timeout (opt-out: # nocheck: timeout-lookup).
- swarm.yml pre-deploy build handler: add the missing timeout (same
network-build hang vector as compose.yml's Build compose).
- Drop 13 pre-existing narration comments in the touched files (comment
lint enforces whole-file on any edited file).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
database_query is a module: it opens query_file on the host it runs on. In swarm the task is delegated to the stack host, which has no repo checkout at role_path, so query_file is not found. A new action plugin (same name auto-wraps the module) reads query_file on the controller and passes the SQL as 'query', so every role's query_file: works in both compose and swarm with no call-site changes. Reverts the per-call-site fider workaround in favour of this root fix, which also covers mattermost, odoo, openproject and pixelfed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Switch database_query from query_file to query + lookup('file', ...) so the SQL is read on the Ansible controller and embedded, instead of opened on the (remote, delegated) swarm DB host which has no repo checkout. Works in both compose and swarm.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove OLLAMA_CONTAINER_ADDRESS, TAIGA_MANAGER_CONTAINER_ADDRESS and
FIDER_CONTAINER_PORT (no remaining references after the cross-node exec
migration and the fider wget removal), and switch the fider swarm tenant
bootstrap from uri to uri_retry for module-lint compliance.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fider image ships only the fider binary (its own healthcheck runs
'./fider ping'), so the swarm in-container wget readiness probe failed
with exec: wget: executable file not found in $PATH. Use './fider ping'
for readiness and post the tenant bootstrap from the hosting node against
the ingress-published routing-mesh port instead of in-container wget.
Compose paths are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Introduce plugins/lookup/path_absolute.py as the single source for the
repo-root anchor: lookup('path_absolute', 'roles/...') returns an absolute
path under playbook_dir, joining every term. Replace all hand-built
playbook_dir path constructions across .yml/.yaml and .j2 templates with it:
[playbook_dir, '<rel>'] | path_join -> lookup('path_absolute', '<rel>')
playbook_dir ~ '/<rel>' -> lookup('path_absolute', '<rel>')
{{ playbook_dir }}/<rel> -> lookup('path_absolute', '<rel>')
Unrelated repo_path symbols in cli/meta/roles/applications/resolution and the
single bare discover_playwright_roles(playbook_dir) call are left untouched.
Add tests/lint/ansible/test_playbook_dir_via_lookup.py: fails the build if any
of the three construction forms reappears in .yml/.yaml/.j2, pointing callers
at the lookup. Unit test covers the plugin.
Touching these files pulled their pre-existing narration comments into
test_comments_valid scope; clean them (delete restatement/banners, keep real
trip-wires as Workaround/Upstream/Invariant markers). Drop caution, important
and pitfall from the comment-lint marker vocabulary and remove the comments
that used them.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_name_emoji_and_length.py: every Ansible task
name and GitHub Actions workflow/job/step name must start with an emoji (first
char not 0-9/A-Za-z) and be <= 80 chars.
- emit emoji names at the source: cli/build/include generator now writes
"🔧 setup <app>" / "🚿 flush handlers after <app>" so generated
tasks/groups/*-roles.yml satisfy the convention too
- convert existing task/step/job/workflow names to emoji-prefixed, choosing a
task-fitting icon per action (📦 install, ✅ ensure, 🔍 probe, 🥾 bootstrap,
🚀 deploy, 🧩 service-module load, ⬇️ checkout, 📂 include-routines, ...)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the per-role swarm host-node/container-id resolve duplicated across fider, matomo, decidim, keycloak (login + realm-import), collabora and listmonk with two shared task utils under roles/sys-svc-compose/tasks/utils/swarm/. resolve_host_cid sets <prefix>_HOST_NODE / <prefix>_LOCAL_CID (swarm: converge + node + cid delegated to the node; compose: inventory host + given cid); wait_converged is the bare convergence gate.
Teach test_variable_definitions to read templated set_fact keys like {{ prefix }}_HOST_NODE as a *_HOST_NODE name pattern (guarded so a pure {{ x }} key cannot become a catch-all), and add the playbook_dir-relative include load-failure exemptions to .ansible-lint-ignore.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Pure 1:1 rename (69 insertions / 69 deletions across 66 files) to
finish the ``docker_*`` → ``compose_*`` alignment started in
8373468e9. The flag configures whether the compose include flushes
handlers right after run; the ``docker_`` prefix was misleading
(handlers are Ansible-side, not docker-side) and the variable is part
of the same ``sys-svc-compose`` SPOT cluster as ``compose_repository_*``.
Touches the role default, the implementation
(``sys-svc-compose/tasks/utils/up.yml``), all 60+ consumer roles,
the network-utility wrapper + its README, and the
``test_network_create_via_util.py`` lint (both the docstring guidance
and the asserted diff).
Breaking change for any external override that still sets the old
name — no alias is provided.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Replace 13 raw `container exec mysql/psql` invocations across
web-app-pixelfed, web-app-fider, and web-app-mattermost with the
canonical `database_query` module + `query_file:` + `named_args:`
pattern. The module already speaks both shared (central `svc-db-*`)
and non-shared (per-role `<entity>-database`) modes via the
`database` lookup, so the same task shape works regardless of DB
topology — the per-role hardcoded container-exec argv was the only
blocker.
- Extract every inline SQL body into per-role `files/sql/` trees.
Pixelfed splits further into `files/sql/oidc-mapping/` because the
OIDC-mapping cluster is a distinct concern from the bootstrap /
administrator queries. Each SQL file is single-purpose and named by
verb-noun (count-users, select-admin-id, upsert, etc.) to match the
precedent set by web-app-odoo.
- Fider's existing `files/{create_admin_user,configure_oidc_provider}.sql`
move to `files/sql/` and switch from psql-specific `:'var'`
variable substitution (which forced `stdin:` piping with `-v`
flags) to engine-neutral `%(name)s` placeholders the module
escapes per backend.
- Pixelfed loses `PIXELFED_MARIADB_ARGS` + 3 derived DB vars
(`*_CONTAINER`, `*_USERNAME`, `*_PASSWORD`) from `vars/main.yml` —
they were the manual argv bundle for the old container-exec
pattern and become orphan once `lookup('database', application_id)`
takes over.
- Every refactored task carries `mask_values:
MASK_CREDENTIALS_IN_LOGS` (module-level credential redaction);
tasks that previously used `no_log: MASK_CREDENTIALS_IN_LOGS`
keep both directives so the task-level masking stays alongside
the module-level one.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two issues bundled because the changes overlap on the same role:
1. SQL extraction
The admin-user bootstrap and the OIDC-provider UPSERT lived inside
bash heredocs in `files/*.sh`, which Ansible invoked via the `script:`
module. The colon-prefix in `:'var'` only routes through psql's lexer
when SQL comes from stdin or `-f`, so passing it through `-c "..."` —
the original form before this branch — failed with `syntax error at
or near ":"`. Extract the SQL into `files/create_admin_user.sql` and
`files/configure_oidc_provider.sql`, embed via `lookup('file', ...)`
into an `ansible.builtin.shell` task, and feed psql via `stdin:`.
Drops the bash wrappers entirely.
2. EMAIL_* env always rendered
Fider's env loader `panic`s at startup with `the environment variable
"EMAIL_SMTP_HOST" is missing` (and friends) regardless of whether the
relay is reachable. Drop the `services.email.enabled` gate around the
SMTP block so the container starts on email-disabled stacks. The
`lookup('email', ...)` helper returns sane infinito.nexus defaults
when the service flag is off.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add tests/lint/repository/test_inline_script_block_size.py: scans
roles/, tasks/, playbooks/ and .github/ YAML for literal block scalars
(`|` / `|-` / `|+`) following the keys `shell`, `command`, `script`,
`raw` (Ansible) and `run` (GitHub Actions). Counts logical lines —
trailing-`\` continuations collapse to one line, blank lines are
ignored — and fails when a block exceeds 12.
FOLDED scalars (`>` / `>-` / `>+`) are exempt: they are one shell
statement after YAML folding, even if they span many source lines for
readability.
The walk goes through `utils.cache.files.iter_project_files(...)` per
`docs/agents/action/testing.md` so the shared project-tree cache is
reused across the lint suite.
The fail message tells the author exactly how to fix: Ansible tasks
get the built-in `script:` module (collapses the task to one line);
GitHub Actions workflows extract into `scripts/github/<name>.sh` and
call it via `run: bash scripts/github/<name>.sh` (or
`_helpers/scripts/github/<name>.sh` when the workflow checks out the
repo into `_helpers/`).
Extract the pre-existing literal-block offenders so the new lint
ships green:
Ansible (script: module, file under <role>/files/)
web-app-friendica sync_addon.sh
web-app-fider create_admin_user.sh
web-app-fider configure_oidc_provider.sh
web-app-mediawiki install_or_update.sh
web-app-mediawiki composer_install_extension.sh
web-app-xwiki debug_superadmin.sh
GitHub Actions (scripts/github/)
pick_distro.sh
guard_release.sh
resolve_release_inputs.sh
disk_pressure_report.sh
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces inline `applications`/`users` merging with dedicated lookup
plugins and runtime hydration. Introduces `add_csp_hash` filter,
integrates sort into `find_dock_val_by_bkp_entr`, splits the Keycloak
one-shot realm import into `05a_realm_import_run.yml`, and fixes the
Mastodon administrator task YAML. Removes obsolete `csp_hashes` filter,
`sys-token-store` Jinja filter, and empty CLI setup modules.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
