75 commits: the socialhome role reimplemented against the project it actually
deploys, native-login specs for funkwhale and shopware, a window.open IdP
handoff and a settings-link fallback in the shared persona helpers, a swarm
converge budget that gives up on a terminal task, and a long tail of role
fixes.
Eleven conflicts. The shared persona helpers (keycloak.js, logout.js) take the
incoming structure — they gained real behaviour this branch never had — with
this branch's resolveTimeout convention re-applied over it and
confirmKeycloakLogoutIfPrompted re-attached at every return point; without it
the Keycloak SSO session outlives the logout. playwright.config.js keeps this
branch's version: both sides set actionTimeout, but ours routes through an env
var and 15s is too short behind an onion. socialhome's spec takes the incoming
content — 2dbd6eb10 found the role described the wrong upstream project
entirely — and keeps gotoOnion and expectHstsWhenTls, since demanding HSTS over
a plaintext onion is a false failure. Both READMEs were regenerated rather than
merged by hand.
Three collisions arrived with no conflict marker at all, because each side
changed a different place in a different file:
- group_vars/all/18_swarm.yml declared swarm_stack_converge_retries twice.
Same value, so nothing breaks today; editing the documented one would have
silently changed nothing.
- svc-net-tor and web-app-socialhome both claimed 192.168.74.0/24. Socialhome,
the newer role, moves to 192.168.75.0/24.
- web-app-socialhome/meta/schema.yml arrived as a new file, so ef5ea7cb5's
rename never saw it. Its credential would not have resolved and the role
would have deployed with an unusable administrator account.
The incoming specs also needed this branch's conventions: raw timeouts wrapped,
four page.goto calls routed through gotoOnion, a request.get given an
onion-aware timeout, and 21 roles marked playwright-service-flag over the tor
block this branch added to them.
make quality green on the merge result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/schema.yml held credential definitions and their runtime values, but the
name said 'schema' and read like structural metadata next to services.yml and
volumes.yml. Nothing in the file is a schema in the sense the other meta files
use the word, and a contributor had to open it to learn it holds secrets.
62 role files renamed. ROLE_FILE_META_SCHEMA becomes ROLE_FILE_META_SECRETS and
keeps its role as the SPOT the path is built from.
Two readers bypassed that constant and spelled the name themselves:
utils/cache/applications.py built meta_dir / 'schema.yml' twice, and
plugins/lookup/application_gid.py carries the filename in the marker list it
detects application roles by. Left behind, the rename would not have raised —
credentials would have resolved empty and the role would have stopped counting
as an application.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tools.allowlist meant two things at once: the surface a consumer may reach,
which is a decision of ours, and the surface the upstream offers, which the
vendor changes without asking. Conflating them made a version bump read as a
contract violation, and left no place to record what was actually measured.
Split them. allowlist stays policy; upstream_serves records the observation at
supported_version. State the second only where it DIFFERS, because a provider
that serves exactly what is allowed would otherwise carry the same list twice
and the copies would drift apart. Absence means identical.
The difference between the two measures unenforced exposure: mattermost 103,
gitlab 2, and zero wherever the fields collapse. A non-empty difference is what
tells us a provider needs a gateway in front of it.
Require a non-empty allowlist for an enabled, non-adapter, server-capable
provider. An empty one used to make the probe's comparison vacuous, which read
as a pass while every tool the upstream serves stayed reachable. Nextcloud was
in exactly that state.
Three exemptions, all semantic: an adapter authors its tools here against a
REST upstream, so there is nothing to observe; a provider declared
enabled: false is never admitted; and upstream_serves: dynamic covers an
upstream composing its surface at runtime, which nextcloud does by proxying
MCP servers an administrator configures.
Reject an allowlist naming a tool the upstream does not serve. That found
mattermost's add_user_to_channel, gone since 11.10.0 renamed it
add_channel_member, which would have been advertised and never callable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The block sat under services but never behaved like a service. A service has
exactly one providing role, declared with provides, so the registry can map the
key to that role and consumers bind their flag to its presence. No role declares
provides: mcp. There are seventeen providers and the pairing is done by
allowed_consumers inside the block itself.
Every rule that enforces the service model therefore needed an exemption for it:
a shared-provider suppression in fourteen roles, a playwright-service-flag
suppression, and two freshly added exemptions in the dynamic-flag and
variants-static-override checks. Four independent rules, four escape hatches,
one cause. They are all gone now, together with a bond key that only the
resource budgeter reads and only for entries under services, and a wordpress
addon bridge that pointed at a service block that no longer exists.
The block moves to meta/mcp.yml. The applications loader maps meta/<name>.yml to
applications.<app>.<name> generically, so the move needs no wiring; the file
takes the same shape as meta/rbac.yml. Every config path follows, the variant
overrides are lifted out of their services map, and the vocabulary module leaves
the services package because it no longer describes one. roles_with_service
gains an explicit topic parameter rather than an implicit fallback, so nothing
silently resolves against the wrong container.
Files that are otherwise unrelated to this move are included where the committed
tree would otherwise keep pointing at a path that no longer resolves: thirteen
vars files, eleven playwright env templates and one task file.
A provider overlay keyed on services.mcp.enabled and .shared admitted every
MCP-capable role, servers included, because those flags carry no direction. Each
provider's network therefore collected every other provider and the result was a
mesh in which any sidecar could reach any application. The new mcp_client
consumer kind additionally requires services.mcp.direction to be client or both,
and the four provider overlays switch to it.
A sidecar rendered its networks with the same lookup as its provider and
inherited that provider's consumer attachments, which it neither needs nor
should have. own_network_only keeps it on the provider network alone. A provider
whose own topology is default_net contributes no attachment line at all, and a
bare networks: key is not a list, so the flag returns nothing in that case and
lets the sidecar fall back to the project default, which is that provider's own
network.
test_has_spec.py forces every roles/web-app-* to ship a Playwright spec and
calls a missing one a hard error, because the persona contract must never be
silently absent from the deploy capstone. Nothing forced the other half. The
runner discovers roles by rglob('templates/playwright.env.j2')
(discover_playwright_roles.py:53), so a spec without that file is staged
nowhere and executed never.
21 of 76 web-app roles were in exactly that state. Verified rather than
inferred: running the discovery filter directly returns 61 roles and none of
the 21 is among them, `make compose-playwright` refuses without a rendered
.env from a prior deploy, and `git log --all` on those paths shows the file
never existed on any branch -- the single hit is web-app-gitlab on a fork
branch. Their specs have never run once. 15 of the 21 deploy in CI today,
reporting green while their only functional verification never fires.
The new lint mirrors test_has_spec and carries a per-role opt-out that is
used nowhere: uniform beats exceptions.
Every one of the 21 now has a template written against its own
meta/services.yml rather than a shared skeleton -- espocrm splits
ADMIN_PASSWORD from ADMIN_NATIVE_PASSWORD because OIDC_ALLOW_ADMIN_USER
leaves both login paths open, the four permanently auth-less roles declare
their personas blocked instead of shipping a knowingly-red spec, and
web-app-mastodon gained a real test-seaweedfs.js because no honest
suppression existed for a role that does route Paperclip to S3.
web-app-magento moves pre-alpha -> alpha here: it now has a spec that runs,
which is what the lifecycle gate is about.
Six meta/services.yml files change only where adding a template falsified an
existing suppression -- gitlab and shopware both claimed the role 'ships no
templates/playwright.env.j2', which stopped being true in this commit.
Not resolved here, and the adversarial review says so plainly: the specs are
enabled, not proven. None has executed against a live stack. The review found
three classes of defect that land in follow-up commits rather than being
smuggled in silently -- persona blocks that are missing or should be tied to
sso.enabled, eleven suppressions naming a sys-svc-mail test suite that does
not exist, and eight claiming a CSP assertion that cannot fire because no
consumer role renders CSS_BASE_URL.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eight server roles declare an `mcp` RBAC role, so the platform mints
roles/<app>/mcp and the administrator joins it. Open WebUI now honours those
groups instead of flattening them to user/admin, and each MCP tool server is
restricted to the group of the role that serves it.
The grant cannot be rendered: Open WebUI matches it against a group id that
insert_new_group mints as a fresh UUID, so tasks/01_mcp.yml resolves the group
after the container is up and writes the grant through the admin API, the way
every instance-minted credential in this repository is provisioned.
Entries render disabled. An empty access_grants is not 'nobody', Open WebUI
reads it as every administrator, and ENABLE_PERSISTENT_CONFIG=false makes the
env authoritative again on every start. Enabling only together with the grant
keeps the fallback at no access rather than too much.
The provisioning account is the role-internal openwebui-api-bot, never a
platform identity: OAUTH_MERGE_ACCOUNTS_BY_EMAIL ships disabled and an OIDC
login whose email a local account already holds is refused, so seeding a
person's address would lock them out.
Known limits are documented rather than claimed away: a bare restart leaves MCP
off until the next deploy, a user's roles list carries no application context so
roles: [mcp] grants every deployed mcp group at once, and Open WebUI drops a
stale membership only while the groups claim is non-empty.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven include templates under sys-svc-container decided how a container was
probed, and they read their inputs from template-local `{% set %}` scope.
That scope leaks: a value set for one service silently reached every service
below it, which is how taiga's front and gateway ended up probing a port
declared for a different service, and how bluesky's frontend probed a PDS
endpoint.
Probes now live in utils/docker/healthcheck.py, one class per flavor behind a
PROBES registry, and each service declares its own:
healthcheck:
flavor: curl
path: health/ready
start_period: 20m
`port_key` picks another entry under ports.internal for the services that
expose their probe elsewhere (keycloak on management, nextcloud talk on
signaling). Without a flavor the entry must carry an explicit test argv, so a
missing declaration fails loudly instead of rendering a probe against nothing.
Rendering HEAD's templates against the new output confirms 49 of 56 call
sites are byte-identical; the remaining seven are covered by an Ansible run.
Three probes change on purpose: erpnext's backend gains a check it never had,
bluesky's frontend drops the leaked PDS path, and msmtp_curl moves from a
hardcoded localhost:80 to the service's own port on 127.0.0.1 like every
other flavor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Provision a personal access token carrying the mcp scope through the
rails runner. The role deploys the cloud-native images, which ship no
omnibus gitlab-rails wrapper, so the invocation names the rails binary
under /srv/gitlab/bin.
Attach clients through the role's overlay, keyed on the MCP service
flags, so a client container reaches the endpoint without publishing it.
Record what upstream does not enforce: the mcp scope grants read and
create, GitLab offers no server-side read-only mode, and the only filter
is a per-request header this deployment does not send. The tool policy in
meta/services.yml is therefore declarative, and every call carries the
blast radius of the root account. The endpoint additionally requires
Premium or Ultimate, so a Community deployment cannot reach it at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rewrite all 254 role descriptions to a uniform noun-phrase schema
(function-first, no leading verb, no trailing period, <=120 chars) and
enforce it via the extended description lint. Rename 12 role README H1
titles for clarity (e.g. Backup Container to Local -> Backup Docker
Volumes, CSS Service -> Corporate Design, NGINX Legal -> Imprint) and
regenerate their Cosmos and Quick Setup sections. Overview table
regenerated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add cli.build.readme.overview (make readme-index / make docs) rendering a
roles overview in the root README: Name, Lifecycle, Homepage, Video,
Description, Integrated with, Install. New info-media lint requires
homepage + video in every invokable role's meta/info.yml (opt out via
'# nocheck: info-media' in meta/main.yml); backfilled 147 roles with
researched, adversarially-verified homepages and official videos.
Retire the 'unsupported' lifecycle value: fold it into 'eol' as a valid
live end-of-life status (shipped but untested/unmaintained) across the
stage lists, symbol glossary, allow-list lint and docs. Retag confluence,
jira, minio, phpldapadmin to eol and magento to pre-alpha.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Per-role deploy-mode opt-out moves from meta/tests.yml 'skip:' to meta/services.yml <entity>.modes.{compose,swarm}.enabled on the primary entity. get_role_skip now derives from it; get_role_mode_enabled added. A new lint requires every role shipping templates/*compose*.yml.j2 to declare both mode flags. Migrates 107 services.yml and drops 11 skip-only tests.yml. Also removes the variant_status deployability gate so swarm runs one job per variant unconditionally (variant_status.py and deployable_variant_indices deleted, complexity model simplified). Adds the i8matix alias.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bump every outdated image pin and repository ref to latest except
web-app-nextcloud (kept on 33-fpm-alpine). Covers keycloak, gitlab (6
services), matrix, opentalk, jira, confluence, espocrm, magento
opensearch, elasticsearch, rabbitmq, qdrant, ollama, seaweedfs,
semaphore, wordpress, taiga, bluesky (image + social-app ref), bookwyrm
ref, roulette-wheel, simpleicons, registry cache/docker, and the
svc-runner base image. Includes major jumps (elasticsearch 9, rabbitmq
4, opensearch 3, espocrm 10, registry 3, node 26, ubuntu 26.10) that
are pin-only and still need a deploy validation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename the role, its test tree, the primary entity key
(volume-2-local), the derived vars (BKP_VOLUME_2_LOCAL_*,
SYS_SERVICE_BACKUP_VOLUME_2_LOCAL), the run-once fact, and every
group_names consumer flag, include path, doc, and lint reference. The
container_backup service key and the on-disk backup-docker-to-local
tree keep their names: both are contracts with existing consumer
entries, snapshot trees, and the baudolo pull tooling.
Operators must rename the svc-bkp-container-2-local inventory group and
disable the previous versioned units on already-provisioned hosts; the
pre-deploy snapshot glob derives its base from the new role name and
will not find them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Variant 0 pins the backup consumer flag enabled so the matrix baseline exercises the in-group branch; every other variant pins it disabled (shared: false carries the static-override suppression since the base value stays literally true). The matrix and wordpress variant entries move their services mapping to the list-item head so the static-override line parser resolves the override paths; yaml semantics are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every invokable stack role that declares named volumes in meta/volumes.yml now carries the standard container_backup consumer entry: enabled resolves via 'svc-bkp-container-2-local' in group_names, shared stays literally true since there is no per-consumer backup instance. Playwright/dynamic-flag/bond-baseline suppressions ride on the block per the established idioms; stale explanatory comments in the touched files are dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A node-local volume does not follow a swarm reschedule, so its data is lost on drain. Add a meta/volumes.yml lint that fails when a 'nfs: false' opt-out lacks both '# nocheck: nfs-false-data-loss' and a '# Reason:' arguing why that loss is acceptable. Justify the 10 existing entries (checkmk OMD site; gitlab repositories, logs, runtime state).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add registry.gitlab.com to the mirrorable registries so discovery
yields the GitLab CNG images to the version updater and mirror sync.
Seed the generic v2 tag scan with the OCI last cursor for v-prefixed
pins and raise the page size to 1000: CNG repos carry tens of
thousands of commit-sha tags that sort before the release tags, so an
unseeded scan exhausted the page cap without reaching any semver tag.
Pin every gitlab service version in meta/services.yml and drop the
GITLAB_VERSION template override so services.yml is the single source
for per-service versions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CNG's set-config copies the rendered configs into CONFIG_DIRECTORY on
every container start; the first cp creates the destination with the
source mode, so 0440/0400/0444 sources left the copy owner-unwritable
and the next start's truncating cp died with EACCES (permanent
crash-loop after any restart, including the deploy's own recreate).
Grant owner-write to exactly the files set-config copies; files bind-
mounted read-only at their final destination stay tight.
Mount the CNG images' declared VOLUME paths (/var/log*, /var/opt/gitlab)
as named node-local volumes so containers stop spawning anonymous
volumes that fail the hlth-volumes check.
Verified live: HTTP 200, all services healthy across recreate,
hlth-volumes Result=success.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deploy GitLab from the official CNG CE images as six services: webservice (puma) and sidekiq scale to the topology default, workhorse is the sole HTTP entry, gitaly and a one-shot migrations job are pinned singletons, shell runs gitlab-sshd. The Omnibus container, its Dockerfile and the GITLAB_OMNIBUS_* config groups are removed. All rails/workhorse/shell/gitaly configs and shared secrets are pre-rendered by Ansible and mounted read-only; eleven credentials land in meta/schema.yml including an rsa_pem_2048 OIDC signing key (new value_generator algorithm). Postgres, redis and S3 object storage ride the existing lookups; OIDC, SMTP, the public SSH port and the root password are preserved. compose_volumes gains a per-volume nfs:false opt-out so the gitaly repositories volume stays node-local in swarm (upstream forbids git repositories on NFS); nfs_prep skips volumes without driver_opts accordingly. The Nextcloud gitlab integration execs the CNG rails runner. ruby_single_line dies with its only consumer; ruby_dq keeps test coverage. Ports (8181/8080/2222/8075/3808/9122) and service names resolve from single SPOT declarations in meta/services.yml via vars constants; the version pin sits on the webservice image pair so the updater can bump it. Live-validated in compose up to all six services healthy and migrations completed; fresh-install-only semantics for existing Omnibus data are documented.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
make autoformat output across role meta files, a task file and 19_tor.yml (single->double quotes on mode values, Jinja line reflow). No semantic changes. Kept separate from the onion CSP / oauth2-proxy fixes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Each role's meta/variants.yml now pins the tor onion gate both ways so the matrix exercises it: variant 0 pins services.tor.enabled: true, every non-baseline variant pins enabled: false + shared: false (# nocheck: variants-static-override, since tor.shared is a static literal). This gives real test_coverage / test_bond_baseline coverage without the removed per-key code exemptions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Domain-serving roles reference the svc-net-tor provider via a tor: block. Consumers declare enabled (dynamic group_names gate) and shared: true (static — onion is always the shared node model, inline nocheck: dynamic-flag). The provider svc-net-tor additionally declares exclusive/primary defaults that consumers inherit via resolve_service_config unless overridden.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extract the csp, domains, and networks blocks out of roles/*/meta/server.yml
into dedicated meta files, each mapping to its own top-level applications key
via the recursive meta-walk:
csp: -> meta/csp.yml -> applications.<app>.csp
domains: -> meta/domains.yml -> applications.<app>.domains
networks: -> meta/networks.yml -> applications.<app>.networks
Residual nginx/proxy keys (status_codes, client_max_body_size, locations,
ip6_enabled, config_upstream_url) stay in server.yml; an emptied server.yml is
deleted. 98 server.yml -> 249 new files, 74 deleted.
- Consumers repointed applications.<app>.server.{csp,domains,networks} ->
applications.<app>.{csp,domains,networks} across templates, vars, tasks,
lookups and filters.
- Infra readers migrated: canonical_domains_map, application_domain_index,
build_ldap_role_entries, service-registry overlay discovery,
roles_with_service, and the file-based readers utils/domains/list.py +
utils/meta/scan.py (read the now-deleted server.yml directly).
- Detection + mapping: _META_TOPICS gains csp/domains/networks;
ROLE_FILE_META_{CSP,DOMAINS,NETWORKS} constants + ROLE_FILES entries.
- New self-expiring lint guard (test_server_meta_no_split_fields) fails if a
server.yml still carries a split field; auto-fails from 2026-12-28 as a
delete reminder.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-role swarm overlay local subnet was a /28 (14 usable IPs). A role's swarm stack plus its sidecars and per-task network attachments can need more than that, which surfaces as 'node is missing network attachments, ip addresses may be exhausted' and leaves services (sso-proxy, app frontends) unresolvable mid-deploy. Re-allocate all 74 /28 roles to a unique /24 (254 usable) inside 192.168.0.0/16, chosen via the network address suggester so the assignments stay non-overlapping. Validated by tests/integration/infrastructure/networks/test_unique_valid_and_mapped.py and a full make test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Conflicts resolved keeping the branch's structure and taking main's version bumps: web-app-pihole kept the branch's lookup-based implementation (container_image/ports/volumes plus compose_volumes/networks lookups and meta/volumes.yml); web-app-erpnext took main's image version v16.25.0; web-app-gitlab took main's 19.1.1-ee.0; CHANGELOG took main's [10.1.1] entry. The branch's swarm-nfs structural changes (erpnext internal ports/proxy/backend, gitlab redis) were preserved. make test is green on the merged tree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Generalize tests/lint/roles/test_swarm_internal_ports.py from http-only to every
published port kind (http, websocket, ssh, ...) referenced in a compose port
mapping (ports.<local|public>.<kind>): each must declare the matching
ports.internal.<kind> so the role's swarm port surface is complete and
consistent. Fix the roles it flagged:
- pihole, shopware: ports.internal.http (80, 8000)
- erpnext, mastodon, odoo: ports.internal.websocket (9000, 4000, 8072)
- gitea, gitlab: ports.internal.ssh (22)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
