A pin like 'alpine', 'stable', 'lts' or 'buildx-stable-1' names no version,
so nothing can compare it to anything: the bump job skips it and the pin
drifts with whatever upstream last pushed under that name. Each of these
fifteen now names its release, which is the precondition for the freshness
pipeline to see it at all.
Every conversion was placed against the tag it replaces, and none of them
is a downgrade. Six resolve to a byte-identical digest and are pure
relabelling (memcached, redis, the two debian bases, jenkins, xwiki); four
are genuinely newer than what the moving tag carried, so those roles will
deploy a later version than before (nginx moves past the 1.31.5 behind
'alpine', buildkit past the v0.32.2 behind 'buildx-stable-1', dockerize
past a 'latest' older than v0.10.0); openresty and pretix keep their
version and differ only by build.
fider is the one that loses something: 'stable' carries a rebuild pushed
twelve days after v0.36.0, which is still the newest versioned tag
upstream publishes. An unnamed rebuild cannot be tracked, so the pin takes
the named release and the updater can raise it when v0.37.0 lands.
The flavor invariant keeps the families apart: jenkins stays on '-lts' and
xwiki inside '-postgres-tomcat', so no bump can cross into a different
runtime.
svc-runner loses a 'nocheck: docker-version' that has become dead. The
marker suppresses two checks at once, so while it stood buildkit was held
out of the daily freshness warning as well; removing it puts the role back
in.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Thirteen of sixteen MCP sidecar entries carried no enabled at all, so
collect_role_resources fell back to _looks_like_container and billed every
one of them as running. Three roles already gated theirs on mcp.enabled;
the rest now do the same, gitea twice because it is the only role with a
vendor container behind the adapter.
The fifteen adapter roles then read services.<key>.enabled instead of
mcp.enabled, so the compose block and the task gates follow the services
entry rather than running beside it on a second path. The six plugin and
native roles keep reading the topic: they ship no sidecar entry to point at.
The three-level resolution is not new. KEYCLOAK_LOGOUT_PANEL_ENABLED has
the same shape and depth and gates a task the same way, and the same value
reaches JAVASCRIPT_SERVICE_ENABLED, where service-gating.js throws on
anything but "true" or "false" — an unresolved chain would have been
killing keycloak's Playwright run since 575a5a87ac.
This does not change what the resource model counts: _is_enabled reads the
unrendered Jinja and still treats it as on. What it buys is one switch
instead of two, and a declaration that matches it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
83c1c729de gave every provider a group_names-gated services entry per MCP
client, including the clients it refuses with mcp_consumer: false. The gate
is what makes the inventory resolver plan provider and client into one round,
so 19 of the repo's 82 co-deploy pairs started a client for a pairing the
provider had already declined. flowise is a live MCP client of qdrant,
homeassistant and prometheus; the other 17 providers were bringing up its
node stack for nothing.
The entries now read enabled: false, shared: false, mcp_consumer: false, and
the variants that pinned them to true are gone. Setting those variants to
false instead is not an option: test_variants_no_static_override forbids any
variant override of a literal base value in either direction.
bond leaves with them, and a new lint holds that line: collect_role_resources
skips a literally disabled entry before it reads any resource key, so a bond
there describes a cost nobody pays. Thirteen entries unrelated to MCP carried
one and are cleared too.
test_mcp_consumer_services_entry demanded the group_names shape for every
client entry, refused ones included, and now demands the opposite there.
dynamic-flag keeps its rule and takes its own documented per-key exemption.
make docs redraws twenty graphs: the edges go, the nodes stay as
'openwebui ❌' and 'flowise ❌', so the refusal is visible rather than read as
an integration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
allowed_consumers was twenty hand-kept lists of which fourteen were identical, so every new client meant an edit in every provider. A client now declares services.<own-entity>.mcp_consumer: true once and a provider refuses one with mcp_consumer: false, which is also what tells a deliberate exclusion apart from a forgotten entry.
The group_names-gated consumer entries give the inventory resolver the co-deploy edge meta/mcp.yml never had: one of sixty-nine matrix rounds carried an MCP client before, twenty-three role/round pairs do now. bond 0.5 keeps the clients out of round 0 so the host memory budget still holds.
BREAKING CHANGE: meta/mcp.yml no longer accepts allowed_consumers; declare services.<entity>.mcp_consumer on the client instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
72fd9c870 required the onion in every variant but exempted variant 1, on the
argument that the minimal round is the one place the stack runs without it. That
argument does not hold: the onion is part of the deployment under test, not a
feature bolted onto the full shape. A round that skips it is not a smaller
deployment, it is a different one.
So the exemption is gone and 95 variant-1 entries pin the onion on. Two
exclusions remain, both read from data rather than a comment: a role with no tor
block has no onion to pin, and a role whose meta/services.yml pins tor.enabled to
a literal false has already declared it has none.
That put the rule in conflict with test_coverage, which requires both polarities
of every dynamic flag: with tor on everywhere, the false side has no variant left
to live in. 94 roles take the exemption that contract already provides for
exactly this - "flags whose group-derived value is a hard invariant that variants
must never override", the same one 4e6ef2542 used for the seaweedfs frontend.
web-app-nextcloud needs none: its variant 8 pairs the clearnet-only PeerTube and
keeps the false polarity.
Measured, because the declaration is not what CI runs. assign_tor gives the onion
to every second capable entry with the run number as the parity switch, so more
capable variants do not mean proportionally more onion jobs: across five roles
and 21 entries the count goes 8 -> 10 on an even run and 9 on an odd one. The
clearnet path also survives - checkmk reports v1=false on run 0 and v1=true on
run 1, so both states are still covered across two consecutive runs. What the
94 markers give up is a declarative guarantee in variants.yml, not the coverage
itself.
The real cost is weight, not job count. web-app-nextcloud variant 1 goes from
3.09GB reservation / 7.27GB limit to 18.26GB / 43.01GB once svc-net-tor and its
closure join, so the cheap round is no longer cheap. Every variant stays under
the 32/64GB budget, and the same effect applies to the other 94 unmeasured.
Verified: tests/lint 336 passed / 307 subtests, tests/integration/roles/meta/
variants plus the role-meta lints 39 passed. The rule was checked in both
directions - it reported the variant-1 offenders before the flips and none after.
Not measured: what the heavier rounds do to wall-clock, against swarm jobs capped
at 355 minutes and one that has hit that ceiling before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every variants.yml header states the rule the bond feeds: variant 0 carries each
tightly-coupled (bond>=1, same-host) service, while loosely-coupled (bond<1,
separate-host) partners are excluded from the closure and distributed across
later variants to keep every host under budget. tor was declared bond: 1 in all
101 consumers, so it had to sit in every variant-0 closure.
It does not belong there. The onion is a separate-host provider reached over the
node's hidden service, the same shape as the bridged partners that already carry
0.5, and f1dff50fb established both the reasoning and the precedent when it
dropped xwiki and bigbluebutton from 1 to 0.5: bridged partners with their own
role and their own spec, whose bond declaration was the outlier rather than
their placement.
This also unblocks the variant-level tor contract. Requiring a non-baseline
variant to pin the onion on while tor still counted as tightly coupled would
have pulled variant 0's closure requirements along with it.
Verified: tests/lint 336 passed / 307 subtests and tests/integration/roles 107
passed / 610 subtests, including test_resource_budget over every variant's
deduplicated footprint and the bond-aware test_order closure.
Not measured, and the honest cost: f1dff50fb's own caveat applies unchanged
here. Whether redistributing a service out of the baseline closure buys back
wall-clock is not established; role-runtimes has the per-role seconds and no run
has been made with this shape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The adapter was the only image in the tree pinned by a hand-maintained digest,
and the pin was never recomputed by anything. It drifted twice: run 31374959721
lost seven of eight jobs to mirror/docker.io/python@sha256:6771159c... not
found, and the renewal to the newest upstream digest earlier today would have
reproduced it, because builds resolve FROM against the GHCR mirror while the
mirror is copied by tag and lags upstream by design.
The base is now FROM <image>:<version>, taken from the sidecar service in
meta/services.yml - the only file iter_role_images scans, so the tag the build
asks for is by construction the tag the mirror was told to copy. The version
moves from the moving 3.13-slim to 3.13.15-slim, which upstream stops rewriting
once published, and which the existing version drift job renews like every other
image.
With the build reading services.yml, the adapter block's digest, image and
version had no consumer left. They are removed together with the schema check
for the digest form, the warn-only drift test, and the manifest_digest helper
that existed only for it. What remains in the block describes the adapter rather
than the image it is built from.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
container_hostname is computed for sethostname(2) and clamped to 63 chars
(plugins/lookup/container_hostname.py). pretix's onion domain is 74, so the
lookup degrades to the bare entity name 'pretix' -- and container_healthcheck
forwards that value into the curl probe's Host header.
pretix gates on Host in MultiDomainMiddleware, not in ALLOWED_HOSTS (settings.py
hardcodes ALLOWED_HOSTS = ['*']). It accepts the site domain, a KnownDomain, or
one of LOCAL_HOST_NAMES = ('testserver', 'localhost'); everything else gets a
400. The probe could therefore never pass, swarm killed the task with exit 137
and the stack never converged -- Pretix 0 of run 31340965955.
A/B against pretix/standalone, same volume and same PRETIX_PRETIX_URL, probe run
by Docker itself, only the header differing:
Host: pretix -> curl (22) 400 x4, unhealthy streak=6
Host: localhost -> exit 0 x3, healthy
The Tcp flavor already hardcodes Host: localhost for the same reason; curl was
the outlier. Reusing a kernel hostname as an HTTP Host is the underlying design
bug, but defaulting the curl probe would change six roles' probes at once, so it
stays a separate change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ten provider roles served an MCP surface that other roles consume, but
were never declared as shared services, so the registry did not know
them. Declaring them pulls in the recursion contract every registry
service owes: seven roles move their entry tasks into tasks/00_core.yml
behind the run_once flag, so the service loader cannot include them a
second time.
pretix already had a run_once block, but set the flag last; the flag now
comes first, before the body it protects.
The block sat under services but never behaved like a service. A service has
exactly one providing role, declared with provides, so the registry can map the
key to that role and consumers bind their flag to its presence. No role declares
provides: mcp. There are seventeen providers and the pairing is done by
allowed_consumers inside the block itself.
Every rule that enforces the service model therefore needed an exemption for it:
a shared-provider suppression in fourteen roles, a playwright-service-flag
suppression, and two freshly added exemptions in the dynamic-flag and
variants-static-override checks. Four independent rules, four escape hatches,
one cause. They are all gone now, together with a bond key that only the
resource budgeter reads and only for entries under services, and a wordpress
addon bridge that pointed at a service block that no longer exists.
The block moves to meta/mcp.yml. The applications loader maps meta/<name>.yml to
applications.<app>.<name> generically, so the move needs no wiring; the file
takes the same shape as meta/rbac.yml. Every config path follows, the variant
overrides are lifted out of their services map, and the vocabulary module leaves
the services package because it no longer describes one. roles_with_service
gains an explicit topic parameter rather than an implicit fallback, so nothing
silently resolves against the wrong container.
Files that are otherwise unrelated to this move are included where the committed
tree would otherwise keep pointing at a path that no longer resolves: thirteen
vars files, eleven playwright env templates and one task file.
Seven include templates under sys-svc-container decided how a container was
probed, and they read their inputs from template-local `{% set %}` scope.
That scope leaks: a value set for one service silently reached every service
below it, which is how taiga's front and gateway ended up probing a port
declared for a different service, and how bluesky's frontend probed a PDS
endpoint.
Probes now live in utils/docker/healthcheck.py, one class per flavor behind a
PROBES registry, and each service declares its own:
healthcheck:
flavor: curl
path: health/ready
start_period: 20m
`port_key` picks another entry under ports.internal for the services that
expose their probe elsewhere (keycloak on management, nextcloud talk on
signaling). Without a flavor the entry must carry an explicit test argv, so a
missing declaration fails loudly instead of rendering a probe against nothing.
Rendering HEAD's templates against the new output confirms 49 of 56 call
sites are byte-identical; the remaining seven are covered by an Ansible run.
Three probes change on purpose: erpnext's backend gains a check it never had,
bluesky's frontend drops the leaked PDS path, and msmtp_curl moves from a
hardcoded localhost:80 to the service's own port on 127.0.0.1 like every
other flavor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three conflicts, none resolved by picking a side wholesale:
- Makefile: took the incoming ACT_INPUTS, which routes through SWARM_DISTROS
instead of inlining the same expression a second time.
- test-e2e-playwright/02_run_one.yml: took the incoming rc handling. An absent
rc now counts as failure; the previous default(0) let an aborted Playwright
run pass silently, contradicting the debug message right above it.
- web-app-peertube spec: combined both. Kept gotoOnion and resolveTimeout,
which this branch's lint enforces, and adopted the incoming retry-inside-poll
for the OIDC button plus its networkidle wait.
The declared values were guesses that ran up to 60x the size of the image they belong to: seaweedfs asked for 30GB against a 104MB image, nextcloud for 20GB against 353MB. Because min_storage feeds the storage guard, the runner packer and the sufficient-storage filter, those numbers made hosts look too small and pruned caches that deploys did not need to lose.
Every service that declares min_storage together with image and version now carries the measured compressed transfer size of that image plus 200MB of headroom. Sizes were read from the registries through utils.docker.registry.manifest_transfer_size, the same path the headroom test uses, so the declarations and the gate agree by construction. Values are written in MB throughout, matching the existing 500MB / 100MB / 50MB style.
119 declarations across 89 roles. Range: 206MB (memcached) to 3473MB (ollama); four exceed 1GB. The 84 min_storage entries that name no image are untouched, since there is nothing to measure them against.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Per-role deploy-mode opt-out moves from meta/tests.yml 'skip:' to meta/services.yml <entity>.modes.{compose,swarm}.enabled on the primary entity. get_role_skip now derives from it; get_role_mode_enabled added. A new lint requires every role shipping templates/*compose*.yml.j2 to declare both mode flags. Migrates 107 services.yml and drops 11 skip-only tests.yml. Also removes the variant_status deployability gate so swarm runs one job per variant unconditionally (variant_status.py and deployable_variant_indices deleted, complexity model simplified). Adds the i8matix alias.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The custom flag moves from 33 hardcoded template kwargs into the merged
service config: 'custom: true' names the shared <entity>_custom build,
a string names a second distinct '<string>_custom' image inside one app
(nextcloud whiteboard). The image lookup reads the declaration as a
fallback, so config consumers (e.g. backup image matching) resolve the
exact deployed reference from one declaration point. nextcloud cron now
runs the nextcloud_custom build like the app service instead of the
upstream image.
A new services lint couples the declaration to reality in both
directions: custom requires a role Dockerfile, and a role Dockerfile
requires a custom declaration. The eight legacy roles deploying local
builds under upstream tags or ad-hoc image= overrides are ratcheted via
'# nocheck: dockerfile-custom'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rename the role, its test tree, the primary entity key
(volume-2-local), the derived vars (BKP_VOLUME_2_LOCAL_*,
SYS_SERVICE_BACKUP_VOLUME_2_LOCAL), the run-once fact, and every
group_names consumer flag, include path, doc, and lint reference. The
container_backup service key and the on-disk backup-docker-to-local
tree keep their names: both are contracts with existing consumer
entries, snapshot trees, and the baudolo pull tooling.
Operators must rename the svc-bkp-container-2-local inventory group and
disable the previous versioned units on already-provisioned hosts; the
pre-deploy snapshot glob derives its base from the new role name and
will not find them.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every invokable stack role that declares named volumes in meta/volumes.yml now carries the standard container_backup consumer entry: enabled resolves via 'svc-bkp-container-2-local' in group_names, shared stays literally true since there is no per-consumer backup instance. Playwright/dynamic-flag/bond-baseline suppressions ride on the block per the established idioms; stale explanatory comments in the touched files are dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Onion routing is transport-layer with no persona/UI surface, so tor cannot be gated from the per-persona Playwright specs. Mark the tor block with '# nocheck: playwright-service-flag' in the 60 domain roles that ship a playwright.env.j2, per test_env_services_match's documented exemption for services that legitimately cannot be Playwright-gated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Domain-serving roles reference the svc-net-tor provider via a tor: block. Consumers declare enabled (dynamic group_names gate) and shared: true (static — onion is always the shared node model, inline nocheck: dynamic-flag). The provider svc-net-tor additionally declares exclusive/primary defaults that consumers inherit via resolve_service_config unless overridden.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
About 38 web-app/web-svc roles now consume the central svc-db-* and svc-dns engines via lookup('engine', ...) instead of embedded sidecars: meta/services flags, variant matrices, and env/conf hosts point at the shared services. Lets these apps run unpinned and NFS-shareable in swarm.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`bond` is a 0..1 factor for how strongly a service is tied to the overall
integration: bond: 1 = always required, bond: 0 = omittable. Groundwork for
tooling that later scores service importance and decides what can be dropped.
Named `bond` (not `bound`) to avoid clashing with the resource bounds/limits.
Add test_service_bond.py (lint): every enabled + shared service must declare a
valid `bond` (number in 0..1). Seed `bond: 1` as a placeholder on all 638
enabled+shared service entries across 88 roles, to be refined later.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Extend test_compose_resource_limits.py to enforce resource limits on every
container a role runs itself (not provided by another role via the service
registry): the primary entity needs min_storage/cpus/mem_reservation/mem_limit/
pids_limit; uncovered sidecar containers need the four resource keys. A service
is exempt only when another role covers it (e.g. discourse's postgres ->
svc-db-postgres, whose entity declares the limits), replacing the previous
blanket shared:True skip.
Declare the previously-missing limits across 18 roles: mostly self-run redis
cache sidecars (cpus "0.5" / 128m / 512m / 512), plus the minio mc client,
the coturn gateway, and the web-svc-* css/html/asset/cdn/file provider entities.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds playwright.env.j2 for mediawiki/pretix/xwiki (admin/biber creds, SSO/LDAP/service gates, addon_env_flags) and extends nextcloud's env with INTEGRATION_OPENAI_EXPECT_API_KEY and NEXTCLOUD_ONLYOFFICE_EXPECTED_DOCUMENT_SERVER_URL. services.yml nocheck flags for the partner DB/CSS services those roles expose.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reconcile main's object-store feature and pre-swarm deploy tasks with the branch's compose/networks/volumes SPOT and swarm lints.
objstore (seaweedfs/minio): standalone compose + embedded service templates migrated to the container_image/container_volumes/container_networks/compose_volumes/compose_networks/depends_on lookups; meta/volumes.yml on the canonical dict schema with s3.json and nginx.conf as type:config; embedded services made swarm-compatible via compose_only/compose_restart.
deploy tasks (erpnext, opentalk, seaweedfs, shopware, snipe-it, wordpress, yourls): container exec routed through the container_address resolver and gated on IS_STACK_HOST, shell module plus stack_host_template, no chdir or compose-verb forms.
Also removed orphaned vars, restored MAILU_INSTANCE_DIR, freed the seaweedfs subnet (192.168.206.0/24), stripped literal quotes from the nextcloud/bookwyrm env templates, and routed main's objstore compose_volumes tests through dir_var_lib.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Give each reverse-proxied app a swarm-resolvable upstream: add services.<entity>.ports.internal.http to the role metadata and rename the main compose service from the generic 'application' to the entity name, so its short DNS alias on the shared overlay is unique. Thread the service name through compose, vars (*_SERVICE_NAME), volume mounts and the oauth2 origin host. Consolidate the split image/ports blocks in bridgy-fed and postmarks into the entity block.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pretix-oidc declared as mechanism: plugin with the 2.3.1 pin as a string version and bridges [sso].
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Neither role can consume the shared SeaweedFS object store, so both keep their
seaweedfs-required exemption — but with reasons that state the real blocker so
nobody retries them blindly:
- discourse: aws-sdk-s3 defaults a custom endpoint to virtual-host addressing
and exposes no force_path_style option; incompatible with SeaweedFS path-style
unless SeaweedFS is switched to virtual-host/wildcard-DNS globally (which would
break the path-style consumers). Uploads stay on the local volume.
- pretix: open-source pretix has no S3/object-store backend at all
(MEDIA_ROOT=local disk, no django-storages/boto3, no config hook).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a meta lint asserting every web-app role with postgres or mariadb declares a seaweedfs object-store service, exemptable via the seaweedfs-required nocheck marker. Mark the 33 database-backed roles that do not use object storage, and register the rule in the suppression catalog.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Neither role can consume the shared SeaweedFS object store, so both keep their
seaweedfs-required exemption — but with reasons that state the real blocker so
nobody retries them blindly:
- discourse: aws-sdk-s3 defaults a custom endpoint to virtual-host addressing
and exposes no force_path_style option; incompatible with SeaweedFS path-style
unless SeaweedFS is switched to virtual-host/wildcard-DNS globally (which would
break the path-style consumers). Uploads stay on the local volume.
- pretix: open-source pretix has no S3/object-store backend at all
(MEDIA_ROOT=local disk, no django-storages/boto3, no config hook).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a meta lint asserting every web-app role with postgres or mariadb declares a seaweedfs object-store service, exemptable via the seaweedfs-required nocheck marker. Mark the 33 database-backed roles that do not use object storage, and register the rule in the suppression catalog.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements docs/requirements/021-sso-flavor-migration.md atomically:
* Schema: every role's meta/services.yml replaces the parallel
`oauth2:` / `oidc:` top-level blocks with a single `sso:` block
carrying `enabled`, `shared`, and a `flavor` discriminator
(`oidc` | `oauth2` | `saml`; default `oidc`). Flavor-specific
sub-keys live under `sso.<flavor>.*` (`sso.oauth2.{origin,acl,
allowed_groups}`, `sso.oidc.plugin`).
* Provider role: roles/web-app-oauth2-proxy/ is deleted in its
entirety; its 5 sidecar templates move under
roles/web-app-keycloak/templates/sso_proxy/, and the per-consumer
rendering task lives at roles/web-app-keycloak/tasks/sso_proxy.yml.
web-app-keycloak now declares `provides: sso`.
* Runtime: new resolver utils/roles/applications/services/sso.py
centralises the SSO predicates (`is_enabled`, `is_proxy_gated`,
`is_oidc_native`) and the oauth2 sub-values
(`oauth2_origin_{host,port}`, `oauth2_acl`, `oauth2_allowed_groups`).
Three lookup plugins consume the resolver: `sso` (single-app
predicate access), `sso_proxy_consumers` (enumerates oauth2-flavored
consumers), `sso_oidc_plugin` (renamed from `oidc_flavor`, Nextcloud
plugin selector). sys-stk-backend, sys-svc-compose, sys-svc-proxy
and plugins/filter/compose_volumes.py all gate on
`lookup('sso', ..., 'is_proxy_gated')` instead of manually combining
`enabled` and `flavor`.
* Identifier renames repo-wide: `services.<entity>.ports.local.oauth2`
→ `ports.local.sso`; `PORT_BANDS.local.oauth2` → `PORT_BANDS.local.sso`;
`OAUTH2_PROXY_*` env/Jinja vars → `SSO_PROXY_*`;
`oauth2_proxy_cookie_secret` → `sso_proxy_cookie_secret`;
`OAUTH2_SERVICE_ENABLED` + `OIDC_SERVICE_ENABLED` collapse into
`SSO_SERVICE_ENABLED`; Playwright gating helpers
(isServiceEnabled/skipUnlessServiceEnabled/requireService/safeIsEnabled/
safeSkipUnlessEnabled) move from `oidc`/`oauth2` to `sso`.
* Dual-block resolution: per req's Background-Table, web-app-bookwyrm
is `flavor: oauth2`; web-app-gitea and web-app-friendica keep
`flavor: oauth2` with their existing `oauth2.acl.blacklist`
(/user/login) and `oauth2.acl.whitelist` (federation + discovery
endpoints) — preserving the operative oauth2-proxy gating.
* Tests: the two mutual-exclusion guards
(tests/integration/iam/oauth2_oidc/test_mutual_exclusive.py and
test_acl_mutual_exclusive.py) are deleted per Decision #10;
test_oauth2_contract.py is renamed and rewritten to
test_sso_contract.py, asserting that `flavor: oauth2` roles carry
`oauth2.origin.{host,port}` plus `ports.local.sso`; test_sso.py
(web-app integration) now requires the unified `sso:` block with an
explicit flavor or a `# nocheck: sso` opt-out; new
tests/lint/repository/no_legacy_sso_paths/ guards the source tree
against re-introducing the legacy strings.
* Docs: req 21 itself has every AC checked off, and reqs 006, 008,
009, 013, 015, 016, 019, 020 plus 10+ role READMEs and
docs/contributing/actions/testing/suppression.md are updated to the
unified SSO naming.
Empirically validated end-to-end against the validation app set
(web-app-keycloak, web-app-nextcloud, web-app-bookwyrm, web-app-gitea,
web-app-friendica, web-app-dashboard) in matrix variants 0 and 1; v2
also passes the 4 target roles' Playwright after a documented
pre-existing KC_HOSTNAME-drift workaround (separate concern, not
caused by this refactor).
BREAKING CHANGE: every consumer of `services.oauth2.*`,
`services.oidc.*`, `OAUTH2_PROXY_*`, `OAUTH2_SERVICE_ENABLED`,
`OIDC_SERVICE_ENABLED`, `oauth2_proxy_cookie_secret`,
`oauth2_proxy_application_id`, `ports.local.oauth2`, the
`web-app-oauth2-proxy` role, or the legacy gating helper service names
(`oidc`/`oauth2`) must migrate to the unified `sso` shape documented
in docs/requirements/021-sso-flavor-migration.md.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Decouples the CSS service concept from its underlying transport
(currently `web-svc-cdn`) so the flag becomes opt-in via inventory
group membership instead of staying literal-`true` everywhere.
- `roles/web-svc-css/`: new role. Marker / wrapper role that owns
the `css` entry in the service registry, declares `web-svc-cdn` as
the upstream that serves the actual bytes, and 301-redirects its
own canonical domain to the cdn domain so health probes return a
valid response.
- `roles/web-svc-cdn/meta/services.yml`: drop the `css` block — the
canonical owner is now web-svc-css; cdn keeps `javascript`.
- 51 consumer roles (web-app-* and web-svc-{coturn,file,html,
libretranslate,logout,onlyoffice}) migrated their `css:` block from
literal `{ enabled: true, shared: true }` to dynamic
`{ enabled: "{{ 'web-svc-css' in group_names }}", shared: ... }`,
with matching matrix coverage added to each role's
`meta/variants.yml` (true-pinning + false-pinning entries) so the
dynamic-flag variant lint stays green.
- 41 `# nocheck: playwright-service-flag` rationale strings updated
from the now-incorrect "canonical:cdn alias of static-asset
provider" to "css consumer, no Playwright surface (provider:
web-svc-css)".
Bundled refactor: `roles/web-svc-asset/tasks/01_core.yml` now
includes `web-opt-rdr-domains` directly with a single-entry mapping
to redirect its canonical domain to web-svc-file, instead of having
that wiring live in the global `tasks/stages/01_constructor.yml`
constructor. The `add_redirect_if_group` filter
(`plugins/filter/redirect_filters.py`) and its unit test are dropped
as dead code — both call sites are now role-local.
Dashboard's consumption of the new pattern is intentionally not part
of this commit and follows separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* scripts/lint/autoformat.sh: rename AUTOFORMAT_PARALLEL to PARALLEL,
declare it at file start with default `true`; still env-overridable.
Reword the worker comment so the literal `# shellcheck` token no
longer appears at start-of-line and trips SC1072/SC1073.
* scripts/lint/galaxy.sh: drop the GALAXY_FATAL / GALAXY_NAMESPACE /
GALAXY_PARALLEL env vars (over-engineered for a single repo). Failure
is always fatal. NAMESPACE is an internal constant. Add the same
PARALLEL env var (default `true`, env-overridable); falsy demotes the
xargs fan-out to one worker.
* docs/contributing/environment/variables.md: collapse the four
AUTOFORMAT_*/GALAXY_* rows into one `PARALLEL` row covering both
consumers.
* docs/contributing/tools/make.md: refer to `PARALLEL=0` everywhere;
add the lint-galaxy parallelism note.
Plus working-tree drift across roles/, group_vars/, inventories/, and
tasks/ that accumulated from the make-test setup phase regenerating
generated artefacts during recent test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reason: ruff parses every `# noqa: <code>` marker as a flake8 directive
and warns on every non-flake8 rule key. Every catalog rule in
`docs/contributing/actions/testing/suppression.md` is project-specific
(kebab-case), so `# noqa:` markers fired the warning on every parse.
`# nocheck:` is the project synonym ruff ignores.
Scope:
- 41 `roles/*/meta/services.yml` + 1 task + 1 .cfg.j2 template: pure
noqa→nocheck swap for catalog rules. Mixed flake8 markers
(`# noqa: E402, X`) preserved verbatim.
- 5 production .py + 2 production scripts: same swap.
- All catalog-rule lints' docstrings updated; redundant "or # nocheck:"
parentheticals collapsed to a single nocheck form.
- `utils/annotations/suppress.py`: docstring rewritten to spell out
the new convention. Parser still accepts both keywords as synonyms.
- `docs/contributing/actions/testing/suppression.md`: convention
rewritten to mandate `nocheck:`; missing `domain-spot` and
`literal-protocol-lookup` rows added; "enforced by" pointer added.
New guard:
- `tests/lint/repository/test_noqa_only_ruff_codes.py`: meta-lint
failing any `# noqa: <code>` whose codes don't match `[A-Z]+\d+`
(real ruff/flake8 shape). Allowlists the impl, its unit test, the
catalog page, the direct-yaml lint, and itself.
- `tests/lint/repository/test_no_direct_yaml_calls.py`: hardened to
reject the `noqa:` form via inlined regex (no longer routed through
`utils.annotations.suppress`).
Tangential cleanups carried in this same commit:
- `utils/cache/yaml.py`, `utils/handler/yaml.py`: 6 `direct-yaml`
markers swapped.
- `utils/cache/base.py`: `from . import PROJECT_ROOT, ROLES_DIR` moved
up to the import block (E402).
- `cli/create/role/__init__.py`: reordered so PROJECT_ROOT is defined
before `__main__` is imported (also fixed a latent circular import).
- PROJECT_ROOT migration tail: 2 new `__init__.py` modules + 10 test
files swapped from `def repo_root()` to `from . import PROJECT_ROOT`.
- ~20 ruff auto-fix removals of unused `from pathlib import Path`.
Verification:
- `ruff check .` clean.
- 23 suppression-related tests pass.
- The new lint detects all three regression shapes (project rule alone,
mixed with real ruff code, lowercase-only).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace literal `enabled: true / shared: true` flags across every
roles/*/meta/services.yml with a Jinja `"{{ '<role>' in group_names }}"`
form so the per-role services map becomes the single source of truth
for shared-service co-deployment, on a per-host basis. Inventory build
already runs through the templar before reaching the static `is True`
check, so runtime behaviour is unchanged; the new form only opens up
static-analysis and CLI tooling to the group-aware semantics.
Mechanism + lint:
- New integration test tests/integration/roles/meta/test_services_
dynamic_flags.py enforces the dynamic shape per top-level service
entry. Registered service keys must reference the registry-resolved
provider role; covered keys (e.g. oauth2 -> web-app-keycloak via
the new `covers:` block) reference the covering role.
- Two suppression placements documented in docs/contributing/actions/
testing/suppression.md: per-flag inline `# nocheck: dynamic-flag`
for databases (`enabled: true` stays literal, `shared` resolves
dynamically) and block-level above the key for entries whose both
flags legitimately stay literal (`css`, `javascript`, role-local
`redis`, `recaptcha`, `hcaptcha`, role-local containers with
`shared: false`).
Registry + helpers:
- utils/service_registry.py: new `is_explicit_truth(value)` helper
accepts `True` literal OR a Jinja string carrying `in group_names`.
Adopted by `resolve_service_dependency_roles_from_config`,
`ServicesResolver._is_enabled_shared`, `test_web_app_email_
integration`, `test_prometheus`, `test_universal_logout`, and the
run_after-explicit guard so static analysis keeps resolving deps.
- New `covers:` field on a primary service entity declares
transitive equivalents (web-app-keycloak.covers: [oauth2]).
Exposed via `build_role_to_covered_keys` and
`build_covered_key_to_role`, replacing the previously hard-coded
`_TRANSITIVE_SERVICE_EQUIVALENTS` / `_SPECIAL_REFERENCE_ROLE` maps
in the run_after and dynamic-flag tests.
Bulk migration:
- All 80 roles/*/meta/services.yml files migrated to the dynamic
form. DB consumer keys (mariadb, postgres, ldap) keep
`enabled: true # nocheck: dynamic-flag` literal with a dynamic
`shared`. Block-nocheck applied to css/javascript/recaptcha/
hcaptcha and role-local containers (e.g. mailu's admin/imap/smtp/
antispam/antivirus/oletools/webdav/fetchmail/webmail/resolver/
front, nextcloud's talk).
- web-app-prometheus's primary entity now declares
`provides: prometheus` + dynamic flags so the registry recognises
it as a service provider. Consequence: tasks/01_core.yml split out
with `- include_tasks: utils/once/flag.yml` as the first task and
tasks/main.yml reduced to the standard run_once-guarded loader.
- Seven minimal no-login apps (web-app-hugo, -bridgy-fed, -chess,
-littlejs, -navigator, -mini-qr, -mig) gained an explicit
`logout: { enabled: false, shared: false }` opt-out (with
`# nocheck: dynamic-flag` block-marker) to satisfy the universal-
logout schema check.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
