- Declare `__all__` in every package `__init__.py` whose `PROJECT_ROOT`
is imported by its modules (163 packages), so the constant counts as
exported; list every public name where the package defines more.
- Empty the 15 package `__init__.py` files whose `PROJECT_ROOT` no module
imports.
- Declare the public API of `utils.github.variant.axes` in `__all__`,
re-exports included, instead of silencing them with `noqa: F401`; the
list matches the one `feature/store-i18n` already carries.
- Use `import unittest.mock as mock` in `test_applications.py`, as the
rest of the suite does.
- Spell `_NON_PRINTABLE` in `utils/yaml_bootstrap.py` as the equivalent
BMP-only positive class; it matches exactly the same 2111 code points
as the old negated class across U+0000..U+10FFFF.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The guard recognised meta/services.yml as the only place a git URL may be
declared. 84a4b04f0 made meta/addons/<id>.yml the one place a Discourse plugin
pins itself, so the addon files now carry the clone URL literally, and
_iter_role_files picks up every .yml under the role: each declaration reported
itself as an undeclared reference.
Two halves. The declaration set is now the union over meta/services.yml and
every meta/addons/*.yml, so an addon pin legitimises that URL for the whole
role, not just for the file holding it. Path.glob on a missing meta/addons
yields nothing, so roles without addons need no special case. The self-skip
that kept services.yml out of the scan is now a membership test against exactly
those declaration paths.
Scope stays per role: declared is rebuilt inside the role loop, so a
declaration in one role legitimises nothing in another. The failure message and
the docstring name both sites, otherwise they send the next reader to the wrong
file.
This is the test-side counterpart of e4141cb826, which taught collect_entries
to see the same addon refs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An MCP client spec signs in through the client's own OIDC button, which
redirects to Keycloak. Every provider put its MCP pairing in the lean variant,
and that variant pinned sso off - so the round that exists to prove the pairing
was the one round without an identity provider. Run 33574056480 shows what that
costs: openwebui's MCP specs reached _shared.js:131 and died on "openwebui OIDC
sign-in button must be visible on the auth page".
The rule now demands two things of the variant that carries the consumers: the
provider pins sso on, and each admitted consumer, in the variant that round
actually pulls in, does not pin it off. The second half matters because a round
deploys every dependency at its own index and falls back to variant 0 only when
the dependency has fewer, so the client shipping alongside is that one and not
the client's variant 0. A role that declares no sso, or pins it to a literal
false because upstream speaks no OIDC, is exempt from a flag it cannot have.
Twenty roles moved to satisfy it. Ten already had a second sso: false pin and
only needed the flip. Six could host the pairing in variant 0, which already
has sso on, and dropped back to two variants. Six kept a third lean variant to
carry the false polarity variants-coverage wants. moodle and mattermost took
swaps instead: moodle's variant 1 is the mandated LDAP-only branch, and
mattermost was already at the three-variant cap with sso on in its variant 2.
nextcloud also turns context_agent on in that round, the ExApp that serves the
endpoint - with it off the round proved nothing it was built to prove.
The rule moves to tests/lint/ansible/services/ because it reads meta YAML and
executes nothing, which is where integration.md says such a check belongs, and
its suppression key joins the catalog it was never listed in.
nextcloud_exapp_reregister becomes nextcloud_exapp_renew: mcp-secret-masking
matches `register:` unanchored, so the substring in the old name read as a task
that prints a secret. register.yml already masks the two tasks that carry one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mcp-consumer-variant guard demanded three things of a provider's MCP
round: every admitted consumer enabled, each of them pinned `services: null`
so it dragged no dependencies along, and no other service left enabled. Two
of those were the rule legislating a round's contents rather than checking
that the pairing gets proven at all.
Keep the first and drop the rest. What else the round carries is the role's
own call, so the tor/litellm exemption, the datastore exemption and the
`allowed` set they fed have no reader left and go with them.
Strip the 63 `services: null` pins the rule forced across 20 variants, all
written bare as `services:`. The topic-emptying capability behind them
stays: apply_topic still normalises None to {} for the inventory payload,
and the unit tests still cover it. Only the uses and the mandate are gone.
That leaves overridden_providers scanning a repository with nothing left to
find, so its three assertions passed over an empty dict and would have
stayed green however the function broke. Rebuild them on a synthetic role
tree, which also covers the case the repository scan never held: a variant
naming a plain flag is not an override.
Removing the pins changes what deploys. Measured across all 20 variants: 18
identical, prometheus#1 grows 23 to 31 roles, moodle#2 grows 31 to 32 and
crosses the memory budget until the following commit shrinks it back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A round deploys a provider once, so the role that pulls it in is the only place that can say how lean it has to be. A variant entry may now carry any application config topic under a service it depends on: a mapping deep-merges, so naming one key changes that key alone, and null empties the topic. Empty was the case that needed a marker of its own, because merging an empty mapping is a no-op and would leave exactly the services the round was trying to shed.
The topic vocabulary and the merge semantics live in utils.roles.applications.topics, which is also where resources.py now gets them rather than spelling the same null-or-merge out a second time. Collecting who demands what stays in the inventory tree, refuses two roles demanding different values for one provider in the same round, and skips a role's own entity entry so the existing services.<own>.users declarations are not read as overrides.
The CI set-cover learns the same fact. Its greedy cover retires a role's variant-0 row when an earlier row of another role embeds it, on the stated premise that embedding pulls the provider's variant 0. A variant that dictates that provider's config breaks the premise from the other side: whatever ran, it was not the provider as the provider declares it. Such a pair now covers nothing, so the retired row keeps its job, its artefact and its verdict.
Measured on the current tree: 20 override pairs, 41 covered rows before and after, so nothing is revoked today. The rule bites the moment an override lands in a row that covers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
enabled and shared were hand-maintained in 24 roles and wrong in both
directions. Both now resolve from the mcp_consumer edges already declared
in meta/services.yml: a server takes the consumers it admits, a client
takes the servers that admit it, and the pair renders as "'peer' in
group_names or ...", or False when no peer is admitted at all. setdefault
keeps meta/mcp.yml and meta/variants.yml as the override channel, so
pinning either key still wins.
The resolved block replaces the mapping instead of being written into it.
_build_role_base_config parks the YAML cache's own object under mcp, so an
in-place setdefault injected the derived keys into every later raw read and
made test_mcp_schema pass alone but fail in a directory run.
The derivation keys on peer presence, never on a peer's own value: a
re-entrant read under _RENDER_GUARD returns the unrendered dict.
Four lints now ask the resolver instead of reading meta/mcp.yml, which they
could not do while the key was hand-written. That is what exposed the eleven
missing hostile-path specs the previous commit closed.
Carries one change that belongs to the WordPress probe but shares a file
with this one: the tool allowlist moves to the hyphenated names the adapter
actually serves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
7a70d27b7 made a variant that switches SSO on also switch the onion on, because
that pairing is where the platform breaks in ways clearnet never shows. The same
argument reaches further than SSO: a Secure cookie is dropped, an https identity
provider refuses to embed, and crypto.subtle is undefined for any feature on a
non-secure origin, not only for a login. A variant that never runs the onion is
not testing the deployment users get.
So the rule now covers every variant of a role that declares a tor block.
Variant 1 stays exempt - it is the minimal, partner-free round every role
declares and the one place the stack is exercised without the onion at all - and
a role whose meta/services.yml pins tor.enabled to a literal false is skipped
outright, because that declaration already says the role has no onion and needs
no comment repeating it.
That found 137 variants. 101 were variant 1. Of the rest, 28 are the LDAP-only
auth-isolation rounds, which gain real coverage from running LDAP over the onion
and are flipped on here. Four cannot: svc-net-tor is the provider itself, and
nextcloud's variant 8 pairs the clearnet-only PeerTube. Both carry a marker
naming the reason; opentalk and coturn need none, since their services.yml pins
the literal false the rule now reads.
Flipping them exposed a contract that contradicted itself. test_auth_isolation
and test_non_baseline_explicit_disables demanded `shared: false` beside
`enabled: false`, while variants-static-override flagged that same line as a
static override - which is why all 133 of them carried a nocheck marker. The
demand only makes sense where `shared` is dynamic: a literal base value cannot
leak through deep-merge, so restating it protects nothing. Both contracts now
ask for it only in that case, and the 133 lines are gone.
The result is a stricter rule with fewer lines: 189 insertions against 193
deletions.
Verified: tests/lint 336 passed / 307 subtests, tests/integration/roles 107
passed / 610 subtests. The rule is checked in both directions - it reported 137
offenders before the resolution and none after.
Not measured: 28 additional onion rounds cost matrix wall-clock, against swarm
jobs already capped at 355 minutes and one that has hit that ceiling before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`cli build include` generated one tasks/groups/<group>-roles.yml per invokable category during setup, and the stage plays included those files. They were written by the in-container UID, which is why a dedicated clean-container-owned helper had to delete them from inside the container.
The stage plays now loop over lookup('group_roles', <group>) and include tasks/utils/setup/{group,role}.yml directly. ordered_roles() in utils/roles/order.py holds the run_after topological sort the generator used to perform at build time.
utils/roles/stage.py:_group_names() no longer globs the generated files; it reads get_invokable_paths(), the same categories.yml source those files were derived from.
Measured: handlers still flush after every role (ansible-core 2.20.2, three loop items, three separate handler runs), and resolving all 18 groups costs 0.60s cold against 1.4s for the generator.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Alphabetical make target order, and the integration test walked the project
tree itself instead of the cached helpers, which share one walk and one read
per file across the whole session.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nine roles name their MCP principal by a literal key, and a typo there does
not fail loudly: the lookup yields an empty password and the application
answers with a rejected login, on the one variant that enables the surface.
The rule stays scoped to the mcp- principals because the registry is open by
design, so a repository-wide version would report the inventory's own
Playwright persona as missing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/schema.yml held credential definitions and their runtime values, but the
name said 'schema' and read like structural metadata next to services.yml and
volumes.yml. Nothing in the file is a schema in the sense the other meta files
use the word, and a contributor had to open it to learn it holds secrets.
62 role files renamed. ROLE_FILE_META_SCHEMA becomes ROLE_FILE_META_SECRETS and
keeps its role as the SPOT the path is built from.
Two readers bypassed that constant and spelled the name themselves:
utils/cache/applications.py built meta_dir / 'schema.yml' twice, and
plugins/lookup/application_gid.py carries the filename in the marker list it
detects application roles by. Left behind, the rename would not have raised —
credentials would have resolved empty and the role would have stopped counting
as an application.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two violations that tests/lint cannot see, because ruff runs under the
autoformat target rather than the lint suite:
TC003 test_db_tor_transport_gate.py imported pathlib.Path at runtime
for an annotation only. The module has `from __future__ import
annotations` and never constructs a Path -- the values come out
of ROLES_DIR.glob() -- so the import moves into TYPE_CHECKING.
RUF012 test_render.py's ENTRY is a mutable class attribute; annotate it
ClassVar[dict].
Plus three reflows from ruff format. No runtime behaviour changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The three svc-db-* credential tests each hand-rolled the same transport
gate in their test.env.j2:
(lookup('config', application_id, 'services.tor.enabled') | bool
and lookup('config', application_id, 'services.<entity>.exposed') | bool)
Three copies of one predicate, each naming its own entity key, and each
free to drift from the onion mapping that actually decides reachability.
Extract it into plugins/lookup/tor_reachable.py, split the same way as
its sibling tor_ports.py: a plain is_reachable() for callers holding the
merged applications view, plus a thin LookupModule resolving it through
lookup('applications'). Because collect_exposed_ports() scans
services.values(), the entity name disappears from the templates
entirely -- they collapse to
TOR_ENABLED={{ lookup('tor_reachable', application_id) | lower }}
Rework the integration guard accordingly: it now asserts that every
svc-db-* template delegates to the shared lookup, and drives its
base/variant agreement table through the real is_reachable() instead of
re-implementing the predicate. Add unit coverage for the new lookup.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Each of these guards a failure the work already hit. An adapter path
declared in two files drifts silently and surfaces as a 404 on the first
tool call. A provider borrowing the administrator token hands every
client that account's rights, and the read-back never matched the store,
so every deploy minted a fresh secret. A task that both carries a secret
and produces printed output puts it in the deploy log verbatim.
Provider probes, consumer transports, the schema and variant coverage
are checked the same way.
The block sat under services but never behaved like a service. A service has
exactly one providing role, declared with provides, so the registry can map the
key to that role and consumers bind their flag to its presence. No role declares
provides: mcp. There are seventeen providers and the pairing is done by
allowed_consumers inside the block itself.
Every rule that enforces the service model therefore needed an exemption for it:
a shared-provider suppression in fourteen roles, a playwright-service-flag
suppression, and two freshly added exemptions in the dynamic-flag and
variants-static-override checks. Four independent rules, four escape hatches,
one cause. They are all gone now, together with a bond key that only the
resource budgeter reads and only for entries under services, and a wordpress
addon bridge that pointed at a service block that no longer exists.
The block moves to meta/mcp.yml. The applications loader maps meta/<name>.yml to
applications.<app>.<name> generically, so the move needs no wiring; the file
takes the same shape as meta/rbac.yml. Every config path follows, the variant
overrides are lifted out of their services map, and the vocabulary module leaves
the services package because it no longer describes one. roles_with_service
gains an explicit topic parameter rather than an implicit fallback, so nothing
silently resolves against the wrong container.
Files that are otherwise unrelated to this move are included where the committed
tree would otherwise keep pointing at a path that no longer resolves: thirteen
vars files, eleven playwright env templates and one task file.
Defines what a services.mcp block may declare and enforces it: classification,
transport, auth, implementation and the provenance fields that make an upstream
claim checkable. The lint refuses an unknown key anywhere in the block and
refuses a provider whose allowed consumers cannot speak its transport or its
auth scheme.
Discovery resolves each provider's credential and endpoint per consumer and
per deployment, rejecting a server with a named reason. Transport, auth and
endpoint faults are fatal; a missing credential is not, because it also occurs
when the provider is simply not provisioned yet in the same play.
The filters render the Authorization header from the declared scheme, build
Open WebUI's tool-server connections with their group grants, and derive the
members of those groups from the RBAC assignments. Role scoping gains
application-scoped roles, so an mcp role only counts when it is granted on a
specific application.
roles_with_service gains a scope and a direction filter, and drops the canonical
domain requirement for direction-filtered entries: an MCP URL is built from the
service key and the internal port, so a provider without a public vhost stays
valid.
Read platform_arch from the role config and append a node.platform.arch
constraint where it is set, so a role whose image exists for one
architecture only cannot be scheduled onto a node that could never run
it. LM Studio ships x86_64 alone and is the first consumer.
The constraint lives with the other placement rules rather than in the
consuming role, because architecture binding is a property any role with
a platform-specific image can carry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Six roles have an upstream MCP path this deployment cannot use, so they
declare a blocker instead of a direction. Without it they would read as
"upstream has no MCP", which is false for all of them and worst for
OpenProject: the server ships, it is Enterprise-only, and upstream's own
request specification asserts a 404 on the Community edition this role
deploys. Jira, Confluence, WordPress, Odoo and Discourse are hosted-only
or unreviewed third-party paths.
The dynamic-flag suppressions are load-bearing: a blocker block is
mutually exclusive with a served surface, so a group_names expression
there could never evaluate true.
Add the test that keeps a served surface honest: every role declaring one
must gate it on the role's MCP-enabled flag, so switching MCP off removes
the endpoint instead of leaving it reachable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Open WebUI talked to Ollama directly. It now speaks the OpenAI-compatible surface of svc-ai-litellm with a per-consumer virtual key, so a second backend or an external provider becomes a gateway config change instead of a role change. A lint keeps future roles off the model servers.
The gateway advertised a hardcoded model list (ollama/llama3, ollama/nomic-embed-text) while the matrix variant preloads smollm2:135m, so it offered only models the backend did not have and hid the one it did. The list is derived from the ollama preload SPOT now.
The chat spec followed the rename: it gates on litellm instead of the removed ollama service flag, and picks the first non-embedding model from /api/models rather than pinning a variant-specific name, so it exercises the chain rather than a fixture.
Routing openwebui through the gateway pulls both model backends into the closure of every role that consumes it, which pushed nextcloud variant 2 over the memory budget. The budget stays at 32/64 GB; instead the matrix variants of svc-ai-ollama and svc-ai-lmstudio now declare the size their test model needs (1g/2g) while the base keeps the 6g/8g that llama3 and mistral require. The budget guard only saw half of that: it replaced the evaluated role with its variant but counted every dependency from meta/services.yml, a combination no matrix round deploys. It now evaluates every variant-carrying role at the same round index, falling back to variant 0, the way the deploy planner does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ruff toolchain was unpinned, so host venvs, the lint container and CI
installed different releases over time. That drift is why lint-python only
failed on CI: locally ruff 0.15.x preview-gated ISC004/FURB192 and (falsely)
fired S310 on urllib.request.Request, while CI's 0.16.0 has ISC004/FURB192
stable and drops the S310-on-Request false positive.
Pin ruff==0.16.0 in pyproject.toml's dev extras (Dependabot keeps it current)
and have utils.install.lint.ruff read that pin and reinstall whenever the
found version drifts, so every environment resolves the same ruff. Then fix
what 0.16.0 flags: parenthesise the implicit string concatenations (ISC004),
use min() over sorted()[0] (FURB192), drop the now-unused S310 noqa on the
Request() builders (RUF100), and delete the narration comments that the
comment lint surfaces in the touched files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.
Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- matrix 02_render_inventory: the mdad-state volume does not exist yet on a
fresh install, so `container volume inspect` errored the reconfigure task
(`no such volume`) on the row-1 baseline. Guard the marker drop behind the
inspect succeeding; the runner bounce still runs if the container is up.
- test_coverage: read services.yml via utils.cache.files.read_text so the
no-uncached-file-reads lint passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- _shared.js: wrap the boot-poll delay in resolveTimeout (raw-timeout lint)
- 02_render_inventory: docker -> container wrapper (engine-abstraction lint)
- seaweedfs: api becomes the primary canonical so the deploy readiness
probe hits a served vhost when the frontend (filer/master) is disabled
- seaweedfs frontend: exempt from variant pinning — the onion gate is a
hard invariant, no variant may force the frontend on a tor node
- test_coverage: add the `# nocheck: variants-coverage` exempt (same
mechanism as bond-baseline) for such invariant flags
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The run-once guard lint required the when to be exactly the guard string, rejecting roles that legitimately combine it with further conditions (svc-bkp-container-2-local gates its whole core on IS_STACK_HOST). A when-list containing the guard now passes. The variants coverage test learns the same block-above '# nocheck: dynamic-flag' key suppression the services dynamic-flags test already honors, so infrastructure consumer flags can opt out of two-polarity variant coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extract the csp, domains, and networks blocks out of roles/*/meta/server.yml
into dedicated meta files, each mapping to its own top-level applications key
via the recursive meta-walk:
csp: -> meta/csp.yml -> applications.<app>.csp
domains: -> meta/domains.yml -> applications.<app>.domains
networks: -> meta/networks.yml -> applications.<app>.networks
Residual nginx/proxy keys (status_codes, client_max_body_size, locations,
ip6_enabled, config_upstream_url) stay in server.yml; an emptied server.yml is
deleted. 98 server.yml -> 249 new files, 74 deleted.
- Consumers repointed applications.<app>.server.{csp,domains,networks} ->
applications.<app>.{csp,domains,networks} across templates, vars, tasks,
lookups and filters.
- Infra readers migrated: canonical_domains_map, application_domain_index,
build_ldap_role_entries, service-registry overlay discovery,
roles_with_service, and the file-based readers utils/domains/list.py +
utils/meta/scan.py (read the now-deleted server.yml directly).
- Detection + mapping: _META_TOPICS gains csp/domains/networks;
ROLE_FILE_META_{CSP,DOMAINS,NETWORKS} constants + ROLE_FILES entries.
- New self-expiring lint guard (test_server_meta_no_split_fields) fails if a
server.yml still carries a split field; auto-fails from 2026-12-28 as a
delete reminder.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The resource right-sizing and variant rebalancing bring every variant well under the original 37GB/73GB, so lock in the tighter 32GB/64GB host budget.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The bond<1 partner bins (and the coupled v0 baseline) sit just under these
limits; the previous 32/72 cap was the single-all-on-host figure that the
6-variant distribution replaces. Honest accounting (every enabled service still
counts) — no bond-zeroing or per-variant exemption.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
test_order now only pulls bond>=1 (tightly-coupled, same-host) services into a
role's closure; bond<1 partners deploy on their own host and are excluded, so a
later variant may enable them without variant 0 having to list them.
New test_bond_baseline enforces the placement: a bond>=1 dynamic flag must be
pinned true in variant 0 (the host baseline); a bond<1 flag must be pinned true
in at least one variant. Escape: '# nocheck: bond-baseline'.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
test_limit.py hard-capped meta/variants.yml at MAX_VARIANTS (3) with no opt-out.
Allow a role whose extra matrix rounds are justified (e.g. distributing many
bridged partners so each variant stays within the memory budget) to opt out via
a '# nocheck: variant-limit' comment in its meta/variants.yml.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an integration guard that, for every role and every meta/variants.yml
entry, sums the deduplicated resource footprint (own containers plus shared
dependencies, each service counted once) via the shared
utils.roles.applications.services.resources implementation and fails when a
variant exceeds mem_reservation 32 GB or mem_limit 48 GB, to avoid an OOM kill
at deploy time. The failure message recommends disabling services in the
offending variant or moving them to other variants.
Currently surfaces 6 over-budget variants (nextcloud, shopware, openproject,
flowise, gitlab, openwebui v0) that still need their footprint trimmed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reconcile main's object-store feature and pre-swarm deploy tasks with the branch's compose/networks/volumes SPOT and swarm lints.
objstore (seaweedfs/minio): standalone compose + embedded service templates migrated to the container_image/container_volumes/container_networks/compose_volumes/compose_networks/depends_on lookups; meta/volumes.yml on the canonical dict schema with s3.json and nginx.conf as type:config; embedded services made swarm-compatible via compose_only/compose_restart.
deploy tasks (erpnext, opentalk, seaweedfs, shopware, snipe-it, wordpress, yourls): container exec routed through the container_address resolver and gated on IS_STACK_HOST, shell module plus stack_host_template, no chdir or compose-verb forms.
Also removed orphaned vars, restored MAILU_INSTANCE_DIR, freed the seaweedfs subnet (192.168.206.0/24), stripped literal quotes from the nextcloud/bookwyrm env templates, and routed main's objstore compose_volumes tests through dir_var_lib.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a meta lint asserting every web-app role with postgres or mariadb declares a seaweedfs object-store service, exemptable via the seaweedfs-required nocheck marker. Mark the 33 database-backed roles that do not use object storage, and register the rule in the suppression catalog.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add plugins/lookup/container_image.py: returns the full `image: "<ref>"`
line so templates drop the redundant image: prefix and surrounding
quotes. Supports `custom=True` to auto-construct `<entity_name>_custom`
for locally-built images, plus explicit `image=`/`version=` overrides
for exotic cases (taiga back-auth/front-oidc variants, decidim infinito
suffix, etc.).
* 25 unit tests for the lookup cover compose vs swarm prefix, custom
flag, overrides, tag_only=True, missing version warnings.
* New strict lint (tests/lint/.../test_no_raw_image_in_compose_templates)
fails on any raw `image:` line in roles/*/templates/*.j2 unless
nocheck-suppressed. Legacy `image: "{{ lookup(...) }}"` form no
longer accepted.
* Migrate 159 image refs across ~80 roles to the standalone
`{{ lookup('container_image', application_id, '<service_key>') }}`
form. 26 roles use `custom=True` (mariadb, postgres, openldap,
jenkins, jira, gitlab, nextcloud, mediawiki, shopware, wordpress,
odoo, joomla, openproject, bookwyrm, espocrm, hugo, littlejs,
fediwall, roulette-wheel, openresty, confluence and more).
* Fix svc-db-redis application_id leak: service.yml.j2 is included
into consuming roles where application_id evaluates to the outer
role, so the image lookup needs the hardcoded 'svc-db-redis'.
* Integration tests (test_templates_require_image,
test_build_requires_image) recognise the standalone lookup as an
implicit `image:` at the same indent.
* Mediawiki swarm tasks: add explicit `:{{ MEDIAWIKI_VERSION }}` to
the registry-prefixed image string because the custom var is now
bare.
Only 3 legitimate exotic nocheck-suppressed image lines remain:
matrix-mdad runner (nested services.matrix.runner.image path),
matrix mautrix bridges (per-iteration dynamic) and svc-registry-docker
(chicken-and-egg self-reference).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a strict lint that flags every `type: bind` entry in
roles/<role>/meta/volumes.yml whose source or any mount target points
to a single FILE (path ends in a recognised file extension or a
Jinja var with a file-suffix name like *_CONF/*_FILE). Single-file
binds are swarm-fragile: the source path is only present on the node
where the file was rendered, so swarm rejects the task as soon as it
lands on a different node.
Fix is `type: config`: the manager loads the file at `stack deploy`
time and swarm distributes it through raft to every node.
compose-mode honours the same canonical entry (the compose_volumes
filter renders it as a top-level `configs:` block; container_volumes
emits the per-service reference), so the role stays mode-agnostic.
Per-line opt-out: `# nocheck: meta-volume-file-bind-should-be-config`
with a short `# Reason:` for legitimate exceptions (sockets, kernel
interfaces, runtime-rotated files, multi-MB blobs, host writebacks).
This commit migrates 23 file-bind entries across 16 roles to
type:config: web-app-flowise, friendica, kix (2), listmonk, mailu,
matrix (2), mediawiki, mobilizon, nextcloud, pgadmin, pixelfed,
prometheus (4), shopware (2), suitecrm, taiga (2), zammad. All gain a
default mode:"0440"; mount-level read_only:true is dropped (configs
are read-only by default and the schema forbids the field for
type:config).
Updates the prometheus role-wiring test to accept both type:bind and
type:config for the alert_rules mount probe.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Migrate every role's meta/volumes.yml from ad-hoc dict-of-named-volumes
to a canonical dict-of-dicts schema where the YAML key is the semantic
short name and the entry body carries type / name / source / mounts /
nfs / mode / read_only. The optional 'name:' field is the docker volume
name (defaults to the key); the previous 'docker_name:' field is gone.
What's in this commit (210 paths):
Infrastructure (23)
- New filter plugins/filter/{compose,container}_volumes.py — render the
top-level volumes:/configs:/secrets: section and per-service mount
blocks from the canonical meta. container_volumes uses the SEMANTIC
name as per-service mount source so it resolves against compose_volumes'
top-level dict key (docker_name as source produced 'undefined volume'
on swarm).
- New lookup plugins/lookup/{compose,container,volume}_volumes.py.
volume(application_id, name) returns a normalised single entry;
volume(application_id) returns the whole canonical dict.
- utils/cache/applications.py keeps a sibling registry
_CANONICAL_VOLUMES_BY_ROLE outside the merged applications payload so
raw Jinja source: strings never enter the templar walk (root-cause of
the 200x cache-miss + recursive get_merged_applications storm).
- utils/roles/applications/mounts.py — schema validator + normaliser;
list-shape is rejected.
- plugins/lookup/nginx.py: _volume_path_by_name reads canonical directly.
- scripts/tests/deploy/swarm/_context.sh: NFS_VOLUMES extraction handles
both legacy dict and canonical dict shapes.
- roles/svc-prx-openresty/handlers/restart_and_validate.yml: gate
compose-only validate+reload behind a single when-block; swarm uses
service-update instead (bind-mounts of a FILE pin source inode at
task-create time, so atomic write-tmp+rename leaves the running task
on a deleted inode).
- roles/sys-svc-compose/tasks/utils/nfs_prep.yml: switch to
lookup('volume', application_id) | dict2items so NFS subdir
pre-creation works under the new cache split.
Tests (12)
- New strict lints flagging inline mount sections in compose templates,
single-file bind mounts, and legacy lookup('config', ..., 'volumes.X.Y')
call sites — all volume-mount surfaces MUST flow through container_volumes
/ compose_volumes / volume lookups.
- Integration: test_volumes_compose_chain + test_volume_constants_are_lookups
accept both the legacy and the new lookup('volume', app, 'X').name forms
during the migration window.
- Unit tests for the new lookup, filter, mounts helper, plus updated
nginx lookup test.
- Prometheus role-wiring test reads canonical meta for the
alert_rules.yml bind mount.
Role migrations (175)
- 62 meta/volumes.yml converted to dict-of-dicts canonical shape
- 63 templates/compose*.yml.j2 replace inline volumes:/configs:/secrets:
with lookup('container_volumes', application_id, '<service>') |
indent(N) per service + a single trailing lookup('compose_volumes',
application_id) for the top-level block
- 50 vars/main.yml drop dead volume constants and refactor 23 remaining
legacy lookup('config', ..., 'volumes.X.Y') to lookup('volume',
application_id, 'X').Y across 9 roles
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add a meta lint asserting every web-app role with postgres or mariadb declares a seaweedfs object-store service, exemptable via the seaweedfs-required nocheck marker. Mark the 33 database-backed roles that do not use object storage, and register the rule in the suppression catalog.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
