- Declare `__all__` in every package `__init__.py` whose `PROJECT_ROOT`
is imported by its modules (163 packages), so the constant counts as
exported; list every public name where the package defines more.
- Empty the 15 package `__init__.py` files whose `PROJECT_ROOT` no module
imports.
- Declare the public API of `utils.github.variant.axes` in `__all__`,
re-exports included, instead of silencing them with `noqa: F401`; the
list matches the one `feature/store-i18n` already carries.
- Use `import unittest.mock as mock` in `test_applications.py`, as the
rest of the suite does.
- Spell `_NON_PRINTABLE` in `utils/yaml_bootstrap.py` as the equivalent
BMP-only positive class; it matches exactly the same 2111 code points
as the old negated class across U+0000..U+10FFFF.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The guard against hardcoding a topology: a role carrying networks.meshes or
networks.interfaces must ship it empty. The shipped layouts keep their
subnets inside NETWORK_MESH_POOL, their ports unique, and every membership
names a declared mesh.
The guarantees were asserted against the declared swarm and data pair,
so they proved two meshes work rather than that the planner works. A
deployment declares its own, and the review asks for any number of them with
the nodes arranged however the client wants.
The same invariants now run over generated topologies: one to eight meshes,
three arrangements each, hosts dealt out so some belong to several meshes and
some to none. 2247 subtests, four seconds, and the planner needed no change to
pass them.
A variant that pins `services.tor.enabled: true` while enabling a role whose
own `meta/services.yml` pins `tor.enabled: false` declares a combination the
deployment cannot hold. The role rides no node onion, so on that round its
surface lives at a clearnet host no onion browser reaches, and the dashboard's
`frame-src 'self' http://*.<onion>` forbids embedding it outright. Run
36651124458 job 109692287333 shows the result: five specs fail at once,
dashboard tile, matomo tracker, css injection, logout and the coturn metric
label, all on hosts that are the only non-onion ones in the whole job.
web-app-nextcloud variant 11 was such a round: bigbluebutton on, tor on, while
bigbluebutton declares no onion. Variant 8 already answers the identical
question for PeerTube (65538c40da, "clearnet, because the OIDC plugin rejects a
plaintext onion issuer"), so variant 11 now does the same. With that,
bigbluebutton never meets a tor node and needs no consumer gates of its own.
Two roles keep a gate, for reasons that differ from each other:
web-app-seaweedfs is not clearnet-only. It deploys on tor as the object store
other roles reach over the internal network, and only its browser frontend is
off there, which `frontend.enabled` on main already says. Its logout, dashboard
and prometheus declarations claimed a surface that round does not deploy, so
they take the same gate the file's own frontend and sso blocks use.
web-svc-coturn belongs on a tor node and cannot serve one: STUN/TURN runs on
UDP plus a 20000-port relay range that no onion descriptor can publish, and its
only HTTP is a gateway answering a plaintext OK. Deploying it is right,
advertising it as a prometheus consumer is not, because that scrape goes
through the browser path. It is exempt from the rule itself, since it claims no
visitable surface, and its `dynamic-flag` markers now carry the reasons the
rule's own message asks for; one of the four was a duplicate of the line below
it and is gone.
The lint states the rule. It found these five pairings on its own, matching a
hand count made before it existed, and reports none after the variant fix.
`test_sso_variant_requires_tor` covers the complementary half, every variant of
a role that has an onion must pin it on, and exempts the clearnet-only role
itself; this rule covers the consumers that bind one. The suppression marker
belongs on the bound role when the reason is that role's own shape, because a
consumer-side marker would also silence every other clearnet role in the file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Of the 76 open code scanning alerts, 39 point at code that can change.
svc-ai-mcp-adapter:
- A path argument that is empty or made only of dots is refused.
Percent-encoding keeps a separator inside its segment, but "." and ".."
need none, so an upstream that resolves dot segments could answer from a
path the contract does not name.
- parse_duration splits on the units. The pattern it used before rescanned
a long run of digits from every start position.
Tests and the README generators:
- Every Jinja environment built with autoescape=False and a "noqa: S701"
uses select_autoescape now, as six test modules and
utils/templating/jinja.py already did. It escapes templates named
*.html, *.htm and *.xml, which none of these call sites loads.
- Dead code goes: a doubled "rel" block in the email lint, an unread
_SKIP_DIRS, and a module that _resolve returned to no caller.
- Two "except ...: pass" blocks become contextlib.suppress and
assertRaises.
- Nine test modules import unittest.mock one way instead of two, and one
lookup test imports its module once.
Closes#807
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Eight meta/variants.yml repeated their last entry verbatim. The six svc-* roles
held the same tor.enabled override twice; fusiondirectory and phpldapadmin held
a third entry identical to their second, nocheck comments included.
The matrix plans a row per entry, assigns it its own axes and spends a deploy
job on it, around a hundred minutes here. The repetition is invisible in the
plan table, which lists rows by role#variant, so the two read as different rows
and only their identical embeds and weight betray them.
test_variants_distinct.py compares the parsed payload rather than the file text,
so key order, indentation and comments do not hide a duplicate, and null
normalises to {} because a bare list item and an explicit {} declare the same
no-override entry. A role whose variants are told apart by something outside the
entry opts the file out with `# nocheck: variants-duplicate` and a reason; no
file needs it today.
Every deletion is the trailing entry, so the surviving indices keep their
meaning and nothing is renumbered. The vanishing tokens are svc-db-elasticsearch#1,
svc-db-memcached#1, svc-db-rabbitmq#1, svc-db-redis#1, svc-db-typesense#1,
svc-prx-openresty#1, web-app-fusiondirectory#2 and web-app-phpldapadmin#2;
`git grep` against the commit finds no consumer naming any of them.
Negative control executed: re-adding the redis duplicate fails the lint with
"svc-db-redis: variants #0, #1 declare the same override", removing it passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cc6da24174 measured Alpine on nginx:1.31.6-alpine alone and recorded curl as
something the family ships. nginx installs it: /usr/bin/curl is there at 272432
bytes, and at 321880 bytes in the Debian nginx:1.31, while redis:8-alpine and
node:26-alpine measured the same way have no curl at all. The table therefore
passed a curl flavor on any alpine image, which is the failure the rule exists
to catch, spelled with the rule's own name on it.
The family now holds what busybox provides, sh, wget and nc. What one image
adds on top of its family goes in IMAGE_PROVIDES, keyed on the image name with
the registry prefix dropped, and nginx is its only entry today.
The Debian half was re-measured while checking this, over /usr/bin, /bin,
/usr/local/bin and /sbin rather than one path: debian:13-slim, node:26-slim and
python:3.13.15-slim carry /bin/bash at 1298416 bytes and none of curl, wget or
nc. bash is Essential in Debian and the other three are not, which is the whole
difference between the stripped tag and the full one.
Nine cases ran against the predicate. The two that matter are curl on
redis:8-alpine and on node:26-alpine, which passed before this commit and flag
after it; curl on nginx still passes, and the six alpine declarations the tree
actually holds stay green, fediwall and nextcloud on nginx's curl, hugo and
littlejs on busybox wget, taiga on either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
c31c62efa checked four of the repository's 86 healthcheck declarations. It read
the probe's CMD form only, scoped itself to slim, and counted bash as a shell
builtin, which is where most of the gap sat: tcp, connect and http_status all
render CMD bash -c over /dev/tcp, so 31 declarations were skipped as needing
nothing. It now examines 18, and what it still skips it skips for a reason.
bash is a program like any other here. Its absence is not hypothetical: alpine
has no /bin/bash at all, and its busybox /bin/sh does not implement /dev/tcp,
so the redirection those three flavors are built on needs the real shell rather
than any shell. No role pairs them today, which is why this is prevention and
not a fix.
The CMD-SHELL form is parsed instead of ignored: || reads as alternatives, &&
and ; as a sequence needing all its parts, and shell builtins are dropped so a
trailing || exit 1 cannot make every requirement look satisfiable. That brings
in nc, http and the msmtp compositions, six declarations the old shape could
not see. http is the one that needed the alternatives: it runs wget or curl and
is satisfied by either.
alpine moves from a docstring exemption into the scope with a measured table
behind it. Read out of the images with docker create and docker cp, which
reports an absent path rather than a shell's opinion of it: debian slim carries
/bin/bash at 1298416 bytes and none of curl, wget or nc; nginx alpine carries
curl at 272432 bytes as a real binary, wget and nc as busybox applets, and no
bash. Adding a family means measuring it the same way.
Vendor images stay out. Their contents are not knowable from the tree and most
carry no Dockerfile of ours, so the rule's remedy has nowhere to land; skipping
them honestly beats a table that rots.
Sixteen cases were run against the predicate, one per defect class and its
satisfied counterpart. Two corrected the work: nc is a busybox applet on alpine,
which the first table omitted and which made nc-on-alpine flag wrongly, and this
file's own prose named a Debian codename, which test_no_debian_codename_tags
caught in the gate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
10c5b35e88 moved simpleicons and roulette-wheel from node:26-bookworm to
node:26-slim. Both declare healthcheck flavor curl, which renders
CMD curl -f <url>, and neither Dockerfile installed anything. Measured with
docker create and docker cp: node:26-bookworm carries /usr/bin/curl at 280800
bytes, node:26-slim carries neither curl nor wget. The probe therefore fails
from the first beat and the container never reports healthy.
sys-svc-compose waits on the database container alone in compose and on every
stack task in swarm, so the consequence splits by mode. Under swarm the task
dies with non-zero exit (137): dockerexec: unhealthy container, is rescheduled,
dies again, and the converge wait fails after 600 attempts. Five jobs across
the two runs on a534d5d46 died that way, and none of them on the role they are
named after: ERPNext#0, Decidim#0, Hugo#0, Nextcloud#9 and Nextcloud#12 all
carry web-app-dashboard in their closure, dashboard enables simpleicons, and
the failure reads 'not converged: simpleicons_simpleicons'.
All five are swarm and tor, which reads like the tor axis until XWiki#1 is put
beside them: swarm, tor, green, and simpleicons is not in its closure. The
discriminator is the service, not the axis.
Of the seven roles 10c5b35e88 rebased, bridgy-fed and postmarks probe over tcp,
and flowise and bookwyrm install curl themselves. These two were the remainder.
roulette-wheel had not been deployed yet when this was written, so it is the
same defect caught before it was observed. The install goes into the runtime
stage of the multi-stage build; the builder stage would have left the runtime
image unchanged and nothing would have gone red.
The lint reads the binary out of the probe the declaration renders rather than
restating a flavor-to-binary table, so a flavor that changes its command moves
the check with it, and it locates the flavor line per service so one marker
cannot silence a sibling. Its scope stops at slim: nginx:1.31.6-alpine was
measured too and ships curl as a real 272 KB binary plus busybox wget, so the
four alpine roles that declare those flavors are served by their base, and
widening the rule to them needs that measurement per image rather than an
assumption from the word alpine.
Not built. The image with the new layer has not been produced, and whether a
second fault waits behind the healthcheck is decided by the next deploy. The
apt path itself is not blind: BookWyrm#1 installs curl the same way on
python:3.11-slim and is green in this run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The private blocks were written out wherever something needed to say "this is us": NETWORK_TRUSTED_PROXY_CIDRS listed all three, Tor's SocksPolicy repeated them, unbound's access-control repeated them again. Four copies of a fact, and nothing tying them together, so narrowing one narrows nothing else.
Worse, the block role networks are carved from was never declared at all. The allocator behind 'infinito contributing network address suggest' derives its umbrella from the subnets already in use, so the tree defined the rule instead of the rule defining the tree. Anything consuming "our own address space" had to guess it, and a guess that is one role out of date silently excludes that role. Measured: 106 of 107 role subnets sit in 192.168.0.0/16 and one does not, so a firewall range derived from the pool by eye would have cut BigBlueButton off.
The blocks are now named once and everything else is derived, NETWORK_TRUSTED_PROXY_CIDRS included, which renders byte-identically to before for its four consumers.
BigBlueButton stays outside because its upstream pins both the block and its gateway, which meta/services.yml repeats as upstream_host 10.7.7.1. The exception list therefore carries the role id rather than the CIDR: the block is already declared in that role's meta/networks.yml, and repeating it here would be the same duplication one layer up. The role_subnets lookup resolves the ids and raises on a role that declares no subnet, because a silently dropped entry would narrow whatever firewall or trust list consumes the result.
A lint keeps the declaration honest in both directions: every role subnet must sit in the declared space, and an exception whose block the pool already admits is rejected so it cannot outlive its reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ansible renders the values of a structure loaded through include_vars and leaves its keys as written, so the mapping key a564385068 wrote as a Jinja expression reached Nextcloud as its own source text. occ config:app:get sociallogin custom_providers on a live variant-1 instance returns "groupMapping": {"/{{ lookup('rbac_group_path', application_id=application_id, role='administrator') }}": "admin"} while every neighbouring value of the same provider is rendered, scope among them.
Keycloak emits /roles/web-app-nextcloud/administrator, so the mapping matched nothing and the account sociallogin provisions on first login stayed out of admin. occ group:list on that instance shows admin holding only the local administrator while keycloak-ac165525-e9e8-4230-9381-13b1acd83fc0 exists and belongs to no group, and every Playwright spec that needs an admin settings section answered 'Access forbidden'.
An independent reproduction outside this repo showed the same split on ansible-core 2.21.4 and 2.20.6: in a payload of the same shape the sibling value rendered and the key did not, so it is not a version artifact. Nextcloud 34.0.3 settles the other half: Manager.php hands an admin the unfiltered settings of a section and ai is a core section of the always-enabled settings app, so an admin cannot receive 403 there whatever is installed.
Writing the whole mapping as one expression makes jinja2_native, which this deployment has on, return a real mapping with the key computed. The lint takes the class: no meta payload may carry Jinja in a mapping key. It found exactly this one case in the whole meta tree, and it ignores a line that itself opens with the expression so the corrected form does not trip it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
What `make quality-high` rewrote on its own, kept apart from the changes it ran
against so neither has to be read through the other.
The matomo healthcheck test moves from a flow sequence to a block one, which is
the formatter's threshold for a long line and not a change to the command: both
revisions parse to the same two-element list, escapes included. The healthcheck
timing lint loses two blank lines between methods.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The marker names the reason a role has no upstream homepage or demo video, and
it sat in meta/main.yml because a comment-only info.yml parses to an empty
document, which the schema lint rejected outright. So the exemption lived one
file away from the two fields it explains, and a reader looking for the video
found neither the field nor the reason it was missing.
The schema lint now accepts an empty document exactly when it carries the
marker, and that is what allows the move. The media lint reads the marker from
info.yml rather than main.yml, and its failure message names that same file, so
the fix it suggests lands where the absent field would have gone.
A hundred and two roles follow. Every occurrence was a standalone comment on
the second line of main.yml, directly under the document start, so the move is
a line delete and an insert at the same offset. Seventy-three roles already had
an info.yml and gain the line; twenty-nine get one whose whole content is the
document start and the marker. All hundred and two reasons carry over verbatim,
including the thirty-five bespoke ones.
Nothing downstream sees a change. Of the four readers of meta/info.yml, three
coerce a null parse through `or {}` behind an isinstance guard, and
utils/cache/applications.py skips a falsy parse before assignment, so
applications.<role>.info stays absent for the twenty-nine exactly as it was
while the file did not exist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
start_interval is the cadence while start_period runs, and only while it runs.
Once the window closes the probe falls back to interval, so a service that
becomes ready just after it pays a whole tick anyway and the cadence never
reached it. Six of the seven flavors were shaped that way, Curl worst: a thirty
second window against a one minute interval.
Measured across 151 converge waits in four job logs, 63 sit exactly on a tick
boundary, 36 in the thirty second step and 27 in the sixty second one.
Attributed by application, web-app-seaweedfs is the largest at 1501s over
sixteen waits with a median of 57.7s: it probes with curl and fell straight into
the gap. web-app-zammad probes with connect, whose window is fifteen minutes,
and its 154s median is real boot time with nothing to recover.
So the window moves to ten minutes and the steady state interval to two, and the
flavor overrides that sat below them go: Curl, Wget and Connect lose interval
1m, Http its twenty second window, Nc its ten. Connect keeps fifteen minutes.
Neither number costs anything while a container works. start_period delays
nothing, it only stops failures counting, and interval no longer touches
convergence at all since that hangs on the window alone; it is steady state
load, which drops from 120 probes an hour per service to 30.
Forty six role files follow. Twenty seven declarations go that repeat the new
default or undercut it, an undercut window being the defect itself. Eight
windows longer than ten minutes keep their value and now say why. The literal
blocks and the three Dockerfiles this repository builds spell every timing out,
because outside the lookup an omission takes docker's value and not the
platform's.
A fourth lint rule keeps the shape: start_period must outlast one interval,
checked against the effective values, so raising interval without the window
cannot quietly bring the cliff back.
Two faults in the lint surfaced while building it and are fixed here. It
compared timings as strings, so 600s and 10m read as different values, and it
matched the healthcheck key exactly, so a line carrying a trailing nocheck was
skipped whole, which is how the two swarm-init partials went unexamined.
One regression, bounded. A container that starts and never turns healthy is
neither rejected nor errored, so stack_ready.sh only calls it fatal after three
killed tasks, and each cycle now spans the longer window. That reaches twenty
minutes rather than six, capped there by swarm_stack_converge_retries at 600
with a two second delay, and it lands only on runs that were failing anyway.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three checks over the two places a timing can be written.
In meta/services.yml the lookup supplies the defaults, so a declaration is
either noise or a decision. A value equal to what the service would have
inherited is deleted rather than justified: it changes nothing and hides the day
the default moves. A value that differs is kept and carries its reason.
That reason is required for interval, timeout and start_interval only. Those
three cost something: steady state load for the life of the deployment, the
threshold at which a slow probe counts as failed, and the only timing that moves
the moment a stack reports healthy. start_period and retries buy patience with a
container that is already broken and cost nothing while it works. Demanding a
defence for them would have produced sixty one sentences nobody had evidence
for.
Outside the lookup, in a literal healthcheck block and in a role's own
Dockerfile, an omission is a finding too. Nothing there inherits from the SPOT,
and docker has no daemon-wide default to fall back on, so a missing key lands on
docker's value rather than the platform's. They disagree on start_period, zero
against thirty seconds, which is exactly what left start_interval without a
window in every hand-written block.
Comparison is against the effective default, resolved through the same compose
the lookup uses. Curl probes every minute, Connect grants a fifteen minute start
window, and a prefix raises timeout, retries and start_period further; measuring
against Probe alone would have called out every service that simply agrees with
its flavor.
Negative control for each check. Before the three commits that precede this one
the rule named 47 restatements, 20 unexplained deviations and 34 definitions
outside the lookup. It names none now.
Two limits worth knowing. The line number for a meta timing is found by
indentation and service name rather than by a parser, so an unusually formatted
file makes the rule ask for a reason it cannot anchor; it errs towards asking,
never towards silence. And the Dockerfile reader follows a backslash
continuation but not a comment inside one, which no Dockerfile in the tree uses.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The SSO gate handed X-Forwarded-Access-Token to every oauth2-fronted upstream, so each of the 28 gated roles received a full-scope bearer for the signed-in user while exactly one, n8n, decodes it. The token also grows with the realm's registered origins, and LAM's Apache rejects a header field above 8190 bytes (Moodle#0, run 33689701895).
Consumers now declare services.sso.oauth2.pass_access_token. Without it oauth2-proxy already answers the auth request without the token, and the gate blanks the upstream header rather than dropping the directive, because nginx inherits proxy_set_header as an array and an unset header would let a client-supplied one through. The new lint keeps declaration and use in step in both directions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mount pipeline was split across two layers: compose_volumes and
container_volumes lived as filter modules that ansible's filter loader
rejected on every start ("has no attribute 'FilterModule'", two warnings
per run), while the same-named lookups wrapped them thinly. The filter
bodies now live inside their lookups (the tor_socks precedent), the
filter files are gone, and the render tests moved next to the lookup
tests.
As lookups they see the templating context, so the application_id term
every call site passed back to its own role is gone too: both lookups
read application_id from context and fail loudly when it is missing.
107 templates and nfs_prep.yml drop the term; the calls collapse to
lookup('compose_volumes') and lookup('container_volumes', service_name).
The container_volumes call is now canonical - four leading spaces,
service_name in the service position, | indent(4) - enforced by a new
lint (test_container_volumes_call_shape). Normalizing the 85 literal
call sites surfaced that erpnext keyed its meta/volumes.yml mounts by
container name instead of compose service key, the only role to do so;
its meta now uses the service keys. wordpress's column-0 call, which
rendered its first line at top level, is indented like every other.
Template logic that only restated meta/volumes.yml is deleted: mailu fed
compose_volumes an extra_volumes dict whose eight names were looked up
from the very meta file compose_volumes reads itself; zammad's init and
railsserver extra mounts were the volumes.yml entries minus the two
services, which the meta now lists (the when-gated OIDC binds included).
Conditionality that meta cannot express stays in code, but in role-local
lookup plugins instead of template loops: matrix_bridge_mounts renders
the per-bridge registration mounts, nextcloud_extra_volumes the
feature-gated whiteboard/recording volumes (type: volume entries are
declared unconditionally by compose_volumes, so gating must happen
before it).
Parity is proven, not assumed: every mechanically rewritten template is
byte-identical to HEAD after the transformation regexes (95/95), and the
semantic roles render identically through the real functions against
HEAD's meta - zammad across six sso/tls polarities, all seven erpnext
services, mailu's top level. make test-lint passes complete.
Also: suitecrm's compose template pins container_extra_hosts with
host_alias (the SOCKS proxy URL host.docker.internal never resolved in
compose+tor), and the five disabled mautrix bridge addons use the
empty-default config lookup so templating the applications dict stops
emitting 125 errors per batch in every CI job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
unsafe-inline re-admits the inline execution an XSS needs and unsafe-eval re-admits string-to-code, so a CSP relaxation must be verified against the running app rather than copied from another role. The new lint requires a nocheck marker plus a Reason on the flag's own line, so a marker cannot drift to the neighbouring directive.
The 197 pre-existing grants across 70 roles are checked out with a TODO: they were never verified and still have to be.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
roles/__init__.py makes roles a Python package, so every pytest run leaves a roles/__pycache__ behind. The required_by coverage lint walked ROLES_DIR.iterdir() unfiltered and reported that cache as roles/__pycache__/meta/services.yml (missing), i.e. as a non-invokable role lacking required_by.
Enumeration now skips names starting with . or _. The only such entries under roles/ are .gitignore and __init__.py, both files that is_dir() already excluded, so no real role leaves the check: 256 -> 255 directories, the single loss being __pycache__. With the cache present the lint still fails on an injected roles/sys-svc-covtest/ and lists only that offender.
A sibling fix for the same class lives on other branches as 38c8b59fe, covering tests/integration/test_roles_have_meta_main.py and test_roles_naming.py -- different files, no overlap with this one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ruff bump in 6320da703 changed how the formatter picks quote style
inside nested strings and where it wraps a generator argument. Both
files were written before that landed and this is the first autoformat
pass since, so make quality rewrote them.
Formatting only; no assertion, message or rule changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
72fd9c870 required the onion in every variant but exempted variant 1, on the
argument that the minimal round is the one place the stack runs without it. That
argument does not hold: the onion is part of the deployment under test, not a
feature bolted onto the full shape. A round that skips it is not a smaller
deployment, it is a different one.
So the exemption is gone and 95 variant-1 entries pin the onion on. Two
exclusions remain, both read from data rather than a comment: a role with no tor
block has no onion to pin, and a role whose meta/services.yml pins tor.enabled to
a literal false has already declared it has none.
That put the rule in conflict with test_coverage, which requires both polarities
of every dynamic flag: with tor on everywhere, the false side has no variant left
to live in. 94 roles take the exemption that contract already provides for
exactly this - "flags whose group-derived value is a hard invariant that variants
must never override", the same one 4e6ef2542 used for the seaweedfs frontend.
web-app-nextcloud needs none: its variant 8 pairs the clearnet-only PeerTube and
keeps the false polarity.
Measured, because the declaration is not what CI runs. assign_tor gives the onion
to every second capable entry with the run number as the parity switch, so more
capable variants do not mean proportionally more onion jobs: across five roles
and 21 entries the count goes 8 -> 10 on an even run and 9 on an odd one. The
clearnet path also survives - checkmk reports v1=false on run 0 and v1=true on
run 1, so both states are still covered across two consecutive runs. What the
94 markers give up is a declarative guarantee in variants.yml, not the coverage
itself.
The real cost is weight, not job count. web-app-nextcloud variant 1 goes from
3.09GB reservation / 7.27GB limit to 18.26GB / 43.01GB once svc-net-tor and its
closure join, so the cheap round is no longer cheap. Every variant stays under
the 32/64GB budget, and the same effect applies to the other 94 unmeasured.
Verified: tests/lint 336 passed / 307 subtests, tests/integration/roles/meta/
variants plus the role-meta lints 39 passed. The rule was checked in both
directions - it reported the variant-1 offenders before the flips and none after.
Not measured: what the heavier rounds do to wall-clock, against swarm jobs capped
at 355 minutes and one that has hit that ceiling before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
7a70d27b7 made a variant that switches SSO on also switch the onion on, because
that pairing is where the platform breaks in ways clearnet never shows. The same
argument reaches further than SSO: a Secure cookie is dropped, an https identity
provider refuses to embed, and crypto.subtle is undefined for any feature on a
non-secure origin, not only for a login. A variant that never runs the onion is
not testing the deployment users get.
So the rule now covers every variant of a role that declares a tor block.
Variant 1 stays exempt - it is the minimal, partner-free round every role
declares and the one place the stack is exercised without the onion at all - and
a role whose meta/services.yml pins tor.enabled to a literal false is skipped
outright, because that declaration already says the role has no onion and needs
no comment repeating it.
That found 137 variants. 101 were variant 1. Of the rest, 28 are the LDAP-only
auth-isolation rounds, which gain real coverage from running LDAP over the onion
and are flipped on here. Four cannot: svc-net-tor is the provider itself, and
nextcloud's variant 8 pairs the clearnet-only PeerTube. Both carry a marker
naming the reason; opentalk and coturn need none, since their services.yml pins
the literal false the rule now reads.
Flipping them exposed a contract that contradicted itself. test_auth_isolation
and test_non_baseline_explicit_disables demanded `shared: false` beside
`enabled: false`, while variants-static-override flagged that same line as a
static override - which is why all 133 of them carried a nocheck marker. The
demand only makes sense where `shared` is dynamic: a literal base value cannot
leak through deep-merge, so restating it protects nothing. Both contracts now
ask for it only in that case, and the 133 lines are gone.
The result is a stricter rule with fewer lines: 189 insertions against 193
deletions.
Verified: tests/lint 336 passed / 307 subtests, tests/integration/roles 107
passed / 610 subtests. The rule is checked in both directions - it reported 137
offenders before the resolution and none after.
Not measured: 28 additional onion rounds cost matrix wall-clock, against swarm
jobs already capped at 355 minutes and one that has hit that ceiling before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The only shape check on meta/variants.yml rejected a single key, `compose`.
Its docstring stated the actual rule -- top keys must be payload roots -- and
never enforced it, and had itself gone stale: it still named `credentials`,
the root f713ba835 retired. That is how the previous commit's four blocks
passed every gate while writing where nothing reads.
The legal roots are now derived from the meta files instead of restated: every
roles/*/meta/*.yml stem, plus addons when that directory carries specs, minus
main and variants, which are mechanism rather than payload. A role adding a new
topic makes its root legal without touching this test, so the list cannot drift
from the loader again.
compose needs no special case anymore -- it is not a meta file, so it fails
like any other unknown key, as does a typo such as `servicse`. Verified by
running the pre-fix keycloak and lam variants through the new whitelist: all
four `credentials` blocks are rejected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SSO over a plaintext onion is where this platform actually breaks, and none of
it shows on clearnet: a framework's Secure session-cookie default makes Chromium
drop the cookie, an https identity provider cannot be embedded in an http page,
and crypto.subtle is undefined because an onion is no secure context. Pixelfed,
Taiga, Joomla and WordPress each cost a red job over exactly that. The
combination is only exercised if a variant deploys both, so the lint requires
it: services.sso.enabled true implies services.tor.enabled true.
69 variants enable SSO; 59 already complied. Of the ten that did not, seven are
flipped on because the mechanism is proven there - matrix and nextcloud both run
SSO behind the onion in variant 0 already (20ddda8c1), and their later variants
were pinned false only by the blanket rule from 1d9d355bc. Nextcloud variant 1
stays false: SSO is off there, and it is what gives the coverage test its
false-pinning variant.
Three roles opt out, each for a structural reason named in the marker. jitsi and
bigbluebutton pin tor.enabled false in meta/services.yml because the conference
needs getUserMedia, which no plaintext onion can offer. seaweedfs gates its own
sso flag on 'svc-net-tor not in group_names', so on a tor node there is no login
flow to test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Under swarm an unpinned role's volumes are rewritten onto the shared NFS
export while its services run one task per node. A role that declares no
RDBMS keeps its state in-process, so several replicas then write the same
SQLite file, Mnesia store or search index over NFS and corrupt it. This
already cost a deploy: openclaw ran three replicas against one shared
export and its state directory answered "file is not a database".
The existing unpinned-no-engine-volume lint states the right rule but
detects state through a hardcoded path list, so app-specific paths slip
through and each case gets pinned reactively after it fails. Add a lint
that joins three facts the roles already declare: the volume is NFS
backed (asked of the production swarm_nfs_backed, not re-derived), it is
mounted read-write, and the role runs no RDBMS. Share-safe payloads carry
a justification marker.
Replicas resolve service-first and fall back to the primary entity, and
a role whose swarm mode is disabled is skipped; both cases exist in tree.
Also correct the requirement and the design doc, which still described
the per-volume opt-in model that b09598dc8 replaced with placement
derived backing. Requirement 023 contradicted itself: line 76 already
described the implemented behaviour.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The applications registry is built before any role-vars binding, so a
`{{ SOME_ROLE_VAR }}` in meta/*.yml stays an unrendered Jinja string and
silently resolves to nothing. The sibling rule already covers
`application_id`; this extends it to every key declared under
`roles/*/vars/`.
String literals, keyword arguments and filter or attribute names are
stripped before matching, so `lookup('domain', ...)` and
`lookup('rbac_group_path', application_id=...)` are not mistaken for
variable reads. Scope stays at meta/*.yml: files under meta/addons/ are
rendered inside the role's own play, where role vars are bound.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
categories.yml moves from roles/ to the repository-root meta/, joining packages.yml. utils/roles/categories.py owns its location plus the tree-walking helpers that were stranded in entity/name.py, so no caller spells the path or duplicates the lookup.
meta/distros.yml becomes the SPOT for every distribution: os_family, Galaxy platform name, dev-environment base image, and the pkgmgr/pkgmgr-virgin/environment reference templates. It replaces seven scattered copies (DISTRO_FAMILY, VALID_DISTROS, _DISTROS, _CANONICAL_PLATFORMS, two shell arrays, a Makefile literal). INFINITO_DISTROS is no longer a static default but generated by utils/env/handlers/infinito/distros.py; the shell consumers read it through the env layer.
Drops dead CLI surface found along the way: --categories-file and --roles-dir on the provision CLI plus three unused parameters of generate_dynamic_inventory, and the positional roles_file of the invokable categories command.
tests/lint/repository/test_distros_spot.py pins the copies that cannot call the SPOT (workflow literals, docs table) and forbids composing a pkgmgr reference outside it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
categories.yml moves from roles/ to the repository-root meta/, joining packages.yml. utils/roles/categories.py owns its location plus the tree-walking helpers that were stranded in entity/name.py, so no caller spells the path or duplicates the lookup.
meta/distros.yml becomes the SPOT for every distribution: os_family, Galaxy platform name, dev-environment base image, and the pkgmgr/pkgmgr-virgin/environment reference templates. It replaces seven scattered copies (DISTRO_FAMILY, VALID_DISTROS, _DISTROS, _CANONICAL_PLATFORMS, two shell arrays, a Makefile literal). INFINITO_DISTROS is no longer a static default but generated by utils/env/handlers/infinito/distros.py; the shell consumers read it through the env layer.
Drops dead CLI surface found along the way: --categories-file and --roles-dir on the provision CLI plus three unused parameters of generate_dynamic_inventory, and the positional roles_file of the invokable categories command.
tests/lint/repository/test_distros_spot.py pins the copies that cannot call the SPOT (workflow literals, docs table) and forbids composing a pkgmgr reference outside it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ruff toolchain was unpinned, so host venvs, the lint container and CI
installed different releases over time. That drift is why lint-python only
failed on CI: locally ruff 0.15.x preview-gated ISC004/FURB192 and (falsely)
fired S310 on urllib.request.Request, while CI's 0.16.0 has ISC004/FURB192
stable and drops the S310-on-Request false positive.
Pin ruff==0.16.0 in pyproject.toml's dev extras (Dependabot keeps it current)
and have utils.install.lint.ruff read that pin and reinstall whenever the
found version drifts, so every environment resolves the same ruff. Then fix
what 0.16.0 flags: parenthesise the implicit string concatenations (ISC004),
use min() over sorted()[0] (FURB192), drop the now-unused S310 noqa on the
Request() builders (RUF100), and delete the narration comments that the
comment lint surfaces in the touched files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolves the runner Dockerfile to the lint-standard Acquire::Retries=3
and keeps the collabora TODO wording; hardens the merged tree for the
stricter lints (apt retries in the tor Dockerfile, shell-or-true
markers, comment cleanup).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Read INFINITO_LIFECYCLES from default.env via a new tested_lifecycles() helper; derive TESTED_LIFECYCLES, the README overview envelope, the guide-select role filter, and the invokable-lifecycle lint from it. Add "maintained" to the envelope and drop the redundant hardcoded "alpha beta rc stable" strings from workflows, CLI help, and docs.
Guide selection now skips pre-alpha/eol roles (153 -> 117 testable).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rewrite all 254 role descriptions to a uniform noun-phrase schema
(function-first, no leading verb, no trailing period, <=120 chars) and
enforce it via the extended description lint. Rename 12 role README H1
titles for clarity (e.g. Backup Container to Local -> Backup Docker
Volumes, CSS Service -> Corporate Design, NGINX Legal -> Imprint) and
regenerate their Cosmos and Quick Setup sections. Overview table
regenerated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add cli.build.readme.overview (make readme-index / make docs) rendering a
roles overview in the root README: Name, Lifecycle, Homepage, Video,
Description, Integrated with, Install. New info-media lint requires
homepage + video in every invokable role's meta/info.yml (opt out via
'# nocheck: info-media' in meta/main.yml); backfilled 147 roles with
researched, adversarially-verified homepages and official videos.
Retire the 'unsupported' lifecycle value: fold it into 'eol' as a valid
live end-of-life status (shipped but untested/unmaintained) across the
stage lists, symbol glossary, allow-list lint and docs. Retag confluence,
jira, minio, phpldapadmin to eol and magento to pre-alpha.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.
Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The checkmk OMD site cannot live on NFS (fcntl locks return EBADF) and must stay node-local, but an unpinned single-replica service loses that data on a swarm reschedule. Pin checkmk to the manager so the task never leaves the node holding its data, and teach the nfs-false-data-loss lint to accept a manager-pinned role as justification (dropping checkmk's now-redundant per-volume opt-out).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
