- Declare `__all__` in every package `__init__.py` whose `PROJECT_ROOT`
is imported by its modules (163 packages), so the constant counts as
exported; list every public name where the package defines more.
- Empty the 15 package `__init__.py` files whose `PROJECT_ROOT` no module
imports.
- Declare the public API of `utils.github.variant.axes` in `__all__`,
re-exports included, instead of silencing them with `noqa: F401`; the
list matches the one `feature/store-i18n` already carries.
- Use `import unittest.mock as mock` in `test_applications.py`, as the
rest of the suite does.
- Spell `_NON_PRINTABLE` in `utils/yaml_bootstrap.py` as the equivalent
BMP-only positive class; it matches exactly the same 2111 code points
as the old negated class across U+0000..U+10FFFF.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
No conflicts across 121 incoming commits. The one file both sides changed is
tasks/stages/01_constructor.yml: upstream parks an applications carrier for
the swarm bootstrap roles after the add_host tasks, and keys the carrier on
the tokens file as well. This branch's mesh bootstrap opens and drops its own
block-scoped carrier before that point, so both keep working as written.
Upstream also brings ruff 0.17.0, distro-derived env keys, a scoped act outer
teardown, the stalwart relay rule, the onion apex redirect and image and
version updates; ruff 0.17 reformats nothing on this branch.
ruff 0.17.0 adds `datetime` -> `dt` to the default aliases of
flake8-import-conventions, and the project selects ICN, so lint-python
failed with ICN001 in seven files.
The files now import `datetime as dt`. ruff's unsafe fix also renamed
two local names in utils/github/playwright_summary.py into the alias:
the `_to_iso_z` parameter and the parsed value in
`_parse_iso_timestamp`. Both are now called `moment`; with the bare fix,
nine tests of test_playwright_summary.py failed with UnboundLocalError
and AttributeError.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The guard against hardcoding a topology: a role carrying networks.meshes or
networks.interfaces must ship it empty. The shipped layouts keep their
subnets inside NETWORK_MESH_POOL, their ports unique, and every membership
names a declared mesh.
The guarantees were asserted against the declared swarm and data pair,
so they proved two meshes work rather than that the planner works. A
deployment declares its own, and the review asks for any number of them with
the nodes arranged however the client wants.
The same invariants now run over generated topologies: one to eight meshes,
three arrangements each, hosts dealt out so some belong to several meshes and
some to none. 2247 subtests, four seconds, and the planner needed no change to
pass them.
A variant that pins `services.tor.enabled: true` while enabling a role whose
own `meta/services.yml` pins `tor.enabled: false` declares a combination the
deployment cannot hold. The role rides no node onion, so on that round its
surface lives at a clearnet host no onion browser reaches, and the dashboard's
`frame-src 'self' http://*.<onion>` forbids embedding it outright. Run
36651124458 job 109692287333 shows the result: five specs fail at once,
dashboard tile, matomo tracker, css injection, logout and the coturn metric
label, all on hosts that are the only non-onion ones in the whole job.
web-app-nextcloud variant 11 was such a round: bigbluebutton on, tor on, while
bigbluebutton declares no onion. Variant 8 already answers the identical
question for PeerTube (65538c40da, "clearnet, because the OIDC plugin rejects a
plaintext onion issuer"), so variant 11 now does the same. With that,
bigbluebutton never meets a tor node and needs no consumer gates of its own.
Two roles keep a gate, for reasons that differ from each other:
web-app-seaweedfs is not clearnet-only. It deploys on tor as the object store
other roles reach over the internal network, and only its browser frontend is
off there, which `frontend.enabled` on main already says. Its logout, dashboard
and prometheus declarations claimed a surface that round does not deploy, so
they take the same gate the file's own frontend and sso blocks use.
web-svc-coturn belongs on a tor node and cannot serve one: STUN/TURN runs on
UDP plus a 20000-port relay range that no onion descriptor can publish, and its
only HTTP is a gateway answering a plaintext OK. Deploying it is right,
advertising it as a prometheus consumer is not, because that scrape goes
through the browser path. It is exempt from the rule itself, since it claims no
visitable surface, and its `dynamic-flag` markers now carry the reasons the
rule's own message asks for; one of the four was a duplicate of the line below
it and is gone.
The lint states the rule. It found these five pairings on its own, matching a
hand count made before it existed, and reports none after the variant fix.
`test_sso_variant_requires_tor` covers the complementary half, every variant of
a role that has an onion must pin it on, and exempts the clearnet-only role
itself; this rule covers the consumers that bind one. The suppression marker
belongs on the bound role when the reason is that role's own shape, because a
consumer-side marker would also silence every other clearnet role in the file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Of the 76 open code scanning alerts, 39 point at code that can change.
svc-ai-mcp-adapter:
- A path argument that is empty or made only of dots is refused.
Percent-encoding keeps a separator inside its segment, but "." and ".."
need none, so an upstream that resolves dot segments could answer from a
path the contract does not name.
- parse_duration splits on the units. The pattern it used before rescanned
a long run of digits from every start position.
Tests and the README generators:
- Every Jinja environment built with autoescape=False and a "noqa: S701"
uses select_autoescape now, as six test modules and
utils/templating/jinja.py already did. It escapes templates named
*.html, *.htm and *.xml, which none of these call sites loads.
- Dead code goes: a doubled "rel" block in the email lint, an unread
_SKIP_DIRS, and a module that _resolve returned to no caller.
- Two "except ...: pass" blocks become contextlib.suppress and
assertRaises.
- Nine test modules import unittest.mock one way instead of two, and one
lookup test imports its module once.
Closes#807
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Eight meta/variants.yml repeated their last entry verbatim. The six svc-* roles
held the same tor.enabled override twice; fusiondirectory and phpldapadmin held
a third entry identical to their second, nocheck comments included.
The matrix plans a row per entry, assigns it its own axes and spends a deploy
job on it, around a hundred minutes here. The repetition is invisible in the
plan table, which lists rows by role#variant, so the two read as different rows
and only their identical embeds and weight betray them.
test_variants_distinct.py compares the parsed payload rather than the file text,
so key order, indentation and comments do not hide a duplicate, and null
normalises to {} because a bare list item and an explicit {} declare the same
no-override entry. A role whose variants are told apart by something outside the
entry opts the file out with `# nocheck: variants-duplicate` and a reason; no
file needs it today.
Every deletion is the trailing entry, so the surviving indices keep their
meaning and nothing is renumbered. The vanishing tokens are svc-db-elasticsearch#1,
svc-db-memcached#1, svc-db-rabbitmq#1, svc-db-redis#1, svc-db-typesense#1,
svc-prx-openresty#1, web-app-fusiondirectory#2 and web-app-phpldapadmin#2;
`git grep` against the commit finds no consumer naming any of them.
Negative control executed: re-adding the redis duplicate fails the lint with
"svc-db-redis: variants #0, #1 declare the same override", removing it passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cc6da24174 measured Alpine on nginx:1.31.6-alpine alone and recorded curl as
something the family ships. nginx installs it: /usr/bin/curl is there at 272432
bytes, and at 321880 bytes in the Debian nginx:1.31, while redis:8-alpine and
node:26-alpine measured the same way have no curl at all. The table therefore
passed a curl flavor on any alpine image, which is the failure the rule exists
to catch, spelled with the rule's own name on it.
The family now holds what busybox provides, sh, wget and nc. What one image
adds on top of its family goes in IMAGE_PROVIDES, keyed on the image name with
the registry prefix dropped, and nginx is its only entry today.
The Debian half was re-measured while checking this, over /usr/bin, /bin,
/usr/local/bin and /sbin rather than one path: debian:13-slim, node:26-slim and
python:3.13.15-slim carry /bin/bash at 1298416 bytes and none of curl, wget or
nc. bash is Essential in Debian and the other three are not, which is the whole
difference between the stripped tag and the full one.
Nine cases ran against the predicate. The two that matter are curl on
redis:8-alpine and on node:26-alpine, which passed before this commit and flag
after it; curl on nginx still passes, and the six alpine declarations the tree
actually holds stay green, fediwall and nextcloud on nginx's curl, hugo and
littlejs on busybox wget, taiga on either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
c31c62efa checked four of the repository's 86 healthcheck declarations. It read
the probe's CMD form only, scoped itself to slim, and counted bash as a shell
builtin, which is where most of the gap sat: tcp, connect and http_status all
render CMD bash -c over /dev/tcp, so 31 declarations were skipped as needing
nothing. It now examines 18, and what it still skips it skips for a reason.
bash is a program like any other here. Its absence is not hypothetical: alpine
has no /bin/bash at all, and its busybox /bin/sh does not implement /dev/tcp,
so the redirection those three flavors are built on needs the real shell rather
than any shell. No role pairs them today, which is why this is prevention and
not a fix.
The CMD-SHELL form is parsed instead of ignored: || reads as alternatives, &&
and ; as a sequence needing all its parts, and shell builtins are dropped so a
trailing || exit 1 cannot make every requirement look satisfiable. That brings
in nc, http and the msmtp compositions, six declarations the old shape could
not see. http is the one that needed the alternatives: it runs wget or curl and
is satisfied by either.
alpine moves from a docstring exemption into the scope with a measured table
behind it. Read out of the images with docker create and docker cp, which
reports an absent path rather than a shell's opinion of it: debian slim carries
/bin/bash at 1298416 bytes and none of curl, wget or nc; nginx alpine carries
curl at 272432 bytes as a real binary, wget and nc as busybox applets, and no
bash. Adding a family means measuring it the same way.
Vendor images stay out. Their contents are not knowable from the tree and most
carry no Dockerfile of ours, so the rule's remedy has nowhere to land; skipping
them honestly beats a table that rots.
Sixteen cases were run against the predicate, one per defect class and its
satisfied counterpart. Two corrected the work: nc is a busybox applet on alpine,
which the first table omitted and which made nc-on-alpine flag wrongly, and this
file's own prose named a Debian codename, which test_no_debian_codename_tags
caught in the gate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
10c5b35e88 moved simpleicons and roulette-wheel from node:26-bookworm to
node:26-slim. Both declare healthcheck flavor curl, which renders
CMD curl -f <url>, and neither Dockerfile installed anything. Measured with
docker create and docker cp: node:26-bookworm carries /usr/bin/curl at 280800
bytes, node:26-slim carries neither curl nor wget. The probe therefore fails
from the first beat and the container never reports healthy.
sys-svc-compose waits on the database container alone in compose and on every
stack task in swarm, so the consequence splits by mode. Under swarm the task
dies with non-zero exit (137): dockerexec: unhealthy container, is rescheduled,
dies again, and the converge wait fails after 600 attempts. Five jobs across
the two runs on a534d5d46 died that way, and none of them on the role they are
named after: ERPNext#0, Decidim#0, Hugo#0, Nextcloud#9 and Nextcloud#12 all
carry web-app-dashboard in their closure, dashboard enables simpleicons, and
the failure reads 'not converged: simpleicons_simpleicons'.
All five are swarm and tor, which reads like the tor axis until XWiki#1 is put
beside them: swarm, tor, green, and simpleicons is not in its closure. The
discriminator is the service, not the axis.
Of the seven roles 10c5b35e88 rebased, bridgy-fed and postmarks probe over tcp,
and flowise and bookwyrm install curl themselves. These two were the remainder.
roulette-wheel had not been deployed yet when this was written, so it is the
same defect caught before it was observed. The install goes into the runtime
stage of the multi-stage build; the builder stage would have left the runtime
image unchanged and nothing would have gone red.
The lint reads the binary out of the probe the declaration renders rather than
restating a flavor-to-binary table, so a flavor that changes its command moves
the check with it, and it locates the flavor line per service so one marker
cannot silence a sibling. Its scope stops at slim: nginx:1.31.6-alpine was
measured too and ships curl as a real 272 KB binary plus busybox wget, so the
four alpine roles that declare those flavors are served by their base, and
widening the rule to them needs that measurement per image rather than an
assumption from the word alpine.
Not built. The image with the new layer has not been produced, and whether a
second fault waits behind the healthcheck is decided by the next deploy. The
apt path itself is not blind: BookWyrm#1 installs curl the same way on
python:3.11-slim and is green in this run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The private blocks were written out wherever something needed to say "this is us": NETWORK_TRUSTED_PROXY_CIDRS listed all three, Tor's SocksPolicy repeated them, unbound's access-control repeated them again. Four copies of a fact, and nothing tying them together, so narrowing one narrows nothing else.
Worse, the block role networks are carved from was never declared at all. The allocator behind 'infinito contributing network address suggest' derives its umbrella from the subnets already in use, so the tree defined the rule instead of the rule defining the tree. Anything consuming "our own address space" had to guess it, and a guess that is one role out of date silently excludes that role. Measured: 106 of 107 role subnets sit in 192.168.0.0/16 and one does not, so a firewall range derived from the pool by eye would have cut BigBlueButton off.
The blocks are now named once and everything else is derived, NETWORK_TRUSTED_PROXY_CIDRS included, which renders byte-identically to before for its four consumers.
BigBlueButton stays outside because its upstream pins both the block and its gateway, which meta/services.yml repeats as upstream_host 10.7.7.1. The exception list therefore carries the role id rather than the CIDR: the block is already declared in that role's meta/networks.yml, and repeating it here would be the same duplication one layer up. The role_subnets lookup resolves the ids and raises on a role that declares no subnet, because a silently dropped entry would narrow whatever firewall or trust list consumes the result.
A lint keeps the declaration honest in both directions: every role subnet must sit in the declared space, and an exception whose block the pool already admits is rejected so it cannot outlive its reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
e90cac4ee2 staged the lint but not what the gate rewrote after it. Whitespace only; the join, the fix hint and the assertion are unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
API is a plain mapping, so an inventory that configures three providers replaces the whole declaration from group_vars/all/18_api.yml instead of merging into it. inventories/development/default.yml declares openai, anthropic and openrouter, so github, google and jira did not exist in CI at all and lookup('api', 'github.client_id') raised.
A raising lookup inside a value the applications renderer is templating does not fail the render: _templar_render_best_effort returns the string untouched and, for a value carrying a non-env lookup(, refuses the fallback as well. The raw template text reached plugin_value.enabled | bool, which coerced it to False and emitted the deprecation warning that fails the deploy gate. Run 34893363518, job 104153194109 lost web-app-nextcloud to exactly that, for three integrations at once. The configvalue entries would have stored template text as the client id.
utils/api.py merges the scope's override over the declaration, so an override narrows values and never the provider set, and holds the enable rule: every credential a provider declares must carry a value. The api lookup reads through it; the new api_enabled lookup gates on it and returns a real boolean instead of a string a filter has to coerce.
The same class sat in group_vars/all/16_ai.yml and roles/svc-ai-litellm/meta/secrets.yml as bare API.<provider>.<key> dereferences; both now go through the lookups.
Two lints keep it shut. test_addon_enabled_renders_to_bool renders the whole applications map against a scope with API blanked, the most hostile override an inventory may legally write, and requires every addon gate to come out a boolean. test_api_accessed_via_lookup forbids a bare API dereference in Jinja under roles/, group_vars/ and inventories/.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
roles/test-e2e-cli and roles/test-e2e-playwright run only in CI, against a
throwaway cluster whose credentials are generated per deploy and discarded with
it. no_log there protects nothing and costs the one thing a red test job exists
for: a masked task reports that its output was hidden, and the whole matrix
round has to run again to learn what a single assertion saw.
The rule is absolute, so it ships without a nocheck escape hatch. A test task
whose output looks too sensitive to print calls for extracting the field the
assertion reads, not for blinding the task.
A second case asserts the glob still reaches the roles. A path rule that
matches nothing passes forever, which is how test_gateway_backend_registered
nearly went quiet earlier today.
Both roles are clean right now, so this starts green. Reproduced against an
injected no_log in run_one.yml to confirm it is not vacuous.
Note for whoever hits it: test_credential_tasks_are_masked scans roles/*/tasks
and roles/*/handlers without excluding these two, and requires no_log on a task
whose printed arguments name a password, secret, token, api_key or private_key.
Nothing in the test roles trips it today, but a future task that does would
deadlock the two rules. The exemption belongs in that lint's file selection.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ansible renders the values of a structure loaded through include_vars and leaves its keys as written, so the mapping key a564385068 wrote as a Jinja expression reached Nextcloud as its own source text. occ config:app:get sociallogin custom_providers on a live variant-1 instance returns "groupMapping": {"/{{ lookup('rbac_group_path', application_id=application_id, role='administrator') }}": "admin"} while every neighbouring value of the same provider is rendered, scope among them.
Keycloak emits /roles/web-app-nextcloud/administrator, so the mapping matched nothing and the account sociallogin provisions on first login stayed out of admin. occ group:list on that instance shows admin holding only the local administrator while keycloak-ac165525-e9e8-4230-9381-13b1acd83fc0 exists and belongs to no group, and every Playwright spec that needs an admin settings section answered 'Access forbidden'.
An independent reproduction outside this repo showed the same split on ansible-core 2.21.4 and 2.20.6: in a payload of the same shape the sibling value rendered and the key did not, so it is not a version artifact. Nextcloud 34.0.3 settles the other half: Manager.php hands an admin the unfiltered settings of a section and ai is a core section of the always-enabled settings app, so an admin cannot receive 403 there whatever is installed.
Writing the whole mapping as one expression makes jinja2_native, which this deployment has on, return a real mapping with the key computed. The lint takes the class: no meta payload may carry Jinja in a mapping key. It found exactly this one case in the whole meta tree, and it ignores a line that itself opens with the expression so the corrected form does not trip it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
What `make quality-high` rewrote on its own, kept apart from the changes it ran
against so neither has to be read through the other.
The matomo healthcheck test moves from a flow sequence to a block one, which is
the formatter's threshold for a long line and not a change to the command: both
revisions parse to the same two-element list, escapes included. The healthcheck
timing lint loses two blank lines between methods.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rule read the character after `timeout:` and objected to a digit, so a
duration bypassed the helper the moment it was given a name. That was already
reachable before, and it became easy to reach now that a poll interval is
allowed to stay unscaled: the two look identical at the call site and only one
of them is correct.
They are opposites, and the rule now says so. An upper bound must scale, because
the answer over a Tor circuit arrives later and a fixed cap expires before it
does. A poll interval must not, because scaling one makes a slow target notice
its element later rather than sooner, which is the reverse of the tolerance the
scaling exists to buy. So `timeout:` rejects a CONSTANT_CASE name as well as a
digit, while `waitForTimeout` and `setTimeout` keep matching on a leading digit
only and accept a named interval.
Across the 1307 `timeout:` sites in scope the sharpened rule finds exactly one,
and it is a true positive that wanted saying out loud. The n8n LiteLLM spec
passes a completion timeout into an `n8n-nodes-base.httpRequest` node: not a
Playwright wait at all, but a parameter for a request n8n makes to LiteLLM
inside the deployment, which never crosses a circuit. Leaving it unscaled is
right and was invisible; it now carries the sanctioned marker and its reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The marker names the reason a role has no upstream homepage or demo video, and
it sat in meta/main.yml because a comment-only info.yml parses to an empty
document, which the schema lint rejected outright. So the exemption lived one
file away from the two fields it explains, and a reader looking for the video
found neither the field nor the reason it was missing.
The schema lint now accepts an empty document exactly when it carries the
marker, and that is what allows the move. The media lint reads the marker from
info.yml rather than main.yml, and its failure message names that same file, so
the fix it suggests lands where the absent field would have gone.
A hundred and two roles follow. Every occurrence was a standalone comment on
the second line of main.yml, directly under the document start, so the move is
a line delete and an insert at the same offset. Seventy-three roles already had
an info.yml and gain the line; twenty-nine get one whose whole content is the
document start and the marker. All hundred and two reasons carry over verbatim,
including the thirty-five bespoke ones.
Nothing downstream sees a change. Of the four readers of meta/info.yml, three
coerce a null parse through `or {}` behind an isinstance guard, and
utils/cache/applications.py skips a falsy parse before assignment, so
applications.<role>.info stays absent for the twenty-nine exactly as it was
while the file did not exist.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
start_interval is the cadence while start_period runs, and only while it runs.
Once the window closes the probe falls back to interval, so a service that
becomes ready just after it pays a whole tick anyway and the cadence never
reached it. Six of the seven flavors were shaped that way, Curl worst: a thirty
second window against a one minute interval.
Measured across 151 converge waits in four job logs, 63 sit exactly on a tick
boundary, 36 in the thirty second step and 27 in the sixty second one.
Attributed by application, web-app-seaweedfs is the largest at 1501s over
sixteen waits with a median of 57.7s: it probes with curl and fell straight into
the gap. web-app-zammad probes with connect, whose window is fifteen minutes,
and its 154s median is real boot time with nothing to recover.
So the window moves to ten minutes and the steady state interval to two, and the
flavor overrides that sat below them go: Curl, Wget and Connect lose interval
1m, Http its twenty second window, Nc its ten. Connect keeps fifteen minutes.
Neither number costs anything while a container works. start_period delays
nothing, it only stops failures counting, and interval no longer touches
convergence at all since that hangs on the window alone; it is steady state
load, which drops from 120 probes an hour per service to 30.
Forty six role files follow. Twenty seven declarations go that repeat the new
default or undercut it, an undercut window being the defect itself. Eight
windows longer than ten minutes keep their value and now say why. The literal
blocks and the three Dockerfiles this repository builds spell every timing out,
because outside the lookup an omission takes docker's value and not the
platform's.
A fourth lint rule keeps the shape: start_period must outlast one interval,
checked against the effective values, so raising interval without the window
cannot quietly bring the cliff back.
Two faults in the lint surfaced while building it and are fixed here. It
compared timings as strings, so 600s and 10m read as different values, and it
matched the healthcheck key exactly, so a line carrying a trailing nocheck was
skipped whole, which is how the two swarm-init partials went unexamined.
One regression, bounded. A container that starts and never turns healthy is
neither rejected nor errored, so stack_ready.sh only calls it fatal after three
killed tasks, and each cycle now spans the longer window. That reaches twenty
minutes rather than six, capped there by swarm_stack_converge_retries at 600
with a two second delay, and it lands only on runs that were failing anyway.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three checks over the two places a timing can be written.
In meta/services.yml the lookup supplies the defaults, so a declaration is
either noise or a decision. A value equal to what the service would have
inherited is deleted rather than justified: it changes nothing and hides the day
the default moves. A value that differs is kept and carries its reason.
That reason is required for interval, timeout and start_interval only. Those
three cost something: steady state load for the life of the deployment, the
threshold at which a slow probe counts as failed, and the only timing that moves
the moment a stack reports healthy. start_period and retries buy patience with a
container that is already broken and cost nothing while it works. Demanding a
defence for them would have produced sixty one sentences nobody had evidence
for.
Outside the lookup, in a literal healthcheck block and in a role's own
Dockerfile, an omission is a finding too. Nothing there inherits from the SPOT,
and docker has no daemon-wide default to fall back on, so a missing key lands on
docker's value rather than the platform's. They disagree on start_period, zero
against thirty seconds, which is exactly what left start_interval without a
window in every hand-written block.
Comparison is against the effective default, resolved through the same compose
the lookup uses. Curl probes every minute, Connect grants a fifteen minute start
window, and a prefix raises timeout, retries and start_period further; measuring
against Probe alone would have called out every service that simply agrees with
its flavor.
Negative control for each check. Before the three commits that precede this one
the rule named 47 restatements, 20 unexplained deviations and 34 definitions
outside the lookup. It names none now.
Two limits worth knowing. The line number for a meta timing is found by
indentation and service name rather than by a parser, so an unusually formatted
file makes the rule ask for a reason it cannot anchor; it errs towards asking,
never towards silence. And the Dockerfile reader follows a backslash
continuation but not a comment inside one, which no Dockerfile in the tree uses.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two of them lived inside docs/ next to the pages they render, which is why both had grown a __file__ upward walk to find the repository root, a sys.path insert to import from utils, and a nocheck for each. A generator parked among documents reads as documentation and drops out of the checks that apply to code.
Moving them surfaced six real violations that the old location had hidden: yaml.safe_load instead of the shared cache, two uncached read_text calls, five hard-coded role paths instead of ROLE_FILE_META_SERVICES and ROLE_FILE_META_MCP, two make targets without an alias, and a requirement reference in code. All six are fixed at the root, none with a suppression, and both generators lose their project-root exemptions because a package supplies PROJECT_ROOT.
cli/build/readme joins them so the three generators that write into docs/ sit together. A new lint keeps docs/ to .md, .html, .rst and .nocheck; it enumerates through git ls-files so untracked __pycache__ is not its business. Checked against HEAD, it names exactly the three .py files that moved.
026-unified-addon-syntax stated that the matrix generator stays under docs/requirements/. The lint makes that impossible, so the requirement now records where the generator actually lives.
Verified: every one of the twelve modules under the new tree imports, no old module path survives anywhere in the repository, 482 lint tests and 1246 cli unit tests pass, and both generators run without drift.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ruff B905 flagged zip(entries, anchors) without strict=. The lines directly above reset anchors to len(entries) whenever the counts disagree, so strict=True can never fire and records that pairing; a future edit that drops the alignment now fails loudly instead of silently truncating and skipping the very offenders this rule exists to catch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
roles/<role>/tasks/main.yml gates 0N_core.yml behind
`run_once_<role> is not defined`. Entering that body from a different role
with `include_role: {name: <role>, tasks_from: 0N_core.yml}` bypasses the
gate, because the guard lives in main.yml and tasks_from never reads it, so
the body runs once per calling task instead of once per play.
For sys-svc-compose that means the shared compose pull-lock directory is wiped
again, and files/python/pull.py skips a pull only while its lock file exists,
so every stack deployed so far pulls again; under MODE_RESET it also reaches
01_reset.yml, which brings every project in DIR_COMPOSITIONS down and deletes
the directory holding every application's instance, env and volume paths. For
sys-svc-container it drains whatever topic is pending through the
unconditional meta: flush_handlers at 00_core.yml:53.
svc-runner carried the only two unguarded entries in the tree. This is not a
new convention: sys-svc-container is guarded this way at seven existing call
sites, in svc-opt-ssd-hdd, sys-ctl-hlth-csp, svc-swarm-node, desk-docker, and
twice inside sys-svc-compose itself.
The guard belongs at the call site rather than inside the core file.
sys-svc-compose/tasks/00_core.yml sets its flag as its first task on purpose:
it includes sys-svc-container, and utils/network/routine.yml leads back into
sys-svc-compose, so setting the flag first is what breaks the recursion. The
first-task lint records that rationale. Moving the flag to the end to let the
body self-guard would trade one defect for another.
The rule parses the task structures rather than matching lines: it walks
block, rescue and always, reads include_role and import_role including the
ansible.builtin spellings, and accepts a guard whether `when` is a scalar or a
list. A line anchor is still used for the report and the nocheck, not for the
verdict. A scratch control covers seven shapes; two of them, an inverted key
order and an entry nested in a block, escape a line-based rule entirely.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The SSO gate handed X-Forwarded-Access-Token to every oauth2-fronted upstream, so each of the 28 gated roles received a full-scope bearer for the signed-in user while exactly one, n8n, decodes it. The token also grows with the realm's registered origins, and LAM's Apache rejects a header field above 8190 bytes (Moodle#0, run 33689701895).
Consumers now declare services.sso.oauth2.pass_access_token. Without it oauth2-proxy already answers the auth request without the token, and the gate blanks the upstream header rather than dropping the directive, because nginx inherits proxy_set_header as an array and an unset header would let a client-supplied one through. The new lint keeps declaration and use in step in both directions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
880bec33bb wrote the regex across three lines; it fits on one under the line
limit, so `make quality` rewrites it on every run. The pattern is unchanged -
identical once whitespace is stripped from both revisions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`vars:` on an `include_role` is in scope for everything the include pulls in,
and the expression is evaluated where it is read rather than where it is
written. `sys-stk-full` pulls in the proxy and the database, and each of those
roles sets `application_id` to its own id, so a value written with
`application_id` answers a different question by the time anyone asks for it.
It surfaced as a missing key in a role that never declared the value: a
repository url written by one app was looked up on `svc-prx-openresty`, and the
git task died on an unresolvable argument rather than on anything to do with
the url.
The seven definitions here are the ones the included chain actually reads, all
of them `compose_repository_url` / `compose_repository_ref` sources. Naming the
application literally makes each expression answer the same in every scope it
travels through. Values that stay inside their own role keep reading
`application_id`; nothing downstream evaluates them.
`4ce4236585` fixed the other half of the same mechanism a while back, in
`meta/services.yml`, where a role-var was referenced from a value the compose
role resolves. Same failure, opposite end of the lookup.
The lint fixes the form. It judges only the `compose_*` keys, because matching
every handed-over key reported thirteen values, six of which nobody downstream
reads.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ruff's PTH206 rejects .split(os.sep) on a path. parts[0] is the same first
component here, since the string always has its roles/ prefix removed and so
never starts with a separator.
It reached the branch because the file was rewritten after its ruff run, to
satisfy the raw-walk rule, and the green check belonged to the earlier version.
quality-high's autoformat step caught it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Role vars do not cross role boundaries, so flowise, hermes, homeassistant,
openclaw and openwebui read nothing where they template entity_name into
resolve_service_key. Seventy-seven other roles set it on the line below
application_id.
Ansible does not stop for it. In compose the undefined value renders into task
names as "<< error 1 - 'entity_name' is undefined >>" and the play carries on,
because that path resolves through resolve_compose_cid and never reads the key.
Swarm templates it into a real argument and dies on the first task, and with
any_errors_fatal that ends the play. Reproduced both ways against a replica of
resolve_host_cid.yml: compose ok=4 failed=0, swarm "'entity_name' is undefined".
Nothing could have supplied it either - role vars reach neither a sibling
include nor the parent, so the value was always undefined rather than sometimes
borrowed from whichever role ran before.
The rule that comes with it compares consumption against definition, which no
existing check does; the four scanners under tests/lint/ansible/variables all
run the other way. It skips a file another role pulls in by absolute path,
since that file executes in the including role's scope and reads that role's
value, and it matches the variable rather than get_entity_name, the filter whose
name ends in it. Both exemptions cost a false finding each while I was writing
it. It caught two roles the review had missed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The glob stopped at roles/*/tasks/*.yml while the docstring claimed every
task file of every role. It missed 226 of 916 files - anything under a
nested directory such as tasks/flavor/compose/ or tasks/utils/. The rule
passed and read as full coverage.
The vacuum guard could not catch that. It fails when no file pairs a
docker call with a stack include at all, which protects against an empty
scan, not against a scan that is merely a quarter short.
Widening it surfaces no new finding, so nothing was hiding down there.
The other limit stays and is now written down instead of implied: the
comparison is per file, so splitting the docker call and the stack include
across two task files escapes the rule. Resolving that needs the include
graph rather than two line numbers, and every current caller keeps both in
one file.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The image ships docker-ce-cli and no daemon; sys-svc-container installs
and starts that, and a role reaches it through a sys-stk-* stack. This
role pre-created its overlay network before that include, so on the
compose path the branch that probes the bridge ran against a socket that
was not there yet:
Error connecting: ... FileNotFoundError(2, 'No such file or directory')
It fails late and unrecognisably. The play spends three minutes in the
constructor and the base roles first, the container reports healthy
throughout because its probe cannot ask about a daemon that is not due
yet, and the error surfaces as a network probe rather than as 'docker is
not running'. Swarm hides it entirely, since those nodes have docker
before the play reaches the role.
minio, discourse, openldap, seaweedfs and prometheus all order it the
other way. minio is the closest model: load the stack, create the
network, and let the later flush_handlers start the stack, so the network
exists before anything needs it and docker exists before anything
creates it.
The rule compares the two line numbers in each task file. It caught this
one at line 5 against line 11.
87f762bd7 replaced the named helper lists in the deploy task and
rerun-spec.sh with a glob. There was a third stager. lint-playwright
keeps its own staging tree, so it kept its own list, and every MCP spec
failed to parse there: 8 of 87 roles, the same 'Cannot find module
./mcp-endpoint' the CI lint job reports.
All three lists held different sets - the deploy task four helpers,
rerun-spec.sh three, this one four again but not the same four - and none
of them the one the specs actually needed.
f14b1ab3f6 already fixed this drift once, by adding addon-gating here to
match the deploy stager. Adding a name to a list that exists three times
is what made a third occurrence certain; the list is gone now.
The rule learns about the third stager as well, since one that goes back
to naming helpers is exactly what nobody notices.
sys-stk-front-proxy prefers a role's own templates/proxy.conf.j2 over a
shared flavour, and an override replaces the flavour whole - including
every cross-cutting include living inside it. Expressing your own
locations should not cost you the MCP policy.
It did. f5272fa85f added that policy to both flavours and to nextcloud,
and the one provider whose native path most needed hiding renders from
its own template, so /mcp kept reaching baserow. The deploy was
failed=0 the whole time: the template produced no line at all, and a
green recap says nothing about a line that never existed. Reading the
rendered vhost is what found it.
Two overrides also lacked the prometheus locations, so nextcloud and
checkmk had no /healthz/live and no per-request metrics. That include
guards itself and emits nothing where prometheus is absent.
The required set is the intersection of the two flavours, so a concern
added to both is demanded of the overrides on the next run without
anyone maintaining a list. Location snippets are excluded, being what an
override exists to replace, and the set is only asked of roles that
actually serve MCP - demanding it of the rest would report noise and
teach the reader to skip the report.
A second rule covers a latent case the intersection cannot: a gated
override needs the oauth2 endpoint and the raised header buffers, both
of which live in one flavour only. All six carry them today, and nothing
made them.
b8759a363e closed this for html.conf.j2 and its message said 'every proxy
location', so the class read as done. Three proxying snippets never got
it: ws, upload and media.
They are not covered by inheritance. nginx inherits proxy_set_header as
an array, so a location declaring any of its own discards every one from
the enclosing block, and proxy_pass_request_headers defaults to on. A
grep for the strip found exactly one include site before this commit,
which means there was nothing above those locations to inherit either.
So every websocket, upload and image location forwarded whatever
X-Forwarded-User the client sent, to applications that authenticate by
believing that header. Reached through ws_generic (mastodon, espocrm,
odoo), nextcloud's signaling and whiteboard, peertube's websocket and
tracker, and collabora's /cool/.
The lint asks the file whether it proxies rather than keeping a list of
snippet names. It grants no exemption for an SSO branch: html.conf.j2
carries both, and an exemption keyed on the SSO reference would keep it
green after the strip was deleted from the other branch.
A helper that no stager copies fails at import, and it fails for every
role that imports it at once, because the whole spec file stops loading.
That is what mcp-endpoint.js did to all ten MCP specs: they ran zero
tests rather than one failing one.
The set lived in three places and no two agreed - the deploy task named
four helpers, rerun-spec.sh named three, and the specs required five.
addon-gating.js was already missing from the rerun path before any of
this. Both stagers now glob the runner's own files/*.js, minus the
config, so the directory listing is the set.
The lint derives what must be staged from the requires the specs
actually make, and holds both stagers to globbing, since a set that can
be typed twice will differ twice.
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mount pipeline was split across two layers: compose_volumes and
container_volumes lived as filter modules that ansible's filter loader
rejected on every start ("has no attribute 'FilterModule'", two warnings
per run), while the same-named lookups wrapped them thinly. The filter
bodies now live inside their lookups (the tor_socks precedent), the
filter files are gone, and the render tests moved next to the lookup
tests.
As lookups they see the templating context, so the application_id term
every call site passed back to its own role is gone too: both lookups
read application_id from context and fail loudly when it is missing.
107 templates and nfs_prep.yml drop the term; the calls collapse to
lookup('compose_volumes') and lookup('container_volumes', service_name).
The container_volumes call is now canonical - four leading spaces,
service_name in the service position, | indent(4) - enforced by a new
lint (test_container_volumes_call_shape). Normalizing the 85 literal
call sites surfaced that erpnext keyed its meta/volumes.yml mounts by
container name instead of compose service key, the only role to do so;
its meta now uses the service keys. wordpress's column-0 call, which
rendered its first line at top level, is indented like every other.
Template logic that only restated meta/volumes.yml is deleted: mailu fed
compose_volumes an extra_volumes dict whose eight names were looked up
from the very meta file compose_volumes reads itself; zammad's init and
railsserver extra mounts were the volumes.yml entries minus the two
services, which the meta now lists (the when-gated OIDC binds included).
Conditionality that meta cannot express stays in code, but in role-local
lookup plugins instead of template loops: matrix_bridge_mounts renders
the per-bridge registration mounts, nextcloud_extra_volumes the
feature-gated whiteboard/recording volumes (type: volume entries are
declared unconditionally by compose_volumes, so gating must happen
before it).
Parity is proven, not assumed: every mechanically rewritten template is
byte-identical to HEAD after the transformation regexes (95/95), and the
semantic roles render identically through the real functions against
HEAD's meta - zammad across six sso/tls polarities, all seven erpnext
services, mailu's top level. make test-lint passes complete.
Also: suitecrm's compose template pins container_extra_hosts with
host_alias (the SOCKS proxy URL host.docker.internal never resolved in
compose+tor), and the five disabled mautrix bridge addons use the
empty-default config lookup so templating the applications dict stops
emitting 125 errors per batch in every CI job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
unsafe-inline re-admits the inline execution an XSS needs and unsafe-eval re-admits string-to-code, so a CSP relaxation must be verified against the running app rather than copied from another role. The new lint requires a nocheck marker plus a Reason on the flag's own line, so a marker cannot drift to the neighbouring directive.
The 197 pre-existing grants across 70 roles are checked out with a TODO: they were never verified and still have to be.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rule only fired one way: a stack role with named volumes had to carry
the flag, but a role without any volume could carry it unnoticed. That
entry pulls svc-bkp-volume-2-local into the role's dependency graph and
schedules a backup run over a role with nothing to back up.
_scan() now classifies every invokable role instead of skipping the ones
that need no flag, and returns both halves. The nocheck opt-out still
exempts a role from both directions.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
roles/__init__.py makes roles a Python package, so every pytest run leaves a roles/__pycache__ behind. The required_by coverage lint walked ROLES_DIR.iterdir() unfiltered and reported that cache as roles/__pycache__/meta/services.yml (missing), i.e. as a non-invokable role lacking required_by.
Enumeration now skips names starting with . or _. The only such entries under roles/ are .gitignore and __init__.py, both files that is_dir() already excluded, so no real role leaves the check: 256 -> 255 directories, the single loss being __pycache__. With the cache present the lint still fails on an injected roles/sys-svc-covtest/ and lists only that offender.
A sibling fix for the same class lives on other branches as 38c8b59fe, covering tests/integration/test_roles_have_meta_main.py and test_roles_naming.py -- different files, no overlap with this one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
