- Declare `__all__` in every package `__init__.py` whose `PROJECT_ROOT`
is imported by its modules (163 packages), so the constant counts as
exported; list every public name where the package defines more.
- Empty the 15 package `__init__.py` files whose `PROJECT_ROOT` no module
imports.
- Declare the public API of `utils.github.variant.axes` in `__all__`,
re-exports included, instead of silencing them with `noqa: F401`; the
list matches the one `feature/store-i18n` already carries.
- Use `import unittest.mock as mock` in `test_applications.py`, as the
rest of the suite does.
- Spell `_NON_PRINTABLE` in `utils/yaml_bootstrap.py` as the equivalent
BMP-only positive class; it matches exactly the same 2111 code points
as the old negated class across U+0000..U+10FFFF.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
An MCP client spec signs in through the client's own OIDC button, which
redirects to Keycloak. Every provider put its MCP pairing in the lean variant,
and that variant pinned sso off - so the round that exists to prove the pairing
was the one round without an identity provider. Run 33574056480 shows what that
costs: openwebui's MCP specs reached _shared.js:131 and died on "openwebui OIDC
sign-in button must be visible on the auth page".
The rule now demands two things of the variant that carries the consumers: the
provider pins sso on, and each admitted consumer, in the variant that round
actually pulls in, does not pin it off. The second half matters because a round
deploys every dependency at its own index and falls back to variant 0 only when
the dependency has fewer, so the client shipping alongside is that one and not
the client's variant 0. A role that declares no sso, or pins it to a literal
false because upstream speaks no OIDC, is exempt from a flag it cannot have.
Twenty roles moved to satisfy it. Ten already had a second sso: false pin and
only needed the flip. Six could host the pairing in variant 0, which already
has sso on, and dropped back to two variants. Six kept a third lean variant to
carry the false polarity variants-coverage wants. moodle and mattermost took
swaps instead: moodle's variant 1 is the mandated LDAP-only branch, and
mattermost was already at the three-variant cap with sso on in its variant 2.
nextcloud also turns context_agent on in that round, the ExApp that serves the
endpoint - with it off the round proved nothing it was built to prove.
The rule moves to tests/lint/ansible/services/ because it reads meta YAML and
executes nothing, which is where integration.md says such a check belongs, and
its suppression key joins the catalog it was never listed in.
nextcloud_exapp_reregister becomes nextcloud_exapp_renew: mcp-secret-masking
matches `register:` unanchored, so the substring in the old name read as a task
that prints a secret. register.yml already masks the two tasks that carry one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mcp-consumer-variant guard demanded three things of a provider's MCP
round: every admitted consumer enabled, each of them pinned `services: null`
so it dragged no dependencies along, and no other service left enabled. Two
of those were the rule legislating a round's contents rather than checking
that the pairing gets proven at all.
Keep the first and drop the rest. What else the round carries is the role's
own call, so the tor/litellm exemption, the datastore exemption and the
`allowed` set they fed have no reader left and go with them.
Strip the 63 `services: null` pins the rule forced across 20 variants, all
written bare as `services:`. The topic-emptying capability behind them
stays: apply_topic still normalises None to {} for the inventory payload,
and the unit tests still cover it. Only the uses and the mandate are gone.
That leaves overridden_providers scanning a repository with nothing left to
find, so its three assertions passed over an empty dict and would have
stayed green however the function broke. Rebuild them on a synthetic role
tree, which also covers the case the repository scan never held: a variant
naming a plain flag is not an override.
Removing the pins changes what deploys. Measured across all 20 variants: 18
identical, prometheus#1 grows 23 to 31 roles, moodle#2 grows 31 to 32 and
crosses the memory budget until the following commit shrinks it back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A round deploys a provider once, so the role that pulls it in is the only place that can say how lean it has to be. A variant entry may now carry any application config topic under a service it depends on: a mapping deep-merges, so naming one key changes that key alone, and null empties the topic. Empty was the case that needed a marker of its own, because merging an empty mapping is a no-op and would leave exactly the services the round was trying to shed.
The topic vocabulary and the merge semantics live in utils.roles.applications.topics, which is also where resources.py now gets them rather than spelling the same null-or-merge out a second time. Collecting who demands what stays in the inventory tree, refuses two roles demanding different values for one provider in the same round, and skips a role's own entity entry so the existing services.<own>.users declarations are not read as overrides.
The CI set-cover learns the same fact. Its greedy cover retires a role's variant-0 row when an earlier row of another role embeds it, on the stated premise that embedding pulls the provider's variant 0. A variant that dictates that provider's config breaks the premise from the other side: whatever ran, it was not the provider as the provider declares it. Such a pair now covers nothing, so the retired row keeps its job, its artefact and its verdict.
Measured on the current tree: 20 override pairs, 41 covered rows before and after, so nothing is revoked today. The rule bites the moment an override lands in a row that covers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
7a70d27b7 made a variant that switches SSO on also switch the onion on, because
that pairing is where the platform breaks in ways clearnet never shows. The same
argument reaches further than SSO: a Secure cookie is dropped, an https identity
provider refuses to embed, and crypto.subtle is undefined for any feature on a
non-secure origin, not only for a login. A variant that never runs the onion is
not testing the deployment users get.
So the rule now covers every variant of a role that declares a tor block.
Variant 1 stays exempt - it is the minimal, partner-free round every role
declares and the one place the stack is exercised without the onion at all - and
a role whose meta/services.yml pins tor.enabled to a literal false is skipped
outright, because that declaration already says the role has no onion and needs
no comment repeating it.
That found 137 variants. 101 were variant 1. Of the rest, 28 are the LDAP-only
auth-isolation rounds, which gain real coverage from running LDAP over the onion
and are flipped on here. Four cannot: svc-net-tor is the provider itself, and
nextcloud's variant 8 pairs the clearnet-only PeerTube. Both carry a marker
naming the reason; opentalk and coturn need none, since their services.yml pins
the literal false the rule now reads.
Flipping them exposed a contract that contradicted itself. test_auth_isolation
and test_non_baseline_explicit_disables demanded `shared: false` beside
`enabled: false`, while variants-static-override flagged that same line as a
static override - which is why all 133 of them carried a nocheck marker. The
demand only makes sense where `shared` is dynamic: a literal base value cannot
leak through deep-merge, so restating it protects nothing. Both contracts now
ask for it only in that case, and the 133 lines are gone.
The result is a stricter rule with fewer lines: 189 insertions against 193
deletions.
Verified: tests/lint 336 passed / 307 subtests, tests/integration/roles 107
passed / 610 subtests. The rule is checked in both directions - it reported 137
offenders before the resolution and none after.
Not measured: 28 additional onion rounds cost matrix wall-clock, against swarm
jobs already capped at 355 minutes and one that has hit that ceiling before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Open WebUI talked to Ollama directly. It now speaks the OpenAI-compatible surface of svc-ai-litellm with a per-consumer virtual key, so a second backend or an external provider becomes a gateway config change instead of a role change. A lint keeps future roles off the model servers.
The gateway advertised a hardcoded model list (ollama/llama3, ollama/nomic-embed-text) while the matrix variant preloads smollm2:135m, so it offered only models the backend did not have and hid the one it did. The list is derived from the ollama preload SPOT now.
The chat spec followed the rename: it gates on litellm instead of the removed ollama service flag, and picks the first non-embedding model from /api/models rather than pinning a variant-specific name, so it exercises the chain rather than a fixture.
Routing openwebui through the gateway pulls both model backends into the closure of every role that consumes it, which pushed nextcloud variant 2 over the memory budget. The budget stays at 32/64 GB; instead the matrix variants of svc-ai-ollama and svc-ai-lmstudio now declare the size their test model needs (1g/2g) while the base keeps the 6g/8g that llama3 and mistral require. The budget guard only saw half of that: it replaced the evaluated role with its variant but counted every dependency from meta/services.yml, a combination no matrix round deploys. It now evaluates every variant-carrying role at the same round index, falling back to variant 0, the way the deploy planner does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ruff toolchain was unpinned, so host venvs, the lint container and CI
installed different releases over time. That drift is why lint-python only
failed on CI: locally ruff 0.15.x preview-gated ISC004/FURB192 and (falsely)
fired S310 on urllib.request.Request, while CI's 0.16.0 has ISC004/FURB192
stable and drops the S310-on-Request false positive.
Pin ruff==0.16.0 in pyproject.toml's dev extras (Dependabot keeps it current)
and have utils.install.lint.ruff read that pin and reinstall whenever the
found version drifts, so every environment resolves the same ruff. Then fix
what 0.16.0 flags: parenthesise the implicit string concatenations (ISC004),
use min() over sorted()[0] (FURB192), drop the now-unused S310 noqa on the
Request() builders (RUF100), and delete the narration comments that the
comment lint surfaces in the touched files.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- matrix 02_render_inventory: the mdad-state volume does not exist yet on a
fresh install, so `container volume inspect` errored the reconfigure task
(`no such volume`) on the row-1 baseline. Guard the marker drop behind the
inspect succeeding; the runner bounce still runs if the container is up.
- test_coverage: read services.yml via utils.cache.files.read_text so the
no-uncached-file-reads lint passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- _shared.js: wrap the boot-poll delay in resolveTimeout (raw-timeout lint)
- 02_render_inventory: docker -> container wrapper (engine-abstraction lint)
- seaweedfs: api becomes the primary canonical so the deploy readiness
probe hits a served vhost when the frontend (filer/master) is disabled
- seaweedfs frontend: exempt from variant pinning — the onion gate is a
hard invariant, no variant may force the frontend on a tor node
- test_coverage: add the `# nocheck: variants-coverage` exempt (same
mechanism as bond-baseline) for such invariant flags
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The run-once guard lint required the when to be exactly the guard string, rejecting roles that legitimately combine it with further conditions (svc-bkp-container-2-local gates its whole core on IS_STACK_HOST). A when-list containing the guard now passes. The variants coverage test learns the same block-above '# nocheck: dynamic-flag' key suppression the services dynamic-flags test already honors, so infrastructure consumer flags can opt out of two-polarity variant coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The resource right-sizing and variant rebalancing bring every variant well under the original 37GB/73GB, so lock in the tighter 32GB/64GB host budget.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The bond<1 partner bins (and the coupled v0 baseline) sit just under these
limits; the previous 32/72 cap was the single-all-on-host figure that the
6-variant distribution replaces. Honest accounting (every enabled service still
counts) — no bond-zeroing or per-variant exemption.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
test_order now only pulls bond>=1 (tightly-coupled, same-host) services into a
role's closure; bond<1 partners deploy on their own host and are excluded, so a
later variant may enable them without variant 0 having to list them.
New test_bond_baseline enforces the placement: a bond>=1 dynamic flag must be
pinned true in variant 0 (the host baseline); a bond<1 flag must be pinned true
in at least one variant. Escape: '# nocheck: bond-baseline'.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
test_limit.py hard-capped meta/variants.yml at MAX_VARIANTS (3) with no opt-out.
Allow a role whose extra matrix rounds are justified (e.g. distributing many
bridged partners so each variant stays within the memory budget) to opt out via
a '# nocheck: variant-limit' comment in its meta/variants.yml.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an integration guard that, for every role and every meta/variants.yml
entry, sums the deduplicated resource footprint (own containers plus shared
dependencies, each service counted once) via the shared
utils.roles.applications.services.resources implementation and fails when a
variant exceeds mem_reservation 32 GB or mem_limit 48 GB, to avoid an OOM kill
at deploy time. The failure message recommends disabling services in the
offending variant or moving them to other variants.
Currently surfaces 6 over-budget variants (nextcloud, shopware, openproject,
flowise, gitlab, openwebui v0) that still need their footprint trimmed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`_build_application_defaults` returned variant 0 of every role with a
meta/variants.yml, which then deep-merged with the variant-N override
that build_dev_inventory bakes into the inventory. Any service flag
that variant N did not explicitly disable carried through from
variant 0 into the runtime applications map. The most visible casualty
was svc-bkp-container-2-local variant 1 (postgres + mariadb only),
where gitea + keycloak leaked in via variant 0, sys-service-loader
pulled in svc-db-openldap via keycloak.services.ldap, and the deploy
crashed in sys-svc-compose/04_files.yml on openldap's env.j2 with
"credentials is defined in schema at roles/svc-db-openldap/meta/schema.yml
but not set in application 'svc-db-openldap'".
Defaults now come from each role's variant-free base config
(meta/<topic>.yml plus schema-defaults, no variants overlay). Variants
stay accessible via get_variants for any caller that explicitly needs
a specific variant. Inventory-level applications.<app> overrides win
exactly as before.
Supporting changes that keep the variant matrix consistent under the
new defaults model:
- New lint guard tests/integration/roles/meta/variants/
test_non_baseline_explicit_disables.py: for every dynamic
service-key declared in a role's meta/services.yml that variant 0
pins to literal true, every non-baseline variant MUST explicitly
pin the same key to either literal true or to literal
enabled: false + shared: false. Statically-enabled keys are exempt
(same rule as test_auth_isolation.py).
- roles/web-app-{fusiondirectory,lam,nextcloud,phpldapadmin}/meta/
variants.yml: explicit `false` pins for dashboard / email / logout /
matomo / prometheus / talk in restrictive variants so deep-merge no
longer hands them the deleted variant-0 inheritance.
- roles/web-app-moodle/templates/playwright.env.j2: gate the LAM
block on `lookup('config', application_id, 'services.lam.enabled',
false)`. LAM is a transitive helper, not a moodle-owned service, so
the default-bearing lookup tolerates the missing path when the
variant does not deploy LAM.
- tests/unit/utils/cache/test_variants.py: rename the legacy
"always variant zero" coverage to TestMergedApplicationsUsesBaseConfig
and add test_get_application_defaults_omits_variant_zero_only_keys
as the regression guard for the leak this commit fixes.
- docs/contributing/artefact/files/role/variants.md +
docs/contributing/design/variants.md: variant 0 now manually
enumerates every dynamic services.yml key with
enabled: true + shared: true; `{}` is only permitted when the role
has no dynamic-enabled service-key.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Commit f63118f34 ("feat(lint): enforce auth-isolation in
variants.yml") added `enabled: false, shared: false` overrides for
mariadb / postgres to every auth-isolated LDAP-only variant. That
disables the role's hard-required DB at deploy time, so e.g. the
moodle baseline blows up with `Required env var MOODLE_DB_HOST is
not set` as soon as round 2 of the matrix hits variant 1.
The auth-isolation rule's premise — "auth-isolated rounds must trim
non-auth services from the include closure" — only buys closure
savings for services whose `meta/services.yml` base declares
`enabled` as a dynamic Jinja expression. DB-engine entries are
already a static `enabled: true`, so the resolver pulls them in
regardless of the variant pin; the variant `false` only breaks the
deploy without trimming anything.
Two changes:
- `test_auth_isolation`: exempt service-keys whose `services.yml`
base has a static literal `enabled: true` from the per-variant
pin obligation. Such keys should not be declared in auth-isolated
variants at all; the inherited base value is correct.
- 10 web-app roles in the active deploy scope (akaunting, baserow,
bookwyrm, espocrm, friendica, gitea, moodle, nextcloud, odoo,
openproject): drop the entire mariadb/postgres stanza from the
LDAP-only variant — both `enabled` and `shared` inherit from
`meta/services.yml`. Matches the pre-f63118f34 state.
- 6 more roles out of the current iteration scope (funkwhale,
opentalk, shopware, snipe-it, suitecrm, taiga) carry the same
broken `enabled: false` line: removed it, kept the pre-existing
`shared: false` to preserve the variant's own-DB-container
topology intent.
`make test` is green; `variants-auth-isolation` and
`variants-static-override` both pass with no per-variant nochecks
required.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Auth-isolated matrix variants (exactly one of LDAP / OIDC-OAuth2
enabled, and no non-auth `enabled: true` alongside) must explicitly
pin every non-auth service-key in the role's variant union to literal
`enabled: false, shared: false`. Without this, the variant's
auth-only test scope balloons into a full integration deploy via the
services.yml-default propagation, wasting CI minutes and obscuring
the regression signal.
Adds `tests/integration/roles/meta/variants/test_auth_isolation.py`
(rule key `variants-auth-isolation`) plus the matching catalog entry.
The trigger is narrow on purpose: a variant that enables auth
alongside other services is a deliberate integration test, not an
auth-isolated one, and the rule does not fire there.
Mass-fixes the 24 legacy auth-isolated variants the rule turned up:
each gets the missing `enabled: false, shared: false` declarations
appended to its `services:` block. Where the false-override clashes
with a static-literal base in meta/services.yml, the override line
also picks up a `# nocheck: variants-static-override` marker. The
svc-bkp-container-2-local variant 2 additionally carries
`# nocheck: variants-services-match` on the new keys because they
are resolver-only matrix hooks with no consumer contract in the base
meta.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The variants → services match guard (`test_services_match.py`) used to
have no exemption mechanism, forcing every variant service-key to also
appear as a top-level entry in `meta/services.yml`. That contract is
correct for application roles whose variants pin per-app service
flags, but it forces noise on invokable infrastructure roles whose
variants exist only to pin the round's companion topology — there the
service-key is a resolver hook, not a consumer-side contract, and
adding a stub in `meta/services.yml` just duplicates the variant
declaration.
Add a `# nocheck: variants-services-match` opt-out (same line or line
above the key) so legitimate resolver-only matrix hooks can stay in
variants.yml without a synthetic services.yml stub. Mirrors the
`variants-static-override` parser pattern (text walk + per-leaf line
indices + is_suppressed_at).
Adopts the new exemption on `svc-bkp-container-2-local`:
meta/services.yml drops the gitea / keycloak / postgres / mariadb /
ldap stubs, meta/variants.yml carries the nocheck marker on each
variant service-key entry. The resolver closure per round stays
identical (16 / 3 / 2 roles).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Removes 67 live references to docs/requirements/<NNN>-*.md files across
43 contributor docs, role READMEs, code comments, task files, and tests.
Requirement docs are initial-creation-only artefacts; live references
freeze a snapshot that rots once the per-feature SPOT (a contributor
doc, a code module, a lint test) takes over the live behaviour.
In the same pass, fixes documentation.md violations on the touched
files: drops em-dashes used as thought breaks, removes "SPOT for ..."
self-labels, strips historical phrasing ("the legacy", "previously",
"used to", "is no longer used", "matching the historical"), and
reflows soft-wrapped sentences so each sentence stays on one logical
line.
The `test_inventory_vars_file_spot.py` whitelist for the literal
inventory path drops the docs/requirements/003 entry and instead
skips docs/requirements/ entirely; future requirement docs that
mention the literal path are then ignored automatically.
Verified by tests/lint/repository/test_no_stale_requirement_refs.py
(green tree-wide).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
TC (flake8-type-checking): typing-only imports moved into
``if TYPE_CHECKING:`` blocks across 8 source files via auto-fix
(117 hits). Reduces runtime import surface and rules out circular
imports for type-only deps. Project-wide ``from __future__ import
annotations`` (FA, already enabled) keeps annotations as strings,
so the moved imports never need to load at runtime.
N (pep8-naming): 32 hits handled.
Renames (semantic + lint):
- ``isSmallerThenMaximumBackupSize`` -> ``is_larger_than_maximum_backup_size``
(snake_case + fixes both the typo "Then -> Than" and the
semantic mismatch: the function returns ``current > maximum``)
- ``isDirectoryDeletable`` / ``deleteVersion`` /
``getAmountOfIteration`` / ``deleteIteration`` -> snake_case
- 8 test functions with ALL-CAPS env-var fragments
(``test_..._DOMAIN_PRIMARY_...`` etc.) -> snake_case
- Mock alias ``VH`` -> ``vh`` in test_host_vars.py
Exception class renames (``N818`` Error suffix):
- ``TimeLimitExceededException`` -> ``TimeLimitExceededError``
- ``AttemptException`` -> ``AttemptError``
- ``PathNotFound`` -> ``PathNotFoundError`` (propagated across
4 callers under tests/integration/lookups/config/)
- ``_ExitJson`` / ``_FailJson`` -> ``*_Error``
Refactor:
- ``METADATA = {...}`` local constant in two functions of
``plugins/filter/invokable_paths.py`` lifted to module-level
``_METADATA_KEYS``.
- ``from typing import TYPE_CHECKING`` in
``tasks/utils/migrate_meta_layout/__main__.py`` moved above
the ``sys.path.insert`` hack to satisfy E402 (collateral from
the TC auto-fix that landed it after the path mutation).
Per-file ignores in pyproject.toml for the legitimate cases the
rule cannot accept:
- ``pull-specific-host.py`` -> ``N999`` (Ansible-invoked role-level
script, hyphenated filename pinned by the surrounding stack).
- 4 matrix-deploy files -> ``N806`` for the ``include_R`` /
``raw_include_R`` round-index notation; the uppercase ``R``
here mirrors the formula in the docstring rather than naming
a constant.
- ``plugins/filter/value_generator.py`` -> ``N806`` for the
lazy-loaded class binding ``ValueGenerator =
_value_generator_cls()``, which retains the imported class
identifier per PEP 8.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- New test_limit.py: each role's meta/variants.yml MUST declare at
most 3 entries. Matches the matrix-deploy footprint cap and keeps
the cross-round purge surface tractable.
- Prune web-app-opencloud's variants.yml to 3 entries: variant 1
was a buggy duplicate of variant 2 (its own comment described the
LDAP-only variant 3, but the body was an all-false copy). Lifts
the LDAP-only comment up to where it belongs.
- Relax test_order.py from "all variants share the same closure" to
"non-baseline variant closures MUST be subsets of variant 0's
closure". The strict-equality version was logically incompatible
with test_coverage.py (which forces variants to flip dynamic flags
on both polarities -- closures necessarily diverge). Subset still
catches the real bug -- a non-baseline variant introducing a dep
variant 0 lacks -- without forbidding the legit "minimal variant
shrinks the closure" pattern. Empty closures pass trivially.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Move tests/lint/applications/test_variant_dep_consistency.py
-> tests/integration/roles/meta/variants/test_order.py
- Drop now-empty tests/lint/applications/ package.
- Replace try/except/pass with contextlib.suppress (ruff SIM105).
- Treat variants whose closure is empty as drop-outs from the
dep ordering and exclude them from the divergence comparison;
only non-empty closures must agree. Removes 5 false-positives
(matomo, socialhome, web-svc-cdn, web-svc-collabora,
web-svc-simpleicons) whose v1 intentionally disables every
dep-pulling service.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_role_file_spot.py: AST-walks every .py
file under the production trees and tests/, flags any reference to a
role file (standalone literal, f-string segment, pathlib `/` chain,
`+` concat chain, os.path.join / Path.joinpath / posixpath.join /
ntpath.join call) whose value or joined trailing operands match a
ROLE_FILE_* constant in utils.roles.mapping. The flag set is
auto-discovered from the module so adding a new constant extends
coverage without further edits. ROLE_FILE_README is exempt as too
generic. Per-line `# nocheck: role-file-spot` (same-or-above) is the
documented opt-out, catalog row added in
docs/contributing/actions/testing/suppression.md.
Migrate 139 .py files (377 occurrences) to use the constants:
- 135 files via direct constant import (`from utils.roles.mapping
import ROLE_FILE_*`), with the symbol used in pathlib chains, in
os.path.join / Path.joinpath calls, in f-string interpolation, and
in glob patterns.
- 4 files via `# nocheck: role-file-spot` where the literal lives in
argparse help text or in an allowlist tuple meant to declare the
literal path.
Adjust utils/roles/dependency_resolver.py to use lazy-formatting
positional args in the logger.exception call (G003-clean) instead of
string concatenation against the constant.
Drop a redundant `if cond: return True; return False` guard in the
new test for `return cond` (SIM103-clean).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move utils/roles/applications/files.py to utils/roles/mapping.py and add a
data-driven role-type detector in utils/roles/type.py. Each entry in
ROLE_FILES carries per-role-type allowed/mandatory flags plus marker entries
that drive get_role_types(); a role MAY belong to several types (e.g. an
application that also ships a system service).
Add cli/meta/roles/mapping with files / types / markers subcommands so
contributors can query the schema in text or JSON. Move
cli/meta/applications/ to cli/meta/roles/applications/ and adjust PROJECT_ROOT
parents indices accordingly. Move docs/contributing/design/services/ to
docs/contributing/design/role/services/ and document the mapping in
docs/contributing/design/role/mapping.md per documentation.md.
Replace the narrow test_app_only_files with a generalist
tests/integration/roles/meta/test_mapping.py that walks every role and every
ROLE_FILES entry and reports forbidden files, missing mandatory files,
forbidden entries, and missing mandatory entries in one structured fail
report. meta/info.yml and meta/users.yml are now allowed across every role
type. Create the seven mandatory files surfaced by the generalist test
(README.md for desk-keepassxc, desk-obs, web-svc-cdn following readme_md.md,
plus tasks/main.yml stubs for svc-db-memcached, svc-db-redis, sys-svc-proxy,
web-app-socialhome).
Audit and rewrite touched markdown for documentation.md compliance: replace
em-dashes with fluent sentences, drop SPOT self-labels, fix all broken
relative links introduced by the services/ move.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Some artefacts under ``roles/<role>/meta/`` only carry meaning when the
role declares an ``application_id`` in ``vars/main.yml`` and is therefore
deployable as a primary app via ``--apps``. The matrix-deploy planner,
the schema/credentials machinery, and the application-specific
server / rbac / volumes blocks all key off that flag, so dropping these
files on a non-app role is dead weight that drifts away from reality
the next time the surrounding mechanism evolves.
- ``utils/roles/applications/files.py`` is the new SPOT: it exports
``APPLICATION_ONLY_META_FILES`` (mapping filename → why it's
app-only) and ``is_application_role(role_dir)`` so the rest of the
test suite can share one boundary instead of re-hardcoding role-name
prefixes.
- ``tests/integration/roles/meta/test_app_only_files.py`` walks every
role directory and fails when a non-app role carries any of the
app-only files. Each failure carries the human description from
the SPOT so the contributor sees why the file is restricted.
- ``tests/integration/roles/meta/variants/test_coverage.py`` now
consults ``is_application_role`` instead of matching role-name
prefixes; that lines its scope up with the matrix-deploy planner
exactly (``desk-nextcloud`` is in scope because it carries
``application_id``, the ``sys-`` family is out because it doesn't).
``meta/services.yml`` is intentionally NOT on the SPOT list: many
non-app roles legitimately use it for ``lifecycle`` and ``run_after``
declarations per req-010.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends ``[tool.ruff.lint].select`` with six additional rule families:
``RUF`` (ruff-specific), ``SIM`` (flake8-simplify), ``C4``
(flake8-comprehensions), ``PERF`` (perflint), ``RET`` (flake8-return),
``PIE`` (flake8-pie). Adds ``RUF001/002/003`` to ``ignore`` because
those exist for homoglyph-attack detection, not English typography —
the project uses em/en-dashes and ``ℹ`` as a UI-hint glyph in CLI help
output deliberately.
Auto-fixed (~225, mechanical & behavior-preserving)
====================================================
- RUF005: ``a + [b]`` → ``[*a, b]``
- C408: ``dict(k=v)`` → ``{"k": v}``
- C420: ``dict.fromkeys(seq)`` shape simplifications
- SIM103/SIM110/SIM118: truthy-return / dict-key / dict-method
idiom simplifications
- RET504/RET505: drop unneeded final assigns / ``else`` after
``return``
- PIE810: combine ``str.startswith(a) or .startswith(b)``
into one call with a tuple
- RUF010: ``f"{str(x)}"`` → ``f"{x\!s}"``
- RUF100: removed unused ``# noqa`` directives that
became stale (mostly old ``# noqa: E402``
markers that ruff's smart E402 implementation
no longer flags after ``sys.path`` shims)
- RUF015 / RUF013 / RUF022 / RUF046 / RUF059: misc small wins
- I001: re-sorted import blocks where the new
imports above shifted them
Hand-fixed (~140, scripted where mechanical)
============================================
PERF401 (42): scripted AST rewrite of
``for x in iter: target.append(expr)`` (with optional ``if`` guard)
→ ``target.extend(expr for x in iter [if cond])``.
``list.extend(genexp)`` consumes the generator immediately, so the
rewrite is semantically equivalent to the original eager loop.
SIM117 (41): scripted AST combiner for nested ``with`` blocks where
the inner is the only statement of the outer (38 sites). The remaining
3 multi-line ``patch.object(...)`` chains were combined by hand using
the parenthesised ``with (\\n A,\\n B,\\n):`` form.
SIM102 (14): scripted AST combiner for nested ``if`` (10 sites). The
remaining 4 carry an inline ``# noqa: SIM102`` with rationale —
combining would either change semantics (the outer body has siblings
beyond the inner ``if``, e.g. ``cli/encrypt/inventory/__main__.py``
where ``encrypted_value = …`` runs regardless of ``ask_confirmation``)
or hurt readability (long literal whitelists).
RUF012 (9): annotated mutable class attributes with ``ClassVar[…]``
and added ``from typing import ClassVar``.
SIM105 (6): ``try/except: pass`` → ``with contextlib.suppress(…):``
with the necessary ``import contextlib`` added. Verified on
``utils/templating.py`` (3 occurrences in a ``finally`` block) —
cleanup behaviour is identical.
SIM115 (4): wrapped ``tempfile.NamedTemporaryFile(..., delete=False)``
call patterns in a ``with`` block; ``delete=False`` keeps the file
on disk after the block, so callers' ``return f.name`` still works.
PERF402, PERF403, B007, SIM108, SIM103: 5 misc fixes (loop→list-copy,
loop→dict-comprehension, unused-loop-var rename, ternary, return
expression).
Verification
============
- ``ruff check .`` → All checks passed
- ``ruff format --check .`` → 894 files already formatted
- 1621 unit tests → all pass
- Integration suite: 44 baseline failures, 1 skipped (unchanged)
- Sample-audited PERF401 rewrites in ``resolver.py``, ``inventory.py``,
``cache/files.py``, ``ressources/__main__.py``; SIM117 in
``test_main.py``; SIM105 in ``utils/templating.py``; SIM102 noqa
rationales in ``cli/encrypt/inventory/__main__.py``.
182 files changed, +993 / −960. With this and the prior strict-rule
commit the branch satisfies 11 ruff rule families end-to-end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds ``[tool.ruff.lint]`` with ``select = ["E", "F", "I", "B", "UP"]``
and ``target-version = "py311"``. ``E501`` is ignored because
``ruff format`` already reflows what it can; the remaining long lines
are unsplittable string literals / URLs.
Auto-fixed (2753 issues, behavior-preserving):
- UP006/UP035: ``List[X]`` → ``list[X]``, ``Dict[X]`` → ``dict[X]``,
``from typing import …`` redirected where superseded by builtins.
- UP045: ``Optional[X]`` → ``X | None``.
- UP010/UP009/UP015: dropped Python-2-era pragmas (``from __future__
import absolute_import, division, print_function``,
``# -*- coding: utf-8 -*-``), redundant ``open(..., "r")`` mode.
- UP004: dropped implicit ``class X(object):``.
- UP022/UP008/UP032: stdlib idiom modernization (``capture_output``,
super-call simplification, etc.).
- B905: ``zip(...)`` → ``zip(..., strict=False)`` (preserves prior
behavior; switching to ``strict=True`` is a separate semantic
choice not made here).
- I001: import-block ordering.
Hand-fixed (66 issues):
- B904 × 54: appended ``from <bound>`` (when the enclosing ``except``
binds a name) or ``from None`` (when it doesn't) so chained
exceptions stay distinguishable from errors-during-exception-
handling. Applied via a small AST-aware script that walked back to
the enclosing ``except`` clause and matched the binding.
- B017 × 4: narrowed broad ``assertRaises(Exception)`` to the actual
``AnsibleError`` / ``AnsibleFilterError`` raised by the code under
test.
- B023 × 2: bound captured loop variables via default args
(``def _record(name, *, _path_real=path_real, ...)``) so each
closure gets its own snapshot.
- B007 × 2: unused loop control variables renamed to ``_`` /
rewritten to ``.values()``.
- UP007 × 2: ``Union[X, Y]`` → ``X | Y`` (plus dropping the now-unused
``from typing import Union``).
- B018 × 1: Odoo addon manifests are bare dict expressions at module
level by design — suppressed with an inline ``# noqa: B018`` and
rationale comment.
- UP031 × 1: printf-style format → f-string.
The 65 ``# noqa: …`` markers introduced by the auto-fixer all carry
real ruff codes (``B018``, ``E402``, ``F401``, etc.), so
``test_noqa_only_ruff_codes`` stays green.
Verification
============
- ``ruff check .`` → All checks passed
- ``ruff format --check .`` → 894 files already formatted
- ``tests.lint.repository.{test_project_root_import,
test_no_direct_yaml_calls, test_noqa_only_ruff_codes}`` and
``tests.unit.utils.annotations.test_suppress`` → 20 tests OK
- 1621 unit tests → all pass
- Integration suite: 44 failures (pre-existing baseline, none from
this change), 1 skipped.
Going forward, the new rules block regressions on E (style), F
(pyflakes), I (import order), B (likely-bug patterns), and UP
(modernization to py311) for any new code.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reason: ruff parses every `# noqa: <code>` marker as a flake8 directive
and warns on every non-flake8 rule key. Every catalog rule in
`docs/contributing/actions/testing/suppression.md` is project-specific
(kebab-case), so `# noqa:` markers fired the warning on every parse.
`# nocheck:` is the project synonym ruff ignores.
Scope:
- 41 `roles/*/meta/services.yml` + 1 task + 1 .cfg.j2 template: pure
noqa→nocheck swap for catalog rules. Mixed flake8 markers
(`# noqa: E402, X`) preserved verbatim.
- 5 production .py + 2 production scripts: same swap.
- All catalog-rule lints' docstrings updated; redundant "or # nocheck:"
parentheticals collapsed to a single nocheck form.
- `utils/annotations/suppress.py`: docstring rewritten to spell out
the new convention. Parser still accepts both keywords as synonyms.
- `docs/contributing/actions/testing/suppression.md`: convention
rewritten to mandate `nocheck:`; missing `domain-spot` and
`literal-protocol-lookup` rows added; "enforced by" pointer added.
New guard:
- `tests/lint/repository/test_noqa_only_ruff_codes.py`: meta-lint
failing any `# noqa: <code>` whose codes don't match `[A-Z]+\d+`
(real ruff/flake8 shape). Allowlists the impl, its unit test, the
catalog page, the direct-yaml lint, and itself.
- `tests/lint/repository/test_no_direct_yaml_calls.py`: hardened to
reject the `noqa:` form via inlined regex (no longer routed through
`utils.annotations.suppress`).
Tangential cleanups carried in this same commit:
- `utils/cache/yaml.py`, `utils/handler/yaml.py`: 6 `direct-yaml`
markers swapped.
- `utils/cache/base.py`: `from . import PROJECT_ROOT, ROLES_DIR` moved
up to the import block (E402).
- `cli/create/role/__init__.py`: reordered so PROJECT_ROOT is defined
before `__main__` is imported (also fixed a latent circular import).
- PROJECT_ROOT migration tail: 2 new `__init__.py` modules + 10 test
files swapped from `def repo_root()` to `from . import PROJECT_ROOT`.
- ~20 ruff auto-fix removals of unused `from pathlib import Path`.
Verification:
- `ruff check .` clean.
- 23 suppression-related tests pass.
- The new lint detects all three regression shapes (project rule alone,
mixed with real ruff code, lowercase-only).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Migrate the simple `PROJECT_ROOT = Path(__file__).resolve().parents[N]`
shape across the repo to the new "import from `__init__.py`" rule
introduced by the `project-root-import` lint:
- `utils/cache/__init__.py` is the canonical source. `PROJECT_ROOT`
and `ROLES_DIR` move out of `utils/cache/base.py`; `base.py`
re-exports them for callers that still import from there.
Internal cache modules (`files.py`, `gitignore.py`) switch to
`from . import PROJECT_ROOT`.
- 24 packages receive a `PROJECT_ROOT` declaration in their own
`__init__.py`, each with the parents[N] depth that resolves to
the repo root.
- ~36 consumer files swap their local
`PROJECT_ROOT = Path(__file__).resolve().parents[N]` for
`from . import PROJECT_ROOT`. `ROLES_DIR` and similar derived
paths stay local (`PROJECT_ROOT / "roles"`).
Status: this commit closes the simple-shape violations. The
multi-line `os.path.abspath(os.path.join(here, os.pardir, …))`
shape, `def repo_root()` while-loops, and `roles/<role>/filter_plugins`
files without a package container remain. The lint stays scharf and
will track each subsequent wave.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two intertwined changes that share the same touched files:
1. Move every meta-related integration test under
`tests/integration/roles/meta/` into one of three sibling
sub-clusters and shorten the redundant filename prefixes:
- `services/` — `test_services_dynamic_flags.py` →
`test_dynamic_flags.py`, `test_services_no_self_reference.py` →
`test_no_self_reference.py`. The `run_after` cluster moves
under it (it is a `meta/services.yml` concern, not a
stand-alone dependency one): `test_services_explicit.py` keeps
its name, the rest drop the redundant prefix.
- `variants/` — `test_variants_coverage.py` → `test_coverage.py`,
plus the two new lints `test_auth_coverage.py` (LDAP / OIDC /
OAuth2 matrix completeness) and `test_services_match.py`
(every variant override key MUST exist in `services.yml`).
- `dependencies/` — relocates the legacy
`tests/integration/roles/dependencies/` cluster so all
`meta/`-shaped tests live under one parent.
Path-index constants (`Path(__file__).resolve().parents[N]`,
`os.pardir` chains) updated for the deeper nesting.
2. Route every yaml read in the migrated tests through
`utils.cache.yaml`. The single-doc helper `load_yaml_any` covers
most of them; the new `load_yaml_all` mirrors `load_yaml_any` for
the multi-doc case (`yaml.safe_load_all`) and is keyed off the
same `(path, mtime, size)` signature, with parallel `_drop_path`
/ `_reset` housekeeping. `tests/lint/ansible/variables/
test_vars_usage_in_yaml.py` is the first user.
READMEs cleaned up to match the new directory structure (path
crosslinks, level-up counts) and four new READMEs added for the
new sub-clusters under the `docs/` README guidelines (RFC 2119,
emojis after headlines, no em dashes, no SPOT self-labels).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
