- matrix 02_render_inventory: the mdad-state volume does not exist yet on a
fresh install, so `container volume inspect` errored the reconfigure task
(`no such volume`) on the row-1 baseline. Guard the marker drop behind the
inspect succeeding; the runner bounce still runs if the container is up.
- test_coverage: read services.yml via utils.cache.files.read_text so the
no-uncached-file-reads lint passes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- _shared.js: wrap the boot-poll delay in resolveTimeout (raw-timeout lint)
- 02_render_inventory: docker -> container wrapper (engine-abstraction lint)
- seaweedfs: api becomes the primary canonical so the deploy readiness
probe hits a served vhost when the frontend (filer/master) is disabled
- seaweedfs frontend: exempt from variant pinning — the onion gate is a
hard invariant, no variant may force the frontend on a tor node
- test_coverage: add the `# nocheck: variants-coverage` exempt (same
mechanism as bond-baseline) for such invariant flags
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The run-once guard lint required the when to be exactly the guard string, rejecting roles that legitimately combine it with further conditions (svc-bkp-container-2-local gates its whole core on IS_STACK_HOST). A when-list containing the guard now passes. The variants coverage test learns the same block-above '# nocheck: dynamic-flag' key suppression the services dynamic-flags test already honors, so infrastructure consumer flags can opt out of two-polarity variant coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TC (flake8-type-checking): typing-only imports moved into
``if TYPE_CHECKING:`` blocks across 8 source files via auto-fix
(117 hits). Reduces runtime import surface and rules out circular
imports for type-only deps. Project-wide ``from __future__ import
annotations`` (FA, already enabled) keeps annotations as strings,
so the moved imports never need to load at runtime.
N (pep8-naming): 32 hits handled.
Renames (semantic + lint):
- ``isSmallerThenMaximumBackupSize`` -> ``is_larger_than_maximum_backup_size``
(snake_case + fixes both the typo "Then -> Than" and the
semantic mismatch: the function returns ``current > maximum``)
- ``isDirectoryDeletable`` / ``deleteVersion`` /
``getAmountOfIteration`` / ``deleteIteration`` -> snake_case
- 8 test functions with ALL-CAPS env-var fragments
(``test_..._DOMAIN_PRIMARY_...`` etc.) -> snake_case
- Mock alias ``VH`` -> ``vh`` in test_host_vars.py
Exception class renames (``N818`` Error suffix):
- ``TimeLimitExceededException`` -> ``TimeLimitExceededError``
- ``AttemptException`` -> ``AttemptError``
- ``PathNotFound`` -> ``PathNotFoundError`` (propagated across
4 callers under tests/integration/lookups/config/)
- ``_ExitJson`` / ``_FailJson`` -> ``*_Error``
Refactor:
- ``METADATA = {...}`` local constant in two functions of
``plugins/filter/invokable_paths.py`` lifted to module-level
``_METADATA_KEYS``.
- ``from typing import TYPE_CHECKING`` in
``tasks/utils/migrate_meta_layout/__main__.py`` moved above
the ``sys.path.insert`` hack to satisfy E402 (collateral from
the TC auto-fix that landed it after the path mutation).
Per-file ignores in pyproject.toml for the legitimate cases the
rule cannot accept:
- ``pull-specific-host.py`` -> ``N999`` (Ansible-invoked role-level
script, hyphenated filename pinned by the surrounding stack).
- 4 matrix-deploy files -> ``N806`` for the ``include_R`` /
``raw_include_R`` round-index notation; the uppercase ``R``
here mirrors the formula in the docstring rather than naming
a constant.
- ``plugins/filter/value_generator.py`` -> ``N806`` for the
lazy-loaded class binding ``ValueGenerator =
_value_generator_cls()``, which retains the imported class
identifier per PEP 8.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_role_file_spot.py: AST-walks every .py
file under the production trees and tests/, flags any reference to a
role file (standalone literal, f-string segment, pathlib `/` chain,
`+` concat chain, os.path.join / Path.joinpath / posixpath.join /
ntpath.join call) whose value or joined trailing operands match a
ROLE_FILE_* constant in utils.roles.mapping. The flag set is
auto-discovered from the module so adding a new constant extends
coverage without further edits. ROLE_FILE_README is exempt as too
generic. Per-line `# nocheck: role-file-spot` (same-or-above) is the
documented opt-out, catalog row added in
docs/contributing/actions/testing/suppression.md.
Migrate 139 .py files (377 occurrences) to use the constants:
- 135 files via direct constant import (`from utils.roles.mapping
import ROLE_FILE_*`), with the symbol used in pathlib chains, in
os.path.join / Path.joinpath calls, in f-string interpolation, and
in glob patterns.
- 4 files via `# nocheck: role-file-spot` where the literal lives in
argparse help text or in an allowlist tuple meant to declare the
literal path.
Adjust utils/roles/dependency_resolver.py to use lazy-formatting
positional args in the logger.exception call (G003-clean) instead of
string concatenation against the constant.
Drop a redundant `if cond: return True; return False` guard in the
new test for `return cond` (SIM103-clean).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move utils/roles/applications/files.py to utils/roles/mapping.py and add a
data-driven role-type detector in utils/roles/type.py. Each entry in
ROLE_FILES carries per-role-type allowed/mandatory flags plus marker entries
that drive get_role_types(); a role MAY belong to several types (e.g. an
application that also ships a system service).
Add cli/meta/roles/mapping with files / types / markers subcommands so
contributors can query the schema in text or JSON. Move
cli/meta/applications/ to cli/meta/roles/applications/ and adjust PROJECT_ROOT
parents indices accordingly. Move docs/contributing/design/services/ to
docs/contributing/design/role/services/ and document the mapping in
docs/contributing/design/role/mapping.md per documentation.md.
Replace the narrow test_app_only_files with a generalist
tests/integration/roles/meta/test_mapping.py that walks every role and every
ROLE_FILES entry and reports forbidden files, missing mandatory files,
forbidden entries, and missing mandatory entries in one structured fail
report. meta/info.yml and meta/users.yml are now allowed across every role
type. Create the seven mandatory files surfaced by the generalist test
(README.md for desk-keepassxc, desk-obs, web-svc-cdn following readme_md.md,
plus tasks/main.yml stubs for svc-db-memcached, svc-db-redis, sys-svc-proxy,
web-app-socialhome).
Audit and rewrite touched markdown for documentation.md compliance: replace
em-dashes with fluent sentences, drop SPOT self-labels, fix all broken
relative links introduced by the services/ move.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Some artefacts under ``roles/<role>/meta/`` only carry meaning when the
role declares an ``application_id`` in ``vars/main.yml`` and is therefore
deployable as a primary app via ``--apps``. The matrix-deploy planner,
the schema/credentials machinery, and the application-specific
server / rbac / volumes blocks all key off that flag, so dropping these
files on a non-app role is dead weight that drifts away from reality
the next time the surrounding mechanism evolves.
- ``utils/roles/applications/files.py`` is the new SPOT: it exports
``APPLICATION_ONLY_META_FILES`` (mapping filename → why it's
app-only) and ``is_application_role(role_dir)`` so the rest of the
test suite can share one boundary instead of re-hardcoding role-name
prefixes.
- ``tests/integration/roles/meta/test_app_only_files.py`` walks every
role directory and fails when a non-app role carries any of the
app-only files. Each failure carries the human description from
the SPOT so the contributor sees why the file is restricted.
- ``tests/integration/roles/meta/variants/test_coverage.py`` now
consults ``is_application_role`` instead of matching role-name
prefixes; that lines its scope up with the matrix-deploy planner
exactly (``desk-nextcloud`` is in scope because it carries
``application_id``, the ``sys-`` family is out because it doesn't).
``meta/services.yml`` is intentionally NOT on the SPOT list: many
non-app roles legitimately use it for ``lifecycle`` and ``run_after``
declarations per req-010.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends ``[tool.ruff.lint].select`` with six additional rule families:
``RUF`` (ruff-specific), ``SIM`` (flake8-simplify), ``C4``
(flake8-comprehensions), ``PERF`` (perflint), ``RET`` (flake8-return),
``PIE`` (flake8-pie). Adds ``RUF001/002/003`` to ``ignore`` because
those exist for homoglyph-attack detection, not English typography —
the project uses em/en-dashes and ``ℹ`` as a UI-hint glyph in CLI help
output deliberately.
Auto-fixed (~225, mechanical & behavior-preserving)
====================================================
- RUF005: ``a + [b]`` → ``[*a, b]``
- C408: ``dict(k=v)`` → ``{"k": v}``
- C420: ``dict.fromkeys(seq)`` shape simplifications
- SIM103/SIM110/SIM118: truthy-return / dict-key / dict-method
idiom simplifications
- RET504/RET505: drop unneeded final assigns / ``else`` after
``return``
- PIE810: combine ``str.startswith(a) or .startswith(b)``
into one call with a tuple
- RUF010: ``f"{str(x)}"`` → ``f"{x\!s}"``
- RUF100: removed unused ``# noqa`` directives that
became stale (mostly old ``# noqa: E402``
markers that ruff's smart E402 implementation
no longer flags after ``sys.path`` shims)
- RUF015 / RUF013 / RUF022 / RUF046 / RUF059: misc small wins
- I001: re-sorted import blocks where the new
imports above shifted them
Hand-fixed (~140, scripted where mechanical)
============================================
PERF401 (42): scripted AST rewrite of
``for x in iter: target.append(expr)`` (with optional ``if`` guard)
→ ``target.extend(expr for x in iter [if cond])``.
``list.extend(genexp)`` consumes the generator immediately, so the
rewrite is semantically equivalent to the original eager loop.
SIM117 (41): scripted AST combiner for nested ``with`` blocks where
the inner is the only statement of the outer (38 sites). The remaining
3 multi-line ``patch.object(...)`` chains were combined by hand using
the parenthesised ``with (\\n A,\\n B,\\n):`` form.
SIM102 (14): scripted AST combiner for nested ``if`` (10 sites). The
remaining 4 carry an inline ``# noqa: SIM102`` with rationale —
combining would either change semantics (the outer body has siblings
beyond the inner ``if``, e.g. ``cli/encrypt/inventory/__main__.py``
where ``encrypted_value = …`` runs regardless of ``ask_confirmation``)
or hurt readability (long literal whitelists).
RUF012 (9): annotated mutable class attributes with ``ClassVar[…]``
and added ``from typing import ClassVar``.
SIM105 (6): ``try/except: pass`` → ``with contextlib.suppress(…):``
with the necessary ``import contextlib`` added. Verified on
``utils/templating.py`` (3 occurrences in a ``finally`` block) —
cleanup behaviour is identical.
SIM115 (4): wrapped ``tempfile.NamedTemporaryFile(..., delete=False)``
call patterns in a ``with`` block; ``delete=False`` keeps the file
on disk after the block, so callers' ``return f.name`` still works.
PERF402, PERF403, B007, SIM108, SIM103: 5 misc fixes (loop→list-copy,
loop→dict-comprehension, unused-loop-var rename, ternary, return
expression).
Verification
============
- ``ruff check .`` → All checks passed
- ``ruff format --check .`` → 894 files already formatted
- 1621 unit tests → all pass
- Integration suite: 44 baseline failures, 1 skipped (unchanged)
- Sample-audited PERF401 rewrites in ``resolver.py``, ``inventory.py``,
``cache/files.py``, ``ressources/__main__.py``; SIM117 in
``test_main.py``; SIM105 in ``utils/templating.py``; SIM102 noqa
rationales in ``cli/encrypt/inventory/__main__.py``.
182 files changed, +993 / −960. With this and the prior strict-rule
commit the branch satisfies 11 ruff rule families end-to-end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds ``[tool.ruff.lint]`` with ``select = ["E", "F", "I", "B", "UP"]``
and ``target-version = "py311"``. ``E501`` is ignored because
``ruff format`` already reflows what it can; the remaining long lines
are unsplittable string literals / URLs.
Auto-fixed (2753 issues, behavior-preserving):
- UP006/UP035: ``List[X]`` → ``list[X]``, ``Dict[X]`` → ``dict[X]``,
``from typing import …`` redirected where superseded by builtins.
- UP045: ``Optional[X]`` → ``X | None``.
- UP010/UP009/UP015: dropped Python-2-era pragmas (``from __future__
import absolute_import, division, print_function``,
``# -*- coding: utf-8 -*-``), redundant ``open(..., "r")`` mode.
- UP004: dropped implicit ``class X(object):``.
- UP022/UP008/UP032: stdlib idiom modernization (``capture_output``,
super-call simplification, etc.).
- B905: ``zip(...)`` → ``zip(..., strict=False)`` (preserves prior
behavior; switching to ``strict=True`` is a separate semantic
choice not made here).
- I001: import-block ordering.
Hand-fixed (66 issues):
- B904 × 54: appended ``from <bound>`` (when the enclosing ``except``
binds a name) or ``from None`` (when it doesn't) so chained
exceptions stay distinguishable from errors-during-exception-
handling. Applied via a small AST-aware script that walked back to
the enclosing ``except`` clause and matched the binding.
- B017 × 4: narrowed broad ``assertRaises(Exception)`` to the actual
``AnsibleError`` / ``AnsibleFilterError`` raised by the code under
test.
- B023 × 2: bound captured loop variables via default args
(``def _record(name, *, _path_real=path_real, ...)``) so each
closure gets its own snapshot.
- B007 × 2: unused loop control variables renamed to ``_`` /
rewritten to ``.values()``.
- UP007 × 2: ``Union[X, Y]`` → ``X | Y`` (plus dropping the now-unused
``from typing import Union``).
- B018 × 1: Odoo addon manifests are bare dict expressions at module
level by design — suppressed with an inline ``# noqa: B018`` and
rationale comment.
- UP031 × 1: printf-style format → f-string.
The 65 ``# noqa: …`` markers introduced by the auto-fixer all carry
real ruff codes (``B018``, ``E402``, ``F401``, etc.), so
``test_noqa_only_ruff_codes`` stays green.
Verification
============
- ``ruff check .`` → All checks passed
- ``ruff format --check .`` → 894 files already formatted
- ``tests.lint.repository.{test_project_root_import,
test_no_direct_yaml_calls, test_noqa_only_ruff_codes}`` and
``tests.unit.utils.annotations.test_suppress`` → 20 tests OK
- 1621 unit tests → all pass
- Integration suite: 44 failures (pre-existing baseline, none from
this change), 1 skipped.
Going forward, the new rules block regressions on E (style), F
(pyflakes), I (import order), B (likely-bug patterns), and UP
(modernization to py311) for any new code.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reason: ruff parses every `# noqa: <code>` marker as a flake8 directive
and warns on every non-flake8 rule key. Every catalog rule in
`docs/contributing/actions/testing/suppression.md` is project-specific
(kebab-case), so `# noqa:` markers fired the warning on every parse.
`# nocheck:` is the project synonym ruff ignores.
Scope:
- 41 `roles/*/meta/services.yml` + 1 task + 1 .cfg.j2 template: pure
noqa→nocheck swap for catalog rules. Mixed flake8 markers
(`# noqa: E402, X`) preserved verbatim.
- 5 production .py + 2 production scripts: same swap.
- All catalog-rule lints' docstrings updated; redundant "or # nocheck:"
parentheticals collapsed to a single nocheck form.
- `utils/annotations/suppress.py`: docstring rewritten to spell out
the new convention. Parser still accepts both keywords as synonyms.
- `docs/contributing/actions/testing/suppression.md`: convention
rewritten to mandate `nocheck:`; missing `domain-spot` and
`literal-protocol-lookup` rows added; "enforced by" pointer added.
New guard:
- `tests/lint/repository/test_noqa_only_ruff_codes.py`: meta-lint
failing any `# noqa: <code>` whose codes don't match `[A-Z]+\d+`
(real ruff/flake8 shape). Allowlists the impl, its unit test, the
catalog page, the direct-yaml lint, and itself.
- `tests/lint/repository/test_no_direct_yaml_calls.py`: hardened to
reject the `noqa:` form via inlined regex (no longer routed through
`utils.annotations.suppress`).
Tangential cleanups carried in this same commit:
- `utils/cache/yaml.py`, `utils/handler/yaml.py`: 6 `direct-yaml`
markers swapped.
- `utils/cache/base.py`: `from . import PROJECT_ROOT, ROLES_DIR` moved
up to the import block (E402).
- `cli/create/role/__init__.py`: reordered so PROJECT_ROOT is defined
before `__main__` is imported (also fixed a latent circular import).
- PROJECT_ROOT migration tail: 2 new `__init__.py` modules + 10 test
files swapped from `def repo_root()` to `from . import PROJECT_ROOT`.
- ~20 ruff auto-fix removals of unused `from pathlib import Path`.
Verification:
- `ruff check .` clean.
- 23 suppression-related tests pass.
- The new lint detects all three regression shapes (project rule alone,
mixed with real ruff code, lowercase-only).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Migrate the simple `PROJECT_ROOT = Path(__file__).resolve().parents[N]`
shape across the repo to the new "import from `__init__.py`" rule
introduced by the `project-root-import` lint:
- `utils/cache/__init__.py` is the canonical source. `PROJECT_ROOT`
and `ROLES_DIR` move out of `utils/cache/base.py`; `base.py`
re-exports them for callers that still import from there.
Internal cache modules (`files.py`, `gitignore.py`) switch to
`from . import PROJECT_ROOT`.
- 24 packages receive a `PROJECT_ROOT` declaration in their own
`__init__.py`, each with the parents[N] depth that resolves to
the repo root.
- ~36 consumer files swap their local
`PROJECT_ROOT = Path(__file__).resolve().parents[N]` for
`from . import PROJECT_ROOT`. `ROLES_DIR` and similar derived
paths stay local (`PROJECT_ROOT / "roles"`).
Status: this commit closes the simple-shape violations. The
multi-line `os.path.abspath(os.path.join(here, os.pardir, …))`
shape, `def repo_root()` while-loops, and `roles/<role>/filter_plugins`
files without a package container remain. The lint stays scharf and
will track each subsequent wave.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two intertwined changes that share the same touched files:
1. Move every meta-related integration test under
`tests/integration/roles/meta/` into one of three sibling
sub-clusters and shorten the redundant filename prefixes:
- `services/` — `test_services_dynamic_flags.py` →
`test_dynamic_flags.py`, `test_services_no_self_reference.py` →
`test_no_self_reference.py`. The `run_after` cluster moves
under it (it is a `meta/services.yml` concern, not a
stand-alone dependency one): `test_services_explicit.py` keeps
its name, the rest drop the redundant prefix.
- `variants/` — `test_variants_coverage.py` → `test_coverage.py`,
plus the two new lints `test_auth_coverage.py` (LDAP / OIDC /
OAuth2 matrix completeness) and `test_services_match.py`
(every variant override key MUST exist in `services.yml`).
- `dependencies/` — relocates the legacy
`tests/integration/roles/dependencies/` cluster so all
`meta/`-shaped tests live under one parent.
Path-index constants (`Path(__file__).resolve().parents[N]`,
`os.pardir` chains) updated for the deeper nesting.
2. Route every yaml read in the migrated tests through
`utils.cache.yaml`. The single-doc helper `load_yaml_any` covers
most of them; the new `load_yaml_all` mirrors `load_yaml_any` for
the multi-doc case (`yaml.safe_load_all`) and is keyed off the
same `(path, mtime, size)` signature, with parallel `_drop_path`
/ `_reset` housekeeping. `tests/lint/ansible/variables/
test_vars_usage_in_yaml.py` is the first user.
READMEs cleaned up to match the new directory structure (path
crosslinks, level-up counts) and four new READMEs added for the
new sub-clusters under the `docs/` README guidelines (RFC 2119,
emojis after headlines, no em dashes, no SPOT self-labels).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
