meta/schema.yml held credential definitions and their runtime values, but the
name said 'schema' and read like structural metadata next to services.yml and
volumes.yml. Nothing in the file is a schema in the sense the other meta files
use the word, and a contributor had to open it to learn it holds secrets.
62 role files renamed. ROLE_FILE_META_SCHEMA becomes ROLE_FILE_META_SECRETS and
keeps its role as the SPOT the path is built from.
Two readers bypassed that constant and spelled the name themselves:
utils/cache/applications.py built meta_dir / 'schema.yml' twice, and
plugins/lookup/application_gid.py carries the filename in the marker list it
detects application roles by. Left behind, the rename would not have raised —
credentials would have resolved empty and the role would have stopped counting
as an application.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.
Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The internal container port slot under <entity>.ports is now a
category-keyed map (matching the local/public shape) instead of a
single int. "internal" replaces "inter" because the latter is
ambiguous in English (could read as "between").
Affected:
- 8 role meta/services.yml (matrix, ollama, mig, shopware, oauth2-proxy,
gitea, listmonk, sphinx) and every caller (vars, templates, group_vars).
- web-app-matrix: 8008 hardcoded across compose.yml.j2, homeserver.yaml.j2,
tasks/01_docker.yml, and 7 mautrix bridges now reads
MATRIX_SYNAPSE_CONTAINER_PORT from services.synapse.ports.internal.http.
- web-app-matrix: admin username/password lookups consolidated as
MATRIX_ADMINISTRATOR_{USERNAME,PASSWORD} in vars/main.yml, sourced from
lookup('users', 'administrator').
- Tooling: cli/meta/ports/suggest, utils/meta/scan, integration scan,
migrate_meta_layout/role_builder updated to the new key name.
- Docs: design/role/services/layout.md, tools/ports-suggest.md,
requirements 009/021, image/origin.md, PR template.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
TC (flake8-type-checking): typing-only imports moved into
``if TYPE_CHECKING:`` blocks across 8 source files via auto-fix
(117 hits). Reduces runtime import surface and rules out circular
imports for type-only deps. Project-wide ``from __future__ import
annotations`` (FA, already enabled) keeps annotations as strings,
so the moved imports never need to load at runtime.
N (pep8-naming): 32 hits handled.
Renames (semantic + lint):
- ``isSmallerThenMaximumBackupSize`` -> ``is_larger_than_maximum_backup_size``
(snake_case + fixes both the typo "Then -> Than" and the
semantic mismatch: the function returns ``current > maximum``)
- ``isDirectoryDeletable`` / ``deleteVersion`` /
``getAmountOfIteration`` / ``deleteIteration`` -> snake_case
- 8 test functions with ALL-CAPS env-var fragments
(``test_..._DOMAIN_PRIMARY_...`` etc.) -> snake_case
- Mock alias ``VH`` -> ``vh`` in test_host_vars.py
Exception class renames (``N818`` Error suffix):
- ``TimeLimitExceededException`` -> ``TimeLimitExceededError``
- ``AttemptException`` -> ``AttemptError``
- ``PathNotFound`` -> ``PathNotFoundError`` (propagated across
4 callers under tests/integration/lookups/config/)
- ``_ExitJson`` / ``_FailJson`` -> ``*_Error``
Refactor:
- ``METADATA = {...}`` local constant in two functions of
``plugins/filter/invokable_paths.py`` lifted to module-level
``_METADATA_KEYS``.
- ``from typing import TYPE_CHECKING`` in
``tasks/utils/migrate_meta_layout/__main__.py`` moved above
the ``sys.path.insert`` hack to satisfy E402 (collateral from
the TC auto-fix that landed it after the path mutation).
Per-file ignores in pyproject.toml for the legitimate cases the
rule cannot accept:
- ``pull-specific-host.py`` -> ``N999`` (Ansible-invoked role-level
script, hyphenated filename pinned by the surrounding stack).
- 4 matrix-deploy files -> ``N806`` for the ``include_R`` /
``raw_include_R`` round-index notation; the uppercase ``R``
here mirrors the formula in the docstring rather than naming
a constant.
- ``plugins/filter/value_generator.py`` -> ``N806`` for the
lazy-loaded class binding ``ValueGenerator =
_value_generator_cls()``, which retains the imported class
identifier per PEP 8.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_role_file_spot.py: AST-walks every .py
file under the production trees and tests/, flags any reference to a
role file (standalone literal, f-string segment, pathlib `/` chain,
`+` concat chain, os.path.join / Path.joinpath / posixpath.join /
ntpath.join call) whose value or joined trailing operands match a
ROLE_FILE_* constant in utils.roles.mapping. The flag set is
auto-discovered from the module so adding a new constant extends
coverage without further edits. ROLE_FILE_README is exempt as too
generic. Per-line `# nocheck: role-file-spot` (same-or-above) is the
documented opt-out, catalog row added in
docs/contributing/actions/testing/suppression.md.
Migrate 139 .py files (377 occurrences) to use the constants:
- 135 files via direct constant import (`from utils.roles.mapping
import ROLE_FILE_*`), with the symbol used in pathlib chains, in
os.path.join / Path.joinpath calls, in f-string interpolation, and
in glob patterns.
- 4 files via `# nocheck: role-file-spot` where the literal lives in
argparse help text or in an allowlist tuple meant to declare the
literal path.
Adjust utils/roles/dependency_resolver.py to use lazy-formatting
positional args in the logger.exception call (G003-clean) instead of
string concatenation against the constant.
Drop a redundant `if cond: return True; return False` guard in the
new test for `return cond` (SIM103-clean).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighten the utils namespace so each subpackage owns one concern:
- utils/applications/* moves under utils/roles/applications/* (helpers
that operate on the runtime applications dict and meta/services.yml).
- utils/service_registry.py and utils/database_service.py move under
utils/roles/applications/services/ as registry.py and database.py.
- utils/entity_name_utils.py becomes utils/roles/entity_name.py.
- utils/invokable.py, utils/valid_deploy_id.py, and the existing
utils/roles/resource_validation.py group under utils/roles/validation/
as invokable.py, deploy_id.py, and resources.py.
- utils/templating.py and utils/jinja_strict.py split into
utils/templating/ansible.py (Templar + TrustedAsTemplate, lookup-env
shortcut) and utils/templating/jinja.py (vanilla Jinja2 Environment,
multi-pass strict render).
Every caller in utils, cli, plugins, roles/<role>/{filter,lookup}_plugins,
tests, and the docs that link to the moved files is updated to the new
import paths. Test files mirror the new source layout
(tests/unit/utils/{roles,templating}/...).
New READMEs at utils/cleanup/, utils/roles/, utils/roles/applications/services/,
utils/roles/validation/, and utils/templating/ follow the
documentation.md and readme_md.md guidelines.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds ``[tool.ruff.lint]`` with ``select = ["E", "F", "I", "B", "UP"]``
and ``target-version = "py311"``. ``E501`` is ignored because
``ruff format`` already reflows what it can; the remaining long lines
are unsplittable string literals / URLs.
Auto-fixed (2753 issues, behavior-preserving):
- UP006/UP035: ``List[X]`` → ``list[X]``, ``Dict[X]`` → ``dict[X]``,
``from typing import …`` redirected where superseded by builtins.
- UP045: ``Optional[X]`` → ``X | None``.
- UP010/UP009/UP015: dropped Python-2-era pragmas (``from __future__
import absolute_import, division, print_function``,
``# -*- coding: utf-8 -*-``), redundant ``open(..., "r")`` mode.
- UP004: dropped implicit ``class X(object):``.
- UP022/UP008/UP032: stdlib idiom modernization (``capture_output``,
super-call simplification, etc.).
- B905: ``zip(...)`` → ``zip(..., strict=False)`` (preserves prior
behavior; switching to ``strict=True`` is a separate semantic
choice not made here).
- I001: import-block ordering.
Hand-fixed (66 issues):
- B904 × 54: appended ``from <bound>`` (when the enclosing ``except``
binds a name) or ``from None`` (when it doesn't) so chained
exceptions stay distinguishable from errors-during-exception-
handling. Applied via a small AST-aware script that walked back to
the enclosing ``except`` clause and matched the binding.
- B017 × 4: narrowed broad ``assertRaises(Exception)`` to the actual
``AnsibleError`` / ``AnsibleFilterError`` raised by the code under
test.
- B023 × 2: bound captured loop variables via default args
(``def _record(name, *, _path_real=path_real, ...)``) so each
closure gets its own snapshot.
- B007 × 2: unused loop control variables renamed to ``_`` /
rewritten to ``.values()``.
- UP007 × 2: ``Union[X, Y]`` → ``X | Y`` (plus dropping the now-unused
``from typing import Union``).
- B018 × 1: Odoo addon manifests are bare dict expressions at module
level by design — suppressed with an inline ``# noqa: B018`` and
rationale comment.
- UP031 × 1: printf-style format → f-string.
The 65 ``# noqa: …`` markers introduced by the auto-fixer all carry
real ruff codes (``B018``, ``E402``, ``F401``, etc.), so
``test_noqa_only_ruff_codes`` stays green.
Verification
============
- ``ruff check .`` → All checks passed
- ``ruff format --check .`` → 894 files already formatted
- ``tests.lint.repository.{test_project_root_import,
test_no_direct_yaml_calls, test_noqa_only_ruff_codes}`` and
``tests.unit.utils.annotations.test_suppress`` → 20 tests OK
- 1621 unit tests → all pass
- Integration suite: 44 failures (pre-existing baseline, none from
this change), 1 skipped.
Going forward, the new rules block regressions on E (style), F
(pyflakes), I (import order), B (likely-bug patterns), and UP
(modernization to py311) for any new code.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a SPOT-aware lint that walks the repo via the cached
project-tree iterator (no `git` subprocess) and fails on any tracked
non-test Python file longer than 500 lines. Files under `tests/` are
intentionally excluded — coverage breadth there reflects feature
matrix, not design weight. A per-file `# nocheck` (or `#nocheck`)
marker in the first 30 lines opts the file out; the marker is
intentionally visible so the cost of carrying long modules is not
silently buried.
Gitignore-aware filtering moves to its own SPOT
-----------------------------------------------
The fnmatch-based `.gitignore` parser/matcher previously lived inline
in `tests/lint/repository/test_test_files_location.py` (and in
near-duplicates across other tests). Pull it into
`utils.cache.gitignore` as a single source of truth:
* `load_gitignore_patterns(root)` — cached pattern loader.
* `is_path_gitignored(rel_path, patterns)` — pure matcher.
* `iter_non_ignored_files(...)` — convenience iterator in
`utils.cache.files` that composes the cached project walk with the
matcher.
`utils.cache._reset_cache_for_tests` now resets the gitignore module
too, so test fixtures that mutate the tree see consistent state.
Refactor / opt-out for existing offenders
-----------------------------------------
Three production files exceeded the cap; addressed each:
* `roles/sys-svc-compose-ca/files/compose_ca.py` — opted out via
`# nocheck: file-size`. The script is delivered to target hosts
via `ansible.builtin.copy` and runs with only the stdlib; splitting
would require also delivering a package, which the role does not
do.
* `tasks/utils/migrate_meta_layout.py` — converted to a package
(`tasks/utils/migrate_meta_layout/`) with focused submodules:
- `__main__` — entrypoint + run-orchestration.
- `yaml_io` — load/dump/empty-dir/deep-merge helpers.
- `credentials` — schema-vs-config collision detection +
runtime-to-schema conversion.
- `role_builder` — per-role meta file emission.
- `indices` — networks + ports + relays index builders.
Run via `python3 -m tasks.utils.migrate_meta_layout`. Largest
submodule is `role_builder.py` at 250 lines.
* `cli/fix/move_unnecessary_dependencies/__main__.py` — split
alongside two new sibling modules in the same package:
- `analysis` — provider/consumer static analysis.
- `apply` — meta rewrite + guarded include-block prepending.
- `yaml_io` — ruamel round-trip helpers + role-tree discovery.
Largest submodule is `analysis.py` at 174 lines.
`make test` green (162 tests, 1 skipped); the new lint passes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
kevinveenbirkenbachandClaude Opus 5